I’m messing around with a few things using automate and the preview service built into speckle-server, and I realized that the preview is a .png with a transparent background that I could invert and use as a mask with for some img2img stuff. I thought it would be handy to get that mask directly from the preview service, but I wasn’t sure if that was supported.
Does the preview service have any hidden ways to retrieve additional image data (like masks)? I poked around in a few scripts in the frontend to see if pulling panoramas from the preview service would lead to any discoveries, but no luck
You can use the viewer API to generate images, or a series of images. The preview-service actually just calls some viewer functions in order to create the preview images. If you go to the sandbox you can use the screenshot button to generate an image (which is dumped in the console as base64). There also the 360 button that simulates just what the preview-service does.
You can also generate depth, normal, AO images instead of the final color. Basically, the various stages of the viewer pipeline can be exported as images if you want to. If you can give us more details about what you want to do with the masks, and how should they look, I can provide further assistance
@alex providing helpful and speedy info as always This helps me understand a bit more of how the service works under the hood.
I’m working on a side project that currently takes the preview images from a version/commit, passes the image into dalle as a ImageEditRequest along with the commit message as the prompt. Dalle has an optional parameter for passing in a image mask as a way to tell la computadora what part of the image to apply the prompt to. It would be great to experiment with a material id rendering down the road, but for now I’m keeping it simple.
For reference, here are a couple of tests I’ve done so far without masks. I’m focusing on using versions/commits at the moment since I like the idea of having commit messages influencing the output.
@haitheredavid I see what you are trying to do, and it’s pretty cool!
Here is a sandbox that shows how to generate a binary mask with the objects ids you choose as enabled.
You can theoretically use the preview-service to generate this mask, since it’s still the viewer that it’s using to generate the images. However I think it would be easier to do it via a realtime web app. What I’m assuming you are currently doing is fetching previews like this and you’d probably need a way to generate a customizable image mask to pair with the preview.
Thank you for this! This makes sense, I’ll have to consider how I could approach it via the webapp.
You got it right! I’m storing that preview and using it make a hacky inverted png mask to pair with it. It’s a pretty rough mask, but including the mask is more a wish list item at the moment so I’m okay with the results. I’m aiming for this to be an automation function so I think I’ll keep relying on the preview-service for this prototype, but I’m sure there will be plenty to explore once it’s completed!
This helps me understand a bit more of how the service works under the hood.
This makes sense, I’ll have to consider how I could approach it via the webapp.