Objective: Exporting a 2D view of a 3D Model and generate a rendering based on a text description.
We’ve seen a lot of open source development these days around generative image design; few solutions are proposed.
New to the AEC field, I’d like to make a “plugin”/“service” for Speckle that would simply allow you to export a model and generate an illustration like “Midjourney” can do.
As illustration, the idea is to build an open source alternative to “Veras” from Evolvelab.
The main goal here, is to control the generation process to respect the model geometry with all composants (door position, windows position, …).
To do this, two things are needed:
1 - Be able to export an image of the structure and the topology (scene) from a Speckle model. Is it possible to export an image from the viewer? Without the viewer?
As I’m not familiar enough with the Speckle ecosystem, I don’t know what the best solution would be.
For instance, some of the control network use a representation based on line segmentation.
2 - The contribution of the community to share renderings with associated models in order to create a suitable training/specialization dataset for speckle data. The idea is to build a tool that outperform standard neural network on speckle model.
Welcome to the community! Feel free to Introduce yourself to the community if you want to!
This sounds like a cool idea! To answer your questions:
Yes, it is possible. As a proof of concept, you can try to use our comments functionality to hack it. Comments/discussions also store a screenshot of the comment view. You can create a webhook that gets triggered when a comment is added and its content contains the word “/imagine” similar to Midjourney😉. Then you can get the comment screenshot and use that for the rendering.
I did something similar a couple of years ago to get a screenshot of a comment. Beware that code might be changed since then.
def get_comments(stream):
query = gql(
"""{
comments(streamId:\""""
+ stream.id +
"""\") {
items {
text
id
authorId
createdAt
data
resources{
resourceId
}
archived
screenshot
}
}
}"""
)
# Making query to Speckle
comments = client.execute_query(query=query)
return comments
I’ll leave the stage to the viewer wizard @alex and @dimitrie to share their expert opinion on this.
I think there are already models out there that are pretty good for doing archviz ai work. Can’t you directly use those?
I hope this helps. Please share your updates on this.
Hi @gokermu, thanks a lot for the idea to hack the comment webhook.
I will start using this solution with a standard model.
The core concept is to develop a model that leverages the speckle model/geometry. Midjourney, DALL-E, and similar networks are essentially “imagination” networks that generate buildings without constraints.
As explained in the edit of the initial comment, we can employ examples to train the model, ensuring it adheres to the speckle model while incorporating additional contexts such as background and foreground. For instance, ControlNet utilizes line segmentation algorithms to extract constraints from an image, allowing it to “control” the output of a generative model. We could do this representation based on the geometry to have a better results.
But that’s the theory, which is why I’m asking for help on this part as well.
to the community if you want to! 