ShapeDiver connector

[snabela (Alex Schiftner)]

Hi specklers,

I have been following Speckle since the beginning. It’s incredible what you have achieved. Kudos to all of you!

Actually we do have a little bit of a common history. @dmitry remember our brainstorming end of 2015 when we had just founded ShapeDiver? And special thanks to @Pavol for working with us along the way.

At ShapeDiver we have been pondering about establishing a connector to Speckle since a long time, now it’s time to finally get the ball rolling. Here are our initial thoughts, looking forward to your feedback and ideas.

Motivation

Connecting ShapeDiver with Speckle will enable use cases like the following ones.

Use case 1 - Feed data from Speckle into ShapeDiver

Get data from Speckle streams into Grasshopper models running on ShapeDiver. Imagine the model has a text input called Speckle stream URL. When interacting with a model on the ShapeDiver Platform, users can paste their Speckle URLs (pointing to specific branches or commits) and trigger computations of the model which read data from the stream.

Use case 2 - Feed data from ShapeDiver to Speckle

Export data from ShapeDiver to Speckle. This could be done as part of any computation of a ShapeDiver model using Speckle’s Grasshopper plugin. As an example, the results of an online configurator can be pushed to Speckle streams. Use cases 1 and 2 can be combined to achieve a workflow that gets data out and back into Speckle in a single step. Depending on the specific application, the fact that Speckle is used could be hidden from the end user.

Use case 3 - Speckle bots

Create bots that run automatically based on events happening in Speckle. Imagine a bot that runs some analysis tasks or creates some geometry in Grasshopper based on every commit sent to a specific branch of a Speckle stream. The results can be pushed again to the same or another Speckle stream.

Eventually, this can orchestrate several processing steps into a complete workflow or pipeline.

Idea for implementation

Quite some technical details here, but it would be great to have your thoughts on them as well.

1. Deploy the Speckle Grasshopper plugin to ShapeDiver’s Geometry Backend systems. Some components may not make sense or pose security risks and must be deactivated on ShapeDiver. This needs to be reviewed but should only be a minor issue.
   Speckle’s Send nodes need to have auto-sending switched on because there is no way to click the Send button when running Grasshopper models on ShapeDiver. The Receive node should have auto-receiving switched off. Ideally, the Speckle stream URLs would be fed into the model using ShapeDiver’s Direct text input components, allowing a model to react to different streams.
   This implementation step enables use cases 1 and 2 described above.

2. Set up a microservice to trigger computations of ShapeDiver models based on Speckle webhook calls. This microservice should allow users of ShapeDiver to set up Webhook receiver URLs, which identify the ShapeDiver model to call and the credentials to use. The microservice must also know the optional secret specified for the Speckle webhook.
   Every time a Webhook receiver URL is called by a Speckle webhook, the following happens:

   • Verify the webhook call using the secret.
   • Open a session with the ShapeDiver model identified by the Speckle webhook receiver URL, inspect the parameters defined by the model, and match them to the properties of the Speckle webhook payload. Some examples:
     • A text input called streamId would be matched to the streamId property of the payload.
     • A text input called event.event_name would be matched to the event.event_name property of the payload.
     • A text input called payload would be matched to the complete webhook payload, which can be decomposed in Grasshopper using JSON components.
   • Trigger the computation. It needs to be clarified whether the microservice should wait for the computation to finish or return immediately. Whether waiting makes sense depends on the expected computation time. Long computation times can lead to trouble with HTTP timeouts, and waiting for them would also significantly increase the cost when running the microservice on Lambda.

3. Speckle’s Send node can send data back to Speckle from the Grasshopper model running on ShapeDiver, i.e., it seems there is no need for a webhook the other way around. The stream URL could be configured using a text input.

This will be amazing, I can’t wait to see this working

[jonathon]

Hey @snabela, thanks for following up on my call to action on LinkedIn. This is a conversation we’d love to have with you. Thanks so much for laying out such detail already.

I’ll send you a calendar booking, and let’s try and align something.

[snabela (Alex Schiftner)]

I started to dig into the Speckle Grasshopper connector to figure out the details of how we are going to deploy it to ShapeDiver’s Geometry Backend systems.

Security

From a security point of view, we need to identify which functionality to prevent because it could potentially harm our backend system. Typically, this kind of functionality doesn’t make much sense when running Grasshopper in the cloud anyway, like file system access on our cloud workers running on EC2.

I suppose the Local Receive and Local sender components read/write data to/from a local SQLite database.

Are there other components (including obsolete ones) that could potentially access the file system? If so, please let me know, because we need to flag them as forbidden components.

Usability

We need to identify best practices when using Speckle on ShapeDiver. It seems to me that we should discourage the use of the common Send and Receive components when running Grasshopper models on ShapeDiver, and encourage the use of Synchronous Receiver and Synchronous Sender. My concerns with allowing the use of the Send and Receive components: Users might not realize by themselves that they need to activate sending or receiving automatically.

One option would be to forbid the use of Send and Receive, and provide feedback to use Synchronous Receiver and Synchronous Sender instead. A better user experience might be getting the Send and Receive components to behave synchronously when running on ShapeDiver (or Rhino Compute).

What are your thoughts on this? How are you dealing with this when using Speckle with Rhino Compute?

Speckle Manager and Accounts

When sending data to Speckle streams from our cloud workers, and also when reading data from streams, which Speckle account do we use? Right now I don’t see how Speckle users could use their own accounts. This would require some “API key” feature in the Speckle Grasshopper connector, allowing users to provide their API keys via Grasshopper.

A current solution would be to use a global “ShapeDiver” account. Users would then add this account as a Contributor or Reviewer of their stream. However, this won’t work ootb for private Speckle servers.

Looking forward to making this work!