BREAKING CHANGE: The way we handle Grasshopper DataTrees is about to change!

AlanRynne · 24 May 2022 08:20

we’re back with a minor announcement for our Grasshopper connectors:

The way we handle DataTree’s in grasshopper is about to change:

Let me explain…

Current behaviour

Up until now, we’ve been converting Grasshopper DataTrees into nested lists, which has been working fine but has some unwanted side-effects in order so fully support Gh->Gh interop.

Mainly, in order to preserve branch indices, we’ve been adding empty placeholder lists whenever necessary. This lead to situations were sending a single point will create many unwanted nested lists before you could get to your data.

This would force your data to be nested it at least 2 lists (one for the data tree, another for the branch) in its simplest form.

New behaviour

Now, we’re changing the way we convert DataTree’s. Instead of becoming nested lists, they will now become Base objects who’s key’s are the DataTree’s branch paths.

This is, in reality, how Grasshopper internally works. A DataTree is just a dictionary where every Path contains a list of objects.

This simplifies the way you treat and deal with Grasshopper data:

DataTree’s are now easier to view and explore. No more deeply nested lists.
Have some ugly DataTree structures? we’re not here to judge! They’ll now be represented more accurately than before in cases where branches of different levels are mixed (i.e. {0;0} and {0;0;0} (these were already supported, but lead to index shifting in some cases and loss of data).

What does this mean for you, the average speckler?

Nothing really, if you’re using our connectors to send and receive your Speckle data everything should work exactly as it did before.

The only difference you’ll notice is that now the Grasshopper data is easier to navigate in our web application, and the increased support for unconventional data structures.

What does this mean for developers?

If you’ve developed your own connector, or have scripts or plugins that processed Grasshopper commits, you should check that they are capable of:

Consuming pre-existing commits that had nested lists
Consuming the new Base object with paths as keys.

The new case can be handled just like any other Base object. GetDynamicMembers will give you the names of all the paths, and the values will always be List<object>.

What if i depend on/need the nested list structure?

Right now, we haven’t exposed a way to convert these Base objects representing DataTrees into nested lists, although it does exist within our codebase.

If you feel the need for this, or anything else…

We’d love to hear from you!!

If you got this far down, you must have an opinion about all these changes, and we’d love to hear about it! Is this what you’ve always been waiting for? Will this break your entire universe?

We’d definitely want to avoid having angry Specklers with pitchforks at out doorstep! Now its your time to let us know, and help us shape Speckle in the process!

JdB · 25 May 2022 09:12

I think people will throw flowers at you instead of picking up the pitch forks, I think it’s a great improvement!
Colleagues have used workarounds because it wasn’t possible to just dump in your data structure and receive it exactly the same way in another GH script, so I believe people will be happy with it.

I’m not sure if there would be use cases where the current behavior would be preferred (maybe during the transition period?).
If so, it might be good to temporary have 2 sender components so the old one can be easily phased out? Or add an option to the right mouse button menu of the sender component where you can toggle between the two behaviours (with the new behaviour being the default one). For the latter option there would need to be a use case (again, not sure if there is one) otherwise it doesn’t really make sense to have the option available (maintenance wise and unnecessary complexity for the users).

AlanRynne · 25 May 2022 10:07

Hey @JdB, thanks for the feedback!

I currently implemented it as a change in the existing Send/Receive nodes, but I think it does make a bit more sense to have them be new ones instead.

That will also allow us to

Not force this change upon user’s unknowingly
flag the existing ones as “old” in the default GH way for better discovery of this “new” functionality
provide auto-upgrade features as we’ve done in the past with other connectors.

I’d prefer separating the 2 (and then just remove the old ones at some point) to adding more options to the right-click menu tbh… I think it’s more maintainable that way.

EDIT: Not sure if I mentioned this yet, but it is backwards compatible with the nested list approach. So all your old commits will still work

haitheredavid · 26 May 2022 12:11

I very much support this change! Displaying the tree path as the object name is really clean and similar to the gh structure.

Curious to knwwwwwwjfsduiohfdsakfhalkjffd faesddssssferaarrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrraaarrrrrr

…. wanted to sleep on the keyboard

Curious to know if there would ever be a DataTree Object within Speckle Core or Objects?

AlanRynne · 26 May 2022 12:24

Well, not sure about this. We’d definitely like to keep “application specific” classes to a minimum, and we’re currently undergoing a test to refactor our BIM objects too (and their app specific Revit ones) to remove the need for some/most of them So following that, I’d argue we won’t add one.

But, as always, we’re willing to let you convince us otherwise, and make a proposal outlining what would be the benefit, how would you like it to work… etc.

I’d definitely be up to it if there’s any other program that deals with data in a similar manner, just for consistency. Otherwise, I’d say if you want to operate on data trees you should use GH
I’m also open to exposing some utility methods to know if an objects fits the “data tree” description, and to go from nested lists to Speckle Object and back if that is something that would be useful to the community.

haitheredavid · 26 May 2022 23:43

I got no real argument to make for em’ It makes sense to keep things generalized. But I do like the ideas of utility method! maybe like a recursive check for when an object is IList or something like that… would mos def cut down on my copy pasta from the speckle-sharp converters.

AlanRynne · 31 May 2022 10:48

So I’ve made some changes to the implementation after your suggestions, but kept them to a minimum. Thanks a lot for all your input.

Only the Send node has been flagged as obsolete, and a new Send node has been created that works as explained before. You can automatically update it as we’ve done in the past when removing other nodes.

The Receive node has no breaking changes, so there is no need to create a new one, it will just work on both cases.

Added handling of this new case in Deconstruct Speckle Object, so it will play nice with “Do not expand” option on the Receiver.

Added chunking to prevent huge lists from being an issue. But that’s had an unwanted side-effect. Since we’re doing dynamic chunking, the name of the prop is now a bit ugglier:

instead of {0;0;0} it would be @(1000){0;0;0}, the prefix containing the maximum allowed items per chunk. This is quite technical, and I’m hoping we could hide it in the Frontend side so you’d really never have to care.

As for exposing the data tree to nested list conversions somewhere, I’d have to have a detailed discussion with the team about where those should go. So they may come a bit later to not freeze this PR from going in

haitheredavid · 1 June 2022 17:13

ay ay! these look great, thank you for passing along the progress updates. This thread gave me a bit inspiration for improving how unity handles lists. The connector created an unity object in the scene hierarchy for each list and then parented each item to that object. This would work for receiving, but ended up being a bit of a pain for sending.

I created a Node unity object that handles the connection of a commit in unity. When a commit is pulled it creates a SpeckleLayer for each list of objects.

A SpeckleLayer is like a rhino layer where it can have nested layers within it, but has a bit more support for unity specific things (like parenting… never a fan of forcing unity peeps to modify their transforms!)

The ending result is like this each layer has a list of objects and a list of nested layers. Would love to know your thoughts on this! I was trying to replicate the rhino / gh structure.

I’d like to test this with the new gh structure you’ve designed! Would you be able to link that stream?