Kit and Converter support for Unity

[haitheredavid (David Morgan)]

Hi

I recently just went through the process of stripping out my project code from the speckle-unity repo and exporting the speckle-unity project as a one off unity package that I could import into my other unity project. All is well in good with that but it made me realize that most of that hassle I went through was separating out the ISpeckleConverter class for unity and how that get’s integrated within the SpeckleManagerConnectorThing. It would be helpful to consider how a client, like unity, might deal with handling kits and converters (mainly converters).

I thought it would be good to have a lil’ convo around this process. Here are some of things I think a solution would need to address.

Serializing the Converter in Unity
Right now in order for a converter to work in Unity a user needs to build out that converter, reference it in the ConnectorUnity namespace, and then modify the Receiver class to use that converter. What would be ideal is creating a simple object that any receiver or sender could have access to and rely on that for converting (just like it does in any other connector) without the need of hardcoding it. But we would need to create a unity friendly object that it could rely on for storing and accessing the different converters. For example, ConverterUnity is the current Object converter that is just a simple object that can be instanced in unity during runtime or editor, but it won’t be serialized since unity doesn’t chill like that with objects. We could attach MonoBehaviour or a ScritableObject class to it, and then unity would be a-okay with serializing that.

I did a pass at addressing this issue by creating an abstract object that a converter could inherit from and then adding those to a container scriptable object. Here is the UnityConverter with that class implemented. The kit and converter object really just a way for the speckle manager to know what converter is should utilize.

Here is an epic gif of the manager with reference to those converters (flexing that logo).

I’m sure there is a more elegant solution that could be built out with reflections, robots, maybe even a skylight, but I’m more of a peon type of developer.

Packaging Converters and Objects
Currently speckle converters and objects need to be added directly to the project. I’m sure on the unity-connector roadmap there some plan to migrate the project into the package manager, but I think it would be helpful to have objects and their converters be separated too. I’ve been using OpenUPM to package my viewobjs and I plan on doing the same with the unity-converter. However, my unity converter piggy backs off the speckle one (ConverterUnity, no need to repeat mesh building code…) so I can’t separate that out from the main project repo. This structure would be a bit more relatable to the current structure of other connectors too.

anndd we out

[Jedd (Jedd Morgan)]

Hi David,

YES! This is definitely worth a discussion.

It sounds similar to a bit of a dilemma I’m having on the Speckle-Unreal side also.

I’m going to take this opportunity to piggyback some of my issues into this discussion because I imagine they will be the same in Unity.

An issue I’m facing with Speckle-Unreal, is there are several types of native meshes that a user might want to convert to.

Secondly, I want to allow users to write their own converters without having to modify the Speckle-Unreal plugin’s source code.

Thirdly, some objects, like point-clouds and breps aren’t natively supported by Unreal Engine, but they are through a plugin. I’m hesitant to make the Speckle-Unreal connector dependent on plugins (particularly third party ones) that most users won’t even need, and so I’m currently toying with this idea of modular converters.

---

In my (half baked) idea of modular converters, we would have Mesh, Camera, Blocks, Point-Cloud etc conversion, all in separate classes implementing an interface (eg. ISpeckleTypeConverter)

We would store the active ISpeckleTypeConverters in a TMap (c# equivalent would be a Dictionary) that maps a Speckle Type (eg. "Objects.Geometry.Mesh") as a string, to an ISpeckleTypeConverter.

During receive, we then use the Speckle Type of our object to get its converter using the TMap.

I think Unity has some similar issue with point clouds being dependent on a third party plugin. And the need to modify source code to extend conversion.

In Unity, these ISpeckleTypeConverter could be serialisable, and grouped into kits using ScriptableObjects.

What do you think (have I gone too far? is this a separate discussion?)

[haitheredavid (David Morgan)]

@Jedd ah you bring many good points! I agree that breaking things down into Type converters could lead to some handy solutions with unity and unreal. For unity we currently convert and pass things back as Gameobjects, but we could modify that to be component or mono based (i.e. you could pass back Unity.Mesh instead of gameobject with all the bells and whistles on it. I think any of the default converters for clients like unity or unreal should be as lean as possible, with little to no additional third-party plugins. I was thinking of pushing the point cloud object over to the particle system or storing all them in a computebuffer field that the vfx graph could play with. If down the road I plan to use a third-party plugin for something like point clouds, I would probably specify it for a singular unity project.

From my experience so far with building out different converters for my viewobjs, I’ve always incorporated a reference to a default converter (usually the main Object converter) in order to handle passing data conversions that already handled by the peeps like yourself on the Speckle Team. So I would be curious to know how you might consider a use case like mine being able to incorporate different component based converters?

[haitheredavid (David Morgan)]

made a little bit of progress on things that I thought would share.

@Jedd

In my (half baked) idea of modular converters, we would have Mesh, Camera, Blocks, Point-Cloud etc conversion, all in separate classes implementing an interface (eg. ISpeckleTypeConverter)

I thought this was a neat idea since converting objects into the unity world sometimes feels a bit strange. After hacking speckle to work the way I need it too, I came across two things I really wanted to improve for unity support.

1. Creating an easier way of relying on pre-written conversion logic

I’ve been finding that most of my converters rely a bit on the converters you lovely peeps have put together… Maybe it’s a bit lazy on my part, but I think it’s a bit silly to copy + any conversion code I might need for something like a mesh or a brep. For the unity converter I started making static methods that would return things I wanted converter (i.e. Geometry.Mesh to UnityEngine.Mesh) but it never really felt clean to do it that way.

2. Having a better understanding of what is actually returned in a conversion.

Por ejemplo, I’ll rely on the UnityConverter to pass a Geometry.Mesh to get a unity mesh, but what ends up being passed back is a GameObject where I’ll need to step into the object to grab the component MeshFilter and copy the mesh data off of it and trash the rest of the object. This gets a bit expensive over time, but not the end of the world.

In order to make these things come to life, I’ve been heavily relying on ScriptableObjects as a way to modularize code and make it more readable from the inspector. Now for the cooking!

Unity Converters in Speckle

The first thing I wanted to do was work off of the ISpeckleConverter and make it something that can be modified from the inspector and/or code. I created a scriptable object that inherits the converter interface and creates a handful of fields open in the inspector. What I like about this is any user can make a copy of this converter in unity, modify all the deets and parameters they want, without having to code anything.

This works pretty fine at the moment, but I think down the road some restructuring will be needed to help with inheriting from this type. For now it works

Component Converters

The second thing is to build out the actually converters. Similar to the main converter I wanted it to be modular and editable from code / editor, so scriptableobject we go! I started making a base class that would acts as the main structure for all converter based components.

ComponentConverter<TBase> : ScriptableObject where TBase : Base

There is one main method that all converters must handle, Component Process(TBase @base). In the previous converter each time an object was converter it was sent into some sort of UnityEngine.Component, attached to a new GameObject and potentially tagging along some other components too. This is a bit of my frustration with not knowing what type of game object was being returned to unity and having to rely on diving into the conversion code to know what to expect. With returning a Component I’m thinking that will help narrow down what to expect in return from a converter.

I threw together some basic converters by stripping out all the conversion logic from the main converter object and separated them into singular objects.

If you select on one of those converters you can get access to the main parameters utilized in that conversions. Similar to the main converter object, you can make a duplicated copy in the editor, modify the parameters and have that information serialized within your project. Here is the editor from the MeshComponentConverter

Some remaining thoughts!

@Jedd

I think Unity has some similar issue with point clouds being dependent on a third party plugin. And the need to modify source code to extend conversion.

I think with the structure I’ve been cooking up this will enable any unity peep to build out their own comp. converter, including any super-slick-plugins they desire, and simply adding the converter back in. A component converter could also be packaged and available in unity’s package manager to make it easier to plug and play.

[Jedd (Jedd Morgan)]

Hi David!

I really like the sound of this, it sounds like we are on the same page here!
Really impressed and excited!

How far along are you to getting something working/MVP? It sounds like you are there/pretty close!

I’ve been working on something similar to this for the Unreal Engine connector, surprising similarity to what you are describing here (smart minds must think alike!)

I have a few questions just to clarify I understand your approach, and to present some open problems I currently have on the Unreal side (to see if you have already found a solution)

1. Firstly, just want to double-check ComponentConverter<TBase> is an interface? or abstract class? that BaseConverter, CurveConverter etc inherit from?

2. Secondly, your component converters are ScriptableObjects, and instances of those live as assets in the project right? We would provide “default” instances in the Package folder (like I think I see in your screenshot)
   The user could, if they wanted to, create their own instances in their project folder?
   Would they need to do that to customise properties, like you have Store Props, Add Mesh Collider etc, or should they be able to edit the one the plugin provides (in the Packages folder)?
   I’m curious as to what you think would be the best design.

3. For ToNative conversion, how are you mapping a SpeckleObject → ComponentConverter (i.e. figureing out which converter to use),
   Are you using a switch statement somewhere, or a Dictionary, or something else?
   I assume this is being done in UnityConverter : ISpeckleConverter

   On the unreal side, we are using a Dictionary<TypeOf<Base>, ISpeckleTypeConverter> (in effect, it’s actually slightly different because C++)
   This has the advantage that users can add in what ever conversions they want (even dynamically at runtime if that’s a valid use case)
   But this opens some issues with how we deal with priority, custom behaviour, etc.

   Looking at your first screenshot, it looks like you’re just exposing some fields? Are these fields a concrete type like MeshConverter or are they ComponentConverter<Mesh>.

   3.1) If A user wanted to add a new type of object converter, They would have to create my own ISpeckleConverter class right, and add it as field?
   Mmm, that doesn’t actually sound that bad, while you lose some ability to customise conversion at runtime (a feature no one has asked for),
   It can make the structure a little more flexible/hackable. My method of using a Dictionary could be causing me more issues than it’s worth.

4. ComponentConverter<TBase>s returns a Component, not a GameObject right? This component is attached to a newly created GameObject right?, since a Component instance can’t exist in isolation.
   How does this work for converters where we want to add multiple components? (Like Mesh + Collider, I guess return the more obvious mesh component)

   Is the advantage of returning a Component not a GameObject to avoid having to do an expensive GetComponent operation? Would you still need to do a checked cast to make sure it’s the type you expect?
   Is there any advantage to perhaps making ComponentConverter<TNativeComponent, TBase> where TNativeComponent : Component, TBase : Base? maybe not??

5. What happens if one converter say a “Block” converter (or to keep it simpler, a Wall converter) want’s to use the Mesh converter’s ToNative method.
   How do you think it’s best for us to facilitate that?

   The approach so far I’ve taken in Unreal is to add a property to the Block converter’s UObject (eqiv of ScriptableObject) instance for a MeshConverter. So in effect, it will ignore what’s setup in the
   ISpeckleConverter but is still customisable by the user. I wondered if you have any different thoughts.

6. I (hopefully) understand all this for ToNative operations, what about ToSpeckle, does a ComponentConverter do both?
   What happens when we want to receive X, Y, and Z, but only send X and Z (I might just be describing filters here which could be a very separate issue)

Thank you immensely for the time you’ve put into this! I’m very excited for progress in this direction.

[haitheredavid (David Morgan)]

ah great questions and input @Jedd. Happy to see this structure is aligning with the previous conversations we were having, and that some of this content is lining up with things in unreal.

The component converters are working nicely with the rest of the project at the moment . I’m roughly halfway to having some sort of MVP with the converters, I’d mos-definitely need to add in the ToSpeckle logic.

On the reply !

1. ComponentConverter<Tbase> is currently an abstract class that that inherits ScriptableObject.

2. Exactly! Each component converter is a serialized object that is stored in the unity projects. I think it currently lives some place like /systems.speckle.connector.unity/Runtime/Converters/Components. A user does not need to create their own comp. converter to modify the parameters like AddCollider, but they are able to create an instance of any comp.converter if they wanted to store different layouts of parameters. It’s pretty easy to modify the parameters without creating a new one, but I imagine some users might like creating their own converters within the assets folder of their unity project.

3. I haven’t really figured out how the ToNative could work best. Currently I’m handling all the mapping in the ConverterUnity with the typical switch statement, but I feel like this prone towards errors down the road with trying to figure out what component converters needs to be relied on.
   The exposed fields are concrete types since unity does not allow serialized fields with generic types.

   I think I’m a bit confused on 3.1 Do you mean ComponentConverter or ISpeckleConverter ? For ISpeckleConverters, a user could create a new instance of SpeckleConverter in unity (another scriptable object!) and swap out any of the comp.converters through the inspector and/or code. For someone that wants to customize a converter to support different ISpeckleKits, they can extend SpeckleConverter and integrate any of the converters they’d like.

4. Ah this part is something I’m still a little stuck on. I’m not sure of any performance advantages of GameObject vs Component, but I thought that knowing whatever is being returned is of type Component you don’t need to call gameobject.GetComponent<MeshFilter>() to get whatever was converted. You still need to do a cast to check to get the component that base was converted thou.

   I don’t have any solutions that are solid for building off of other GameObjects, I’ve only thrown together some quick static extensions that allow for different conversion calls to build off an existing GameObject rather than creating a new one.

   But with your suggestion, I totally agree with setting up the converters to be more strictly typed. I’d like to make the comp.converter have one additional generic param that would require the type of unity component it converts to. Something like dis ComponentConverter<TBase, TComp> where TBase : Base where TComp : Component…Making it look very similar to what you’ve done in unreal.

5. I think we’re on the same page! If you wanted to create a block component converter you could declare a MeshComponentConverter as a public field and throw whatever mesh converter you want into that converter.

6. I’d like to add each comp.converter a similar method for processing components back to Speckle. I haven’t really considered any sort of filtering commands, but maybe you could throw those parameters as fields in each converter?

Thanks for the good chatter on this excited to push this along a bit more

[Jedd (Jedd Morgan)]

Cheers David for your reply!
Sounds great on all fronts!

---

The exposed fields are concrete types since unity does not allow serialized fields with generic types.

I believe starting with version 2020.1, this is no longer the case, as long as the type is known at compile time, so in theory, I’m thinking you could have a

[SerializeField]
private ComponentConverter<Mesh> meshConverter;

This might need some experimenting to see if this works the way we would want.
In theory then, this might offer the user some more flexibility for using their own conversion classes.
Do you see value in this level of abstraction and flexibility? Or do you see this use case better fulfilled by extending / creating a new class : ISpeckleConverter.
We would also have to assess how this would affect Unity 2018/2019.

---

To clarify 3.1, you have answered it perfectly , I was talking about ISpeckleConverter, and yea looks like a user would have to create a new ISpeckleConverter, which is fine.

My approach on the Unreal side has tried to minimise the need to create custom ISpeckleConverters
But honestly, doing it your way seems more than acceptable, it’s easier to wrap your head around, and if a user is going to the trouble to create their own conversions, it’s not too much extra hassle to create a ISpeckleConverter class too. They might even want the extra flexibility, instead of being locked into a class structure we dictate.

---

Making it look very similar to what you’ve done in unreal.

This is not actually how I’m doing it in SpeckleUnreal (since Unreal hates generics more than Unity does ), but I thought I’d just mention it as it popped into my head.

I’m thinking that this starts adding some extra unnecessary complexity which could lead to some complications. Especially with ToSpeckle, and if we implemented serialised ComponentConverter, there may be introducing some limitations (e.g. if a method/class doesn’t actually care about the type of Component being returned). So I would probably recommend against it, but feel free if you want to give it a go if you like.

---

I haven’t really considered any sort of filtering commands

I’d say don’t for now, this is a separate issue, one likely best solved by users to fit their own specific needs.

---

Let me know what you think, and we can discuss next steps!

[haitheredavid (David Morgan)]

woot woot, great chit chat

@Jedd

[SerializeField]
private ComponentConverter<Mesh> meshConverter;

pretty pump to have learned about that this, I did a quick test and you’re correct about this capability I’m not sure if there is any clear benefit from using the concrete type vs generic one, they would still be tied to the scriptable object type with both types. I’ll play around with it and see if there is anything that pops up. If this is plans for targeting earlier builds of untiy, maybe we should double check if you can declare fields in 2018/2019 the same way as 2020.

Just a quick note on the ISpeckleConverter thing. Users can actually create a new SpeckleConverter scriptable obj within the unity editor and decorate it with the desired comp.converters. My goal with this was to make setting up a custom ISpeckleConverter in unity ez. The only reason to write out a new type of converter via code would be to add custom object types.

Something I’m not really familiar with is the ApplicationPlaceHolder objects. What are these? Do you have an example of how you used them in ue? The only time I’ve seen some sort of usage with them is with retrieving materials. I imagine this might be something that would benefit from being exposed in the editor.

@Jedd

This is not actually how I’m doing it in SpeckleUnreal (since Unreal hates generics more than Unity does ), but I thought I’d just mention it as it popped into my head.

i feels for you now

Nexium Steps!

Quick off the ideas

• I’ll implement the second generic parameter for the speckle component converters along with a method for handling that logic.
• Build out the ToSpeckle methods within the component converters
• Iron out how the Unity Connector should know about different converters.
• On a different note, I think there is some work to be done on the sender/receiver objects. I have a few different sketches built out in my view-analysis tool that plays with how unity should interact with the Operations class.

[Jedd (Jedd Morgan)]

Hi David,

@haitheredavid

Something I’m not really familiar with is the ApplicationPlaceHolder objects. What are these?

Good question, Effectively they are a Base wrapper around a native object.
I had to double-check how we are using them in Unity. I think the only place we are using them is when we load Unity Materials from assets, they are added to a ContextObjects list.
In this case, I think we are just using it to store materials. Not sure I fully understand the benefit of doing it this way? I’d be open to suggestions on improvements (if you were interested). @teocomi might be able to chip in if I’ve completely misunderstood.

In Unreal, the way we override materials needs some improvement. Currently, I’m just exposing a serialised array property to the editor the user can drag and drop materials they want to override by name, or by speckle id.
But I imagine some users will want to automatically load materials from assets.

Keep me updated on how you get on, and don’t hesitate to come to me with any problems.

[teocomi (Matteo Cominetti)]

@haitheredavid

ApplicationPlaceHolder

These objects were originally introduced in Revit as a way to store the objects received and converted so that upon receiving a second time we can easily check if they need to be updated or not.

See sample usage here: https://github.com/specklesystems/speckle-sharp/blob/main/Objects/Converters/ConverterRevit/ConverterRevitShared/Partial%20Classes/ConvertWall.cs#L22