as part of an (architecture) university project I’m currently exploring the possibilities to use Speckle in iOS AR Apps.
I’m in the beginning of research but would really appreciate some hints where to start reading.
What should be supported is fetching streams during runtime.
For now, im looking into three ways, using Unity, Unreal Engine, or native Reality Kit which I’d prefer.
I see there are connectors for Unity/Unreal Engine. As I have no experience with those platforms, I don’t know if they fulfill the requirements of fetching content on runtime. Maybe someone can answer this?
As there’s no connector available, this would require using the API. Dealing with geometry could be a challenge but I’ll habe to dive deeper into that.
Has there been any effort yet in this direction?
I’m not an expert on Unreal/Unity, so maybe @Jedd can jump in on that one
To use Reality Kit, you are correct, you’d have to deal with all the calls through the API. As far as I’m aware there hasn’t been any efforts on iOS integration as of yet (but we’d welcome a community connector for iOS )
Dealing with geometry can be a pain for more complex geometry types like curves, surfaces or breps. But for all those types we provide fallback meshes for programs that don’t support them.
I’m assuming all AR stuff is actually done with meshes/polylines, so you could skip most of the complexity by just using the fallback values whenever you receive anything that is not a mesh or a polyline (this is what our online viewer does)
If you’d just want Receiving capabilities in your app, it should be fairly straightforward to implement
I started playing around with Reality Kit and fetching objects via the REST API. For my purposes, meshes are sufficient at the moment, so I thought conversion from Speckle Meshes to Reality Kit meshes would be all I need, and I got it working already.
But I realized that for example a box object, created in Grasshopper via box2pt comes without fallback mesh.
I know converting a simple box isn’t a lot of trouble, just wanted some clarification where to expect meshes and where not.
We definitively have some work to do around cleaning up our displayMesh / displayValue properties and make sure they are as consistently sent across all our connectors and conversions as possible. All non-curve geometry should have a fallback mesh, we’ll get around to cleaning that up when we can!
Issue is tracked here, will add a specific note to check our Box conversions