as part of an (architecture) university project I’m currently exploring the possibilities to use Speckle in iOS AR Apps.
I’m in the beginning of research but would really appreciate some hints where to start reading.
What should be supported is fetching streams during runtime.
For now, im looking into three ways, using Unity, Unreal Engine, or native Reality Kit which I’d prefer.
Unity/Unreal Engine
I see there are connectors for Unity/Unreal Engine. As I have no experience with those platforms, I don’t know if they fulfill the requirements of fetching content on runtime. Maybe someone can answer this?
Reality Kit
As there’s no connector available, this would require using the API. Dealing with geometry could be a challenge but I’ll habe to dive deeper into that.
Has there been any effort yet in this direction?
Welcome to the community! Feel free to Introduce yourself 🙆 to the community if you want to!
I’m not an expert on Unreal/Unity, so maybe @Jedd can jump in on that one
To use Reality Kit, you are correct, you’d have to deal with all the calls through the API. As far as I’m aware there hasn’t been any efforts on iOS integration as of yet (but we’d welcome a community connector for iOS )
Dealing with geometry can be a pain for more complex geometry types like curves, surfaces or breps. But for all those types we provide fallback meshes for programs that don’t support them.
I’m assuming all AR stuff is actually done with meshes/polylines, so you could skip most of the complexity by just using the fallback values whenever you receive anything that is not a mesh or a polyline (this is what our online viewer does)
If you’d just want Receiving capabilities in your app, it should be fairly straightforward to implement
You can have a look at our Dev docs (Introduction | Speckle Docs) and feel free to reach out for any questions you may have
There’s also a bunch of web tech that should be cross platform, and threejs ready for webxr/webar. So you could try to build on top of the existing viewer… See some discussions here:
Our Speckle-Unity connector fully supports receiving and sending streams at runtime.
And Speckle-Unreal also supports runtime receiving (though not sending yet)
As for iOS, I haven’t tested either connector for iOS, but is something we would be looking to support going forward.
Hopefully this is something I will get around to investigating this week. I’ll get back to you with my findings.
I started playing around with Reality Kit and fetching objects via the REST API. For my purposes, meshes are sufficient at the moment, so I thought conversion from Speckle Meshes to Reality Kit meshes would be all I need, and I got it working already.
But I realized that for example a box object, created in Grasshopper via box2pt comes without fallback mesh.
I know converting a simple box isn’t a lot of trouble, just wanted some clarification where to expect meshes and where not.
We definitively have some work to do around cleaning up our displayMesh / displayValue properties and make sure they are as consistently sent across all our connectors and conversions as possible. All non-curve geometry should have a fallback mesh, we’ll get around to cleaning that up when we can!
Issue is tracked here, will add a specific note to check our Box conversions
)