Virtual/Augmented Reality

Features & Ideas
1

I am currently trying to explore methods that could be used to visualize a model that is on a speckle server in VR.

I have looked around this forum a bit and found people mentioning that they are working on unity/unreal based viewers in private, but not many details.

I have also tried to do a quick&dirty patch of the speckle server threejs based webviewer to switch it to VR mode (since threejs already supports WebXR), but I didn’t get it to actually render something when in VR mode. But I haven’t spent much time on it yet and I assume bigger projects would just be too much to handle for e.g. a standalone VR device like a Quest 2 or Pico 4 (though I have read about the ifcjs project working on a method to only stream parts of a model to a mobile device).

So I thought before I start reinventing the wheel I would ask what the current state of this topic is and what experiments people have done so far/what experiences they had and what ideas are floating around out there.

1 Like
2

Hey @mp42 have you looked at our Unity & Unreal connectors?

@Jedd demoed how they can be used to create VR experiences not too long ago, see the video below:

https://drive.google.com/file/d/1ls5zovY15Bxhis9e0SgxjKp9k_rLsxzE/view?usp=sharing

2 Likes
3

Thank you for the video clip. That looks promising.
I have seen the connectors but thought maybe somebody has already shared a public demo project :wink:
I will play around with the Unity connector and see how far I can get.

1 Like
4

Hi @teocomi and @Jedd , I came across this post while working on an AR project, and this demo video is exactly what I wanted to implement!

Is it possible for the Speckle team to share the codebase and/or build configurations used for this demo video?
I tried using the assets & scripts in SpecklePlayground here, to build a project to Meta Quest Pro, but with the different target architecture and Unity version, I could not get it to work. I am also interested in which VR device Jedd used if you can share.

It’s amazing to see how many things you can do with Speckle!
Also thank you @mp42 for starting this conversation.

1 Like
5

Hi @haileyArup

I was using the quest 2, but using pc vr via link (not running natively on the quest 2) .

I was intending on uploading the aforementioned project source code somewhere public. I’ll try and get that sorted pretty soon. You should be able to run it with any pc openxr device.

I did some other experiments building natively for the quest2, and it can be done,
but in order to actually receive/send objects to Speckle, there’s some extra loops to run through, and some of extra legwork you’ll need to do for accounts loading, etc. see Unity Android build

3 Likes
6

Hi @Jedd, thank you so much for your response! It sounds very promising. Looking forward to seeing more of Speckle use in the VR world !!

1 Like
7

Hi, I wonder if there are any examples or attempts to use Hololens or other AR devices?

1 Like
8

Hi @Jedd I was wondering if you published the project and if you have any recommendations to build Speckle natively to a quest pro or a hololens.
Thanks!

9

I’ve uploaded my VR sample here.

Although, in its current form it only works for pc vr, and uses an older version of the Speckle Connector, so likely will have problems receiving commits from anything 2.12 or newer.

I hope to pick this project back up again over the next month and polish it into something a little more complete, and compatible with the latest version of speckle (which, by then, will be 2.14/15)

Thanks for your patience.

As far as building the Unity connector for Android based VR (e.g. Quest 2, Quest Pro, Pico Neo
I have briefly tested building natively for the Quest 2, with some hoops to jump through, it is possible.

see Unity Android build - #9 by Jedd
But the TLDR the connector will build for Android arm 100% fine, but you need to jump through a couple hoops to actually receive/send anything.

Specifically, the SQLite transport (used both for our local object caching, and loading accounts) will not work. (Which the send/receive components rely on)

So you will need to write your own Unity component that call the Operation.Receive and/or Operations.Send manually, where you will need to

  1. Explicitly avoid using the local SQLite transport (Either by passing a MemoryTransport instead, or by explicitly not using any local transports)
    and
  2. Provide an Account object, with an auth token (aka personal access token )

Have a look at the ManualReceiveExample.cs, this is a minimal example of what you need to do.
And this example can actually be used as is, and works on Quest2!
However, of course, in this case, I’m hardcoding an auth token via the editor. If you require a user to login to their own account. You’ll need some way for a user to enter their auth token during the runtime of the application. This might be a sticking point, so let me know if we can support something better.

As far as HoloLens’ goes, I have little experience with that ecosystem, and to my knowledge, our connector has never been tested on that platform.
I have confirmed there are no build errors building for that platform, but unfortunately, I am unable to test running. There’s a good chance it will work, of course, with the same limitations as for Android, and if it doesn’t, I’m happy to assist trying to getting it working, however right now, I can’t make any promises.

2 Likes
10

Thank you! We can let you know if the build succeeds.

We are very excited about integrating user workflows from Grasshopper into AR to enable direct manipulation interactions.

1 Like
11

Hello, I’m working with Ana on using speckle in ar/vr environments. I just tried building your speckle-unity-vr project to hololens 2 on Universal Windows Platform and received a build error “Assets\Scripts\Speckle Helpers\VRReceiver.cs(58,37): error CS0103: The name ‘ObjectFactory’ does not exist in the current context”
Do you know what might have caused that/how to resolve it?

2 Likes
12

Just to follow up, I received the same error when trying to build your project without changes to a meta quest pro.

13

Thanks for your patience.

Seems I had an editor dependency that was preventing standalone builds. :sweat_smile:

I’ve just pushed a quick fix, and also bumped the version of the speckle connector up, which should fix any additional build problems you might have encountered.

Additionally, in order to build for openGL platforms, you may need to switch from Gamma to Linear color space. If you get a build warning, it’s a quick fix.

image
image1400×900 96.6 KB

However, I still don’t expect this VR sample to run out of the box on non-pc environments, due to the aforementioned account loading issue.

3 Likes
14

Thanks so much, I’ll test it out and let you know how it goes!

1 Like
15

Hi @Joyce_Passananti, were you able to deploy this to a HoloLens?