I’m working on trying to add WebXR support to an embedded Speckle Viewer through the Viewer API. How cool would it be to be able to place a speckle model onto a table or view in an XR headset. Unfortunately, I haven’t had much luck.
I have made several simple Three.js viewer (without speckle) that work well in WebXR but when I try to apply the same logic to the Speckle Viewer by hijacking the renderer, I have struggled to get it to render in XR and I also have struggled to add my custom XR interface to the scene graph.
I know that my issues are most likely stemming from a poor understanding of the viewer’s batching system and the use of effect composers to combine visual passes instead of a traditional three.js render pass.
As I see it, I’ve got two options: either I continue to try to fight the viewer’s complex pipeline/batcher/renderTree/worldTree stuff or I figure out how to extract the fully batched three.js scene out of the viewer and load it into my simple WebXR three.js viewer.
Anyone have any advice or insight on how I might accomplish this?
It seems that three.js has some builtin webxr related stuff, but we can still hijack rendering if that’s your preferred approach
The good news is that you don’t need to worry about batching, as that’s made to happen automatically when loading stuff in. The result of the batching is still a SpeckleMesh or SpeckleInstancedMesh which is basically still just another object in three’s Scene which you can access via viewer.getRenderer().scene like Jonathon mentioned.
We also ditched three’s EffectComposer relatively recently as I personally never liked it and believe it offers a poor abstraction over that a render pass is, and what it can do. We now have Pipelines and a few types of Passes which are light wrappers around three’s core rendering functionalities.
I’m sorry to hear you had to fight the speckle viewer, and I can totally relate. I went through the same struggle countless times when using all sorts of rendering frameworks/engines. That’s why I really hope I can be of help and transform the struggle into something more pleasant and worthwhile!
So please let us know in more detail what are the current roadblocks you are facing and I’m sure we can work them out.
I’m continuing to beat on this today. The approach today has been to try to take the speckle scene and render it in a separate THREE.WebGLRenderer. I’ve striped everything down to the nuts and bolts and still can’t get the speckle scene to render in any context/renderer outside of the speckle viewer.
Could this be caused by the RTE (relative to eye) implementation that messes with the Geometry Buffers? I see that it’s doing some fancy position_high & position_low attributes to get double precision. but I’m not sure if I need to recreate/mimic the RTE renderer for my renderer or if I need to “bake” the geometry back into some more standard vertex model?
I think you are trying to take the long way around
What’s stopping you from rendering everything in the speckle viewer’s webgl context? I’d be more than glad to provide you with a live code example for anything that you might find that is not working.
Additionally, you can tell us what did you tried first and couldn’t get working, or perhaps things are missing considering your regular WebXR approaches.
We did branch off from how three.js typically works, however we did try our best to maintain parity between the speckle viewer and what people are usually used to when working with three.js.
We’d really like to help, but it’s very difficult unless you tell us in bit more detail what you need help with.
PS: Unless you initialize the other WebGLRenderer with the same webgl context from the speckle viewer, it will not work. The VBO’s and all other resources are valid only for the webgl context used when creating them, that is the speckle viewer’s. They are invalid in other contexts. WebGLRenderer allows you to pass an existing webgl context when creating it. Maybe try it like that? However it still feels like not the best approach and I’m sure we could figure something out more straightforward
I apologize for not providing more detailed information. I’m sort of learning how the viewer works as I’m going so I don’t really know the right way to word some of my questions.
You might be right that I should go back to trying to add XR support to the viewer’s SpeckleWebGLRenderer instead of trying to add the speckle scene to a different SpeckleWebGLRenderer. It’s just hard to tell if I’m going to be able to get the RTE view stuff to work within the WebXR mode. Right now I have tried enabling XR on the SpeckleWebGLRenderer and it loads an immersive XR window but the model is not visible. Maybe I should go back to trying to solve that. That said, I’m also a little worried that it looks like the speckle viewer is using an older r140 version of three.js that might not have all the WebXR support that I’d want (not sure if that’s true).
One small update on the scene snatching path: I was able to get part of the speckle scene to show in my WebGLRenderer by running this before rendering: this.speckleWebGLRenderer.updateRTEViewModel(this.speckleCamera).
You can see that my viewer is rendering one of the two speckle models that I’m using for testing. It’s not perfect but It’s at least proof that the geometry exists and can be displayed by my viewer. I’m sure the other missing model and the orientation issues are due to my viewer using the out-of-the-box Orbit Controls to move the camera.
I am changing course to try to make the SpeckleWebGLRenderer provide the WebXR session. unfortunately, I see that the speckle viewer uses RequestAnimationFrame instead of setAnimationLoop. According to the three.js docs. setAnimationLoop is required for WebXR:
That shouldn’t be much of a problem. Let me give you a hand with the SpeckleWebGLRenderer part. I’ll make something for you that exposes a working WebXR session. Hopefully
I managed to get a basic setup working. I just need to wrap it up for you to use. We might release an alpha version of the viewer library for this. I’ll keep you posted
