I ended up sending seperate layers to separate branches in my Speckle project, and then control how stuff is displayed by federating models in the Speckle viewer. Here is a public Speckle project with lots of Ground Investigation data that I sent to Speckle from QGIS: