Speckle as sole database. Discussion on limitations

Hello,

I’m not 100% sure this is the right channel, but let’s give it a whirl. Just to preface this I’m a developer and not an experienced Speckle user. So feel free to answer as if to a complete newbie.

Me and my team are currently working on an application which in short does the following:

  • Bring multiple FEM models together
  • Enrich the data
  • Run some calculations
  • Allow users to explore results
    All of this is done via a web application which is pretty feature dense.

While brainstorming our software architecture the idea came up to not use a traditional backend ↔ database structure but have Speckle serving as our only data source. This would mean that it had to contain not only the FEM models but also all of our application specific data. This in itself seems possible as we can have our arbitrary data as part of the models, but this seems to be stretching the limits of what Speckle is supposed to do.

Some of the limitations I’ve run in into up until now are:

  • Not being able to hang arbitrary (meta-)data to a project itself, only the underlying models.
  • The structure purposed would require multiple FEM models to share a single Speckle model stream. This might lead to large amount of data duplication and I/O
  • If a user already has Speckle projects because they use Speckle for multiple applications we need some way of designating which projects do and do not belong to our database (this hooks into the metadata discussion as well)
  • We would likely still need a database for anything which is user specific such as settings or authorization

Does anyone here have experience with using Speckle to store application data or opinions on the workflow I’ve tried to describe here. I can give more details if needed, but I’m trying not the info dump unneeded details.

Any insight or discussion would be greatly appreciated!

1 Like

Hi Ben, that’s one to get the brain juices flowing at the start of the week:

Five main patterns for building your FEM app on Speckle - ALL SPECULATIVE off the top of my head but flexible:

  1. A Config model that holds all your app’s configuration and registry data in a single place.
  2. Using Speckle as your sole database, where every piece of data —geometry, results, and even user settings —lives in versioned Speckle objects.
  3. A hybrid approach that combines Speckle’s 3d data store with an external database for relational or user-scoped data.
  4. Fingerprinting your Speckle projects to mark exactly which ones belong to your FEM workflow.
  5. A grab-bag of additional patterns, from remote references to Automate workflows and multi-model composition.

1. “Project Config” Model for App-Wide Key ↔ Dict Data

Rather than scattering app-wide settings and stream references across multiple places, you create a dedicated “Config” model. In practice, you publish a single object tree to the Config model. Then, whenever your web app starts up, it simply fetches the latest version of config to learn your UI theme, concurrency limits, any feature flags, and the list of FEM project IDs you’ve onboarded. This keeps all your key→dict data in one versioned, centrally queryable location.

globals = {
  registry: [
    { projectId: "abc123", purpose: "FEM", label: "Tower Analysis" },
    …
  ]
}

COMMIT globals AS latest Version TO Globals/main

Your app fetches the latest Version from Globals/main to read configuration, feature flags, and registry data.


2. Speckle as Your Sole Database

If you want a single source of truth, you can store everything in Speckle. You carve out separate models for each domain, one for FEM geometry, one for user profiles, one for config and you commit JSON-like “Base” trees to versions in those models. Real-time GraphQL subscriptions or webhooks drive your UI updates, and Speckle’s content-addressable storage automatically deduplicates identical objects. Stroing indecies in you data model can aid GraphQl queries down the line in entirely bespoke ways.

Store everything in Speckle—geometry, analysis inputs, results, and even user-scoped settings:

  • Models for each domain (e.g. models, users, config)
  • Versions contain arbitrary JSON-like objects (“Base” trees)
  • Subscriptions drive real-time UI updates when new Versions arrive

Pros

  • Single source of truth, full version history
  • Automatic deduplication of identical objects

Cons & Mitigations

  • Ad-hoc filtering/joins can be slower—embed simple indices in your objects or rely on GraphQL filters
  • Transactional or truly private data may be better suited to a dedicated database

3. Speckle as Data Store + Augmented Database

In a hybrid pattern, Speckle remains the home of versioned 3D and analysis data, but you layer on a lightweight external database, Firebase, PostgreSQL, MongoDB, for anything relational or user-scoped (permissions, audit logs, indexed result summaries). On each new version in Speckle (detected via webhook or subscription), your analysis service pulls the model, runs calculations, and writes results to the external DB keyed by projectId + versionId. The client then embeds the Speckle Viewer or SDK to load geometry and separately calls your DB API to fetch the analysis outputs, overlaying charts or annotations on the 3D scene. This gives you the best of both worlds: Speckle’s rich data exchange plus your database’s fast filtering and joins.

  1. Speckle holds versioned 3D and analysis data.
  2. External DB holds relational or user-scoped data (profiles, permissions, indexed result summaries).
  3. On each new Version in Speckle (via webhook or subscription), trigger your analysis service to:
  • Pull the new data
  • Run calculations
  • Write results into the external DB, keyed by projectId + versionId
  1. In your web app:
  • Load the model via the embedded Speckle Viewer or SDK
  • Fetch analysis results from your DB API
  • Overlay charts or annotations in the client

This keeps Speckle focused on shareable 3D data while leveraging the database for fast filtering and joins.

4. Fingerprinting Projects for Your FEM App

To mark which Speckle projects are part of your FEM app, you have three light-touch options:

  • Registry Versions
    In a dedicated APP_REGISTRY/main model you commit an object with fields like appFingerprint: "MyFEM-App-v1", the target projectId, and timestamp. Your app queries that model for versions where appFingerprint == "MyFEM-App-v1" to discover exactly which projects to load.
  • Tags on Projects
    Simply assign tags such as ["fem-app","v1"] to each target project. A GraphQL filter on projects by tag “fem-app” instantly yields your app’s streams.
  • Naming Convention
    Enforce a prefix on project names (e.g. femapp-bridge, femapp-tower) and query for projects whose names start with that prefix.

All three keep your fingerprint data in Speckle’s data layer (versions or tags), avoiding the pitfalls of using the user-editable description field.

  1. Registry Version

    registryEntry = {
      appFingerprint: "MyFEM-App-v1",
      projectId:      "def456",
      modelName:      "main",
      createdAt:      "2025-04-28T12:34:00Z"
    }
    
    COMMIT registryEntry AS new Version TO APP_REGISTRY/main
    

    Then query all Versions in APP_REGISTRY/main where appFingerprint == "MyFEM-App-v1" to list registered projects.

  2. Tags on Projects

    SET tags = ["fem-app","v1"] ON each target project
    

Then filter projects by tag “fem-app” to discover your app’s streams.

  1. Naming Convention
  • Prefix project names with femapp- (e.g. femapp-bridge, femapp-tower)
  • Filter projects whose names start with that prefix

Avoid stuffing JSON into the description field—users can edit it, and it may be truncated. Versions and tags live safely in the data layer.

5. Other Patterns & Options

  • Remote References / Detachment
    Reference shared sub-objects across projects to avoid re-uploading large meshes.
  • Version Messages or Dedicated Mesage Models
    Embed small structured metadata in commit messages or a special branch for annotations.
  • Speckle Automate
    Watch for new Versions, run serverless or containerised calculations, and write results back as child Versions or metadata.
  • Multi-Model Composition
    Use a single project with multiple Models (e.g. “passA”, “passB”), then a “master” Model that references each pass via detached references.

indeed im quite exceited to see if we can work with you on getting something scaffolded here - a web application that is polling ofor automate results is definitely on my agenda…

cc: @chuck

Hi jonathan,

Thanks for your thorough reply. I hadn’t considered splitting the data over multiple (tagged) models/submodels. It makes sense in that the models can follow the natural seems in the data model and aid in decoupling in the same way that database table can. I think I’m going to give that a try and see what I can throw together as a PoC.
I have one follow-up question (for now :wink:). I was looking through the GQL explorer and I couldn’t see any reference to the tags you mentioned. Am I missing something? Otherwise I’ll handle it via naming conventions, but I’d prefer using tags.

Oh that is 1000% pseudocode! :wink:

arbitrary data can still be queried and filtered in GQL

Really? Then I’m definitely missing something. I’m getting:

"message": "Cannot query field \"tags\" on type \"Project\".",
 "extensions": 
  {
      "code": "GRAPHQL_VALIDATION_FAILED"
  }

Seems to be linked to a “Project” type outside of my system. Any values from the explorer work fine, just can’t add or query anything outside of those.
I am working off a modified template from an old project with some internal dependencies. So if you’re sure then it’s probably something to do with those.

Hey just to chime in, as someone who’s done what you’re doing -

You can use Speckle as the database for your entire application, but (as you’re already seeing) I’d make sure that you should! The Speckle API is designed with 3D data upload/retrieval in mind. If you’re storing user metadata or configuration only, it will start to feel slow/silly very quickly. (Especially if you need the database for authentication.)

These days I’d put most things in Speckle, then manage my own lightweight tables for app-specific things like user metadata. If you’re already running Speckle, you have access to postgres for this. If you’re trying to lean on the production server, there are many options available for simple databases.

1 Like

@jonathon @chuck
So I’ve been playing around and my plan for now is the following:

  • Use the Speckle authentication token as a login
  • Use the Project<>Model<>Sub-model structure with naming conventions for data which can be linked to a (set of) fem models
  • Use an application project with models for application data
  • (Probably) use a separate database for any user settings and authorization.
  • Isolate any code which handles transactions between our objects/services and these new Speckle models. That way if at any point the structure becomes cumbersome or slow (or just weird) we can migrate parts of the data structure over to a designated database with minimal code changes. The application models should map nicely 1-on-1 to new db tables and to our frontends local storage
  • See where this design takes us and maybe get a good blog out of it :sweat_smile:

Thanks for all your insights, it really helped me get my head around the possibilities. If you’re interested I’ll post an update later along the line when we see where it all lands.

1 Like

Looks good, and good luck! Please do keep us posted, always interested in what people are building with Speckle and how they’re pulling it off