HI Everyone, thanks to the great work by the Speckle team for opensourcing and the useful docs. I am new to databasing and was reading through the Base Object article to understand how Speckle stores its data. From what I understand is that objects are serialized and stored as Blobs in the DB along with a unique hash id. So the schema can be thought of as two columns; ID + blob column?
My question is: if you are looking for an item in the DB, do you need to deserialize the object to get access to its properties? Doesnt this make searching very slow as you need to deserialize everything? I would like to read more about this way of setting up the DB, so if theres a technical term for this method or any resources, please share!
I have definite good news for you. You are half correct, AND there is a less expensive way to query objects than you imagine.
At its core, a Speckle Base object is a data structure formed of properties and some hints at the schema it belongs to. This is serialized as JSON, and the hash of that object and all its detached properties is stored in the database with that as an ID. Other things are at play in each record, metadata or createdDate, amongst others, but instead of a binary Blob in the strictest sense, the JSON itself is stored as a Record in the DB.
The most straightforward means of querying the stored data is found in the GraphQL documentation;
…the server will interpret your query, perform the search and filtering directly at the DB level, and serve the matching objects. From there, you can deserialize and reattach only the objects relevant to your query.
I’m working on expanding the examples we already have in the documentation for filtering for desired objects, but that is the best way to look for Speckle data.
It is technically possible to craft queries directly at the DB records if you are self-hosting Speckle, but I wouldn’t recommend it.
Thanks alot Jonathon for the detailed response. I imagine the GraphQL is setup to internally search within objects’ JSON to respond to your query? I will need to learn more about this and play around with the explorer you shared.