hey hi I’m on the hunt for some info around model size limitation when sending data using specklepy. I’m pulling data down, applying a dynamic string member to some objects, and then trying to ship that data back up to speckle when I get an exception that batch is too big for the local db.
model = receive(obj_id, transport)
send(model, [transport])
>>> SpeckleException: Could not save the batch of objects to the local db. Inner exception: string or blob too big
Besides begging for more power, how would you recommend going about this limitation? I’ve done a lil work around for now by stripping the objects out and pushing them up as a separate version. I remember from v1 that we could bump up the model size limitation with a setting in a custom server config, but that was needed to push models from any connector or sdk. Is there a similar setting in v3? Would this exception get thrown if we moved the script into an automation function?
Glad to hear your still finding ways to break Speckle
The error your seeing is coming from the SQLite transport, not from the server. From the looks of it, SpecklePy is setup with a very tight 10MB default max object size .
This was a bit of a surprise to me, because C# SDK has had a max object size of 100MB limit for a long while now, I guess we forgot to update SpecklePy .
I’ll look into making 100MB the new default.
But in the mean time, you should be able explicitly setup an SQLiteTransport with a custom max_batch_size_mb:
Bumping the size up didn’t seem to resolve exception being thrown. The model is a bit chunky, around 1228.8 mb, and I pushed the local transport to 2048 but it didn’t seem to do the trick, I even set it to 5120 but no luck so far. I made sure not to use the local default cache and passed that same transport into both operations. I’m using version 3.10 of specklepy and testing the same command on v3 and v2 models.
hey there more questions on model size limitations.
I’ve been getting an error thrown from specklepy when I try to send a large model back to the server that exceeds the maximum object size of 100mb. The models are first sent to speckle through the ACC connector. Receiving the model works as expected but if I try to send that exact same model back up to speckle it throws an error saying the model is too large to send to Speckle. I modified the max cache size for the local transport but I’m not sure if the local transport impacts the final object size sent to the server. Currently on specklepy 3.2.3
Is it possible to modify the maximum model size sent from specklepy? I imagine there is some checkpoint in the sdk that is triggering this warning since the exact model is able to flow through the ACC connector.
Happy to provide any additional context, models, or sub-par jokes
Just so I understand. The Speckle Model in question first comes from an Revit ACC import.
and that works fine.
And you’re able to receive that model in SpecklePy, but trying to send the data again via SpecklePy throws this 100MB exception?
The 100MB object limit is the same in SpecklePy as it is for the C# SDK that powers the revit ACC connection.
So somehow objects are getting larger by going through SpecklePy.
I would suggest debugging your python app to find which object is too large. From there we can inspect the JSON and try and find what is inflating the size. My suspicion is missing object models mean that certain properties are not being detached when they should be.
Yup, that’s what’s happening! I’ll do a bit of debugging with the model and see what I can discover. Does specklepy have any utility functions for measuring object size?
If the object does get successfully sent to Speckle, you can use the https://app.speckle.systems/objects/{projectId}/{objectId}/single endpoint to fetch the raw json object. You can then right click and save as a file where you can then easily see the file size.
You could also try and get your IDE to catch the exception as it’s occurring on send, then you can look for the JSON strings it was trying to send and see if you spot anything out of the ordinary.
You should check for any list of Base objects (like elements lists) that are not detached, likely those should be.
If you manage to get a smaller send that goes through, and share the link with me I can also have a scan through and see if I can spot anything out of the ordinary.
Got some logs to share with ya @Jedd. I captured the size of a few different objects using sys.getsizeof()
single obj=3,023,886 # from the objects endpoint
ops model = 452,637,702 # from operations.receive
deserialized json= 452,637,702 # from serializer.read_json
traversed base=417,550,167 # from serializer.traverse_base
serialized json=114,657,284 # from serializer.write_json
Not really sure if this gives any hints but I figured I would share anyways. Now I’m going to step through the base objects and see if I can spot anything that you mentioned.
If the users code doesn’t reference critical types like Collection, then those types won’t be added to _RegisteredBase’s type registry, and therefore won’t be available when deserializing.
This was causing everything to be deserialized as Base (silent fallback) and therefore lacking the detaching information that Collection has. Thus everything gets sent as one big Base object that was >100MB.
@dogukan and I have created a fix, which is now released in specklepy version 3.2.6
Thanks @haitheredavid for helping us debug this, and for being patient with us.
I’m on the hunt for some info around model size limitation when sending data using 
.
.
Ill add that transport and give it a go when I’m back at my computer.
more questions on model size limitations.
