Batch size limit hit when sending from specklepy

Speckle Help Developers
, ,
1

hey hi :speckle: :front_facing_baby_chick: I’m on the hunt for some info around model size limitation when sending data using specklepy. I’m pulling data down, applying a dynamic string member to some objects, and then trying to ship that data back up to speckle when I get an exception that batch is too big for the local db.

model = receive(obj_id, transport)    
send(model, [transport])
>>> SpeckleException: Could not save the batch of objects to the local db. Inner exception: string or blob too big

Besides begging for more power, how would you recommend going about this limitation? I’ve done a lil work around for now by stripping the objects out and pushing them up as a separate version. I remember from v1 that we could bump up the model size limitation with a setting in a custom server config, but that was needed to push models from any connector or sdk. Is there a similar setting in v3? Would this exception get thrown if we moved the script into an automation function?

2

Hi @haitheredavid

Glad to hear your still finding ways to break Speckle :grin:

The error your seeing is coming from the SQLite transport, not from the server. From the looks of it, SpecklePy is setup with a very tight 10MB default max object size :scream:.

This was a bit of a surprise to me, because C# SDK has had a max object size of 100MB limit for a long while now, I guess we forgot to update SpecklePy :sweat_smile:.

I’ll look into making 100MB the new default.
But in the mean time, you should be able explicitly setup an SQLiteTransport with a custom max_batch_size_mb:

local_cache = SQLiteTransport(max_batch_size_mb = 100.0)

model = receive(object_id, transport, local_cache)
send(model, [local_cache, transport], false)

Please let me know if this resolves your issue.

1 Like
3

Can my “contributor” tag be changed to “breaker”? Makes me feel like I’m trying to learn to breakdance again :man_dancing:

Thanks for the quick response my speckle brother! :folded_hands:t2: Ill add that transport and give it a go when I’m back at my computer.

4

Bumping the size up didn’t seem to resolve exception being thrown. The model is a bit chunky, around 1228.8 mb, and I pushed the local transport to 2048 but it didn’t seem to do the trick, I even set it to 5120 but no luck so far. I made sure not to use the local default cache and passed that same transport into both operations. I’m using version 3.10 of specklepy and testing the same command on v3 and v2 models.

Let me know how I keep breaking shit :melting_face: