Hi @gergo,
Thanks for looking into our data, it’s much appreciated!
First of all, your fix to allow dynamic class creation works and would enable us to upgrade to the latest version of specklepy, so we’re looking forward to have that released : )
I believe we use pydantic for class creation as your Base model used to inherit from the PydanticBase originally. At that point, pydantic was needed to create classes dynamically. This indeed also enforces type checking, which is a nice addition imo. I discussed this a few years ago with Izzy, see the discussion here: 🐍 PySpeckle 2.0 is starting to take shape...be an early tester! - #6 by Rob
So essentially it evolved into using pydantic and it’s not so much a deliberate choice. However, the fact that it performs type checking is definitely an advantage that’s worth keeping it as is, though it’s good to know that there are some caveats. We can transition to using the types option to generate the classes when we would bump into other issues.
Regarding serialization, I can definitely understand that our data structure is far more nested and cross-referenced than you generally intend, and I agree that this inflates the issue to a large extent. So it makes total sense you’re not looking into optimizing serialization for these more nested data structures. However, I do think that the root cause is the redundant, repeated serialization, which is not just an issue for our data structure. Also for data structures that adhere more to your intended structure, this will cause some delays, though of course less severe than for our data. Therefore I think that the proposed change to store already serialized objects would be of added value for any data structure. And in fact, you have the same mechanism already implemented for deserialization.
Altogether, I see it not so much as an update ‘tailored’ to our specific data structure, but an update that could benefit all users. So hope you’re willing to consider implementing some caching mechanism. Would be nice to hear your opinion on it!