🐍 PySpeckle 2.0 is starting to take shape...be an early tester!

[izzylys (izzy)]

Hey @Speckle_Insider!

If you’re a user of pyspeckle 1.0, you’ll probably be excited to hear that pyspeckle 2.0 is currently under development! It is still highly WIP and things might change at a moment’s notice, but I would super appreciate it if you wanted to check it out

It’s far from complete, but there are some cool things in the works that you can have a play with. Have a look at the readme for a quick overview of what’s currently there, but here’s a little whistle stop tour:

First off, there’s the client which lets you interact with the server’s GraphQL API

from speckle.api.client import SpeckleClient
from speckle.api.credentials import get_default_account, get_local_accounts

all_accounts = get_local_accounts() # get back a list
account = get_default_account()

client = SpeckleClient(host="localhost:3000", use_ssl=False)
# client = SpeckleClient(host="yourserver.com") or whatever your host is

client.authenticate(account.token)

# create a stream
new_stream_id = client.stream.create(name="a shiny new stream")

# create a commit
commit_id = client.commit.create("stream id", "object id", "this is a commit message to describe the commit")

One key consideration with pyspeckle 2.0 is to provide many of the advanced features from Core as to not “limit” python devs. To that end, we’ve got a Base object capable of decomposition - and a serializer to handle this! Read more about the Base object here and the decomposition API here

detached_base = Base()
detached_base.name = "this will get detached"

base_obj = Base()
base_obj.name = "my base"
base_obj["@nested"] = detached_base

serializer = BaseObjectSerializer()
hash, obj_dict = serializer.traverse_base(base_obj)

We’re also working on transports . Read more about transports here

transport = MemoryTransport()

# this serialises the object and sends it to the transport 
hash = operations.send(base=base_obj, transports=[transport])

# if the object had detached objects, you can see these as well
saved_objects = transport.objects # a dict with the obj hash as the key

# this receives and object from the given transport, deserialises it, and recomposes it into a base object
received_base = operations.receive(obj_id=hash, remote_transport=transport)

If you have any comments, feedback, suggestion, or ideas please feel free to add them here or flag an issue in the repo! If you give it a whirl, consider joining us for the next Speckle Community Standup on Wednesday 16 Dec @ 16:30 UK time (announcement post to come…) to let us know what you think / ask any questions face-to-face

[TomSvilans]

Hi @izzylys this looks brilliant! Nice and clean

Excited to see how this develops and I hope to contribute soon.

[izzylys (izzy)]

Thanks a lot Tom!

Looking forward to your contributions and keen to hear any of the ideas you’ve been cooking up. Let me know when you start working with this in Blender so we can coordinate

[JdB (Jeroen de Bruijn)]

@Rob this is what you where looking into, maybe you can check it out?

[Rob]

Looks good! This should solve my initial issues with authentication.
Also good to see that the Readme is extended. Will have a try this week.

[Rob]

Hi @izzylys,

Over the last weeks I have been testing pyspeckle 2.0. I have a server running locally to send and receive some objects. This week I tried to set up some very basic classes that inherit from the Base object. As a structural engineer, I start of course with making a SpeckleBeam class, which inherits from the SpeckleLine class, see below.

Part of the problem is that I’m not exactly sure on how to subclass Base, which in turn inherits from the Pydantic BaseModel. To me it seems that the use of this Pydantic class brings quite some limitations in making the subclasses (this is at least my first impression, but might be my own lack of experience with it ). I tried to formulate some clear questions:

• Is it possible to create a read-only attribute/getter? I tried the “const” parameter of Field to achieve this, but that didn’t work so far. Currently that means that I can change the “speckle_type” attribute of the objects to anything at anytime, which is not desirable. My intention is to override the “speckle_type” attribute from Base in each class to have a constant value for each object, e.g. “SpeckleBeam”. In pyspeckle 1.0, we could simply use the property decorator to define a getter, and reference that in a dataclass field, as shown here:

Trying to do that now raises an error when it’s trying to initiate the SpeckleLine class, as it cannot deepcopy a MappingProxy object within the field.

• How should I properly nest an object in another one? Below you can see what I did currently, the SpeckleLine is nested in the baseLine attribute of the SpeckleBeam. Afterwards I serialize both items and commit them using their hashes.

Subsequently I try to retrieve these two items back from my stream. This actually succeeds using the hashes. However, both are Base objects which are more or less empty/default, they do not contain any of the attributes I assigned to them. Am I doing something wrong in the serialization?

• Then another remarkable thing when I retrieve my two objects back. For some reason it expects that there are “speckleType” attributes instead of “speckle_type”. It seems that at some point it converts the “speckle_type” attributes to camelCase.

Those are the main issues I’m currently experiencing as an early tester. Would be nice to hear your thoughts on (subclassing) the Pydantic Base and the serialization as shown above. Looking forward to your answer! I’m also very interested in the experiences of others using pyspeckle 2.0

[izzylys (izzy)]

Hey Rob,

Thanks so much for the testing and feedback! Let me get into answering some of your questions:

Properties
These should work as you’re used to, eg:

class SpeckleLine(Base):
    _type: str = "SpeckleLine"

   def __init__(self, **kwargs) -> None:
       super().__init__(**kwargs)

    @property
    def type(self):
        return self._type

However, I didn’t realise until now that this doesn’t allow you to override an existing field as a property as setting the attribute takes precedence! Note that currently, speckle_type is populated automatically based on the name of the class. You are right though that it is not protected - I think this is probably a change that we would make in Base. A consideration here is the serialisation of properties - the BaseObjectSerializer would need to be updated to reflect the use of properties.

On Pydantic base classes, they offer quite a few neat features for validation, serialisation, and code generation in particular. However, I am still weighing in my mind if it is better to forego them and stick with a more basic dataclass implementation. Definitely keen to hear more thoughts or other ideas on this as I’m not too sure myself at the moment!

Serialisation
If you would just like to serialise and deserialise independent of any transports, I would use the methods in operations:

line = SpeckleLine()
line.value = [0,0,0,4,0,0]
    
beam = SpeckleBeam(material="Timber")
beam['@baseline'] = line

serialized = operations.serialize(beam)
deserialized = operations.deserialize(serialized)

For sending to a server, you would use the send and receive using a ServerTransport. This handles the serialisation internally for you.

client = SpeckleClient(host=host)
account = get_default_account()
client.authenticate(token=account.token)

stream_id = "stream123"
transport = ServerTransport(client, stream_id)

beam_hash = operations.send(beam, [transport])
client.commit.create(stream_id, beam_hash, message="here's a beam!")

beam_obj = operations.receive(beam_hash, transport)

Regarding speckle_type vs speckleType, that’s a bug sorry . Will be fixing this shortly.

[Rob]

Thanks for the quick reply @izzylys!
My considerations regarding both topics are:

Properties
I’m not so sure that these work similarly when using the Pydantic base. For a simple getter, this indeed works. However, when I try to implement a property with a setter and a getter, this immediately gives an error:

It doesn’t allow me to set the “base_line” attribute as it is not a Pydantic Field. Also, including “base_line” in the class setup (commented line) doesn’t make sense, as this is immediately overwritten by the property. It simply seems that they do not blend very well, which is quite a drawback as properties are a nice (and I guess also commonly used) feature in Python. I think the Pydantic Field should provide some more flexibility/control over the attributes to the developer, it however seems a bit limited at first glimpse.

I think using the built-in dataclasses.dataclass functionality would offer a lot more flexibility to developers to customize their own classes. It works fine with properties and the built-in dataclasses.field provides some more useful options for defining attributes:

Maybe I’m not fully objective as I’ve worked more extensively with these dataclasses. But I’m curious to hear if there are ways to overcome above issues with the Pydantic classes.

Regarding protected attributes, I think this would definitely be a useful addition. As a developer, it’s of course desirable to be able to control this.

Serialisation
Thanks for sharing this piece of code. Will definitely have a go with the ServerTransport! One question about it; would you always send and commit objects one-by-one? From the docstrings it seems that this is indeed the procedure.

And regarding the bug, no problem of course

Have a nice weekend!

[izzylys (izzy)]

Thanks a lot sharing your view! Going with dataclasses is definitely something to consider if the pydantic models do not provide the necessary flexibility. I’m a bit biased in the other direction as I’ve heavily used and enjoyed pydantic, but I am of course happy for a move to refactor this if it is the better solution for the majority of users!

Regarding properties, what you want to do is possible with a few tweaks to your code (and one addition I just merged into main! if you’re curious: I didn’t realise that __setattr__ takes precedence over properties, so I’ve added a check to this so setters actually set what you think they should set.)

You are getting the AttributeError for __fields_set__ because you have not initialised the class before setting the base_line attribute. If you want to set attributes in the init function, you would do so after calling init on its parent:

def __init__(self, **kwargs) -> None:
    super().__init__(**kwargs)   # note: you must do this first!
    self.base_line = None

If you want to initialise the property in the class setup, you should be initialising the private/protected backing property, not the property itself since (as you noted) initialising the property as an attribute won’t do anything useful.

class SpeckleBeam(Base):
    _base_line: SpeckleLine = None   # note: not setting `base_line`

    def __init__(self, **kwargs) -> None:
        super().__init__(**kwargs)

    @property
    def base_line(self):
        return self._base_line

    @base_line.setter
    def base_line(self, value: SpeckleLine):
        self._base_line = value

EDIT: oopsie I forgot to address your question re commits

A commit will itself only ever reference one referenceObject which is a Base object. However, this Base object can contain as many objects as you want. This means all the objects you want to add to the commit will be children of the parent Base object. If you make some of these detachable, the parent Base will be decomposed for you when you use operations.send() and will be recomposed for you upon calling operations.receive().

See this post for more on the base object and this post for more on the decomposition API.

[Guus]

Hi @izzylys I am stoked about this development and keen to test it out. In particular, I was curious about this:

@izzylys

One key consideration with pyspeckle 2.0 is to provide many of the advanced features from Core as to not “limit” python devs.

I have been trying to convert Rhino geometry objects to Speckle 1.0 objects using Python.NET (in order to create streams from Rhino Inside Python and/or Rhino Compute output), but with no luck thus far (see this thread).

Would this be easier with PySpeckle 2.0?

[izzylys (izzy)]

Hey Guus, glad to hear you’re excited about pyspeckle!

Re your specific question on Rhino conversions, I am inclined to say yes, but the answer to that has a few different parts

PySpeckle is essentially a Python implementation of Core - the .NET SDK for Speckle. This means we are trying to bring you everything you’d expect (intuitive API calls to interact with your data on your server) as well as more advanced features such as transports, serialisation, and the decomposition API in native Python 3. We’re also planning on generating the Objects classes automatically from JSON schemas published from .NET (see this PR for a tentative look).

However, geometry Kits and Converters are a bit of a different story as they are outside of the scope of Core. You will still need to use Python.NET to use the speckle-sharp converters. At the moment there aren’t plans to implement this logic into “Python Kits”. Keep in mind though, Speckle 2.0 has undergone lots of improvements and I am pretty confident that your experience using the New & Improved 2.0 will be smoother sailing

[Rob]

Thanks Izzy,

After pulling the latest changes, I can indeed use properties as demonstrated! Your comments regarding Pydantic and properties make sense, I should have a better look into it before complaining
Also noticed that the speckle_type vs. speckleType bug is fixed, nice!

Your answer regarding decomposition of the base object including the nested objects is also clear.
I do (again) have two additional question regarding this topic:

• Regarding the nested attributes (e.g. speckle_beam["@baseLine"] in line with the previous example), from the Decomposition API topic I understand that you would not declare this also as a class attribute (e.g. baseLine: str = None). Looking at how the decompostion works, I think that makes sense. However, it also seems a bit strange at first, that you wouldn’t declare this attribute at all, and just nest it in the object after initialisation. I was thinking of adding a getter, that retrieves the nested item or returns None if it is not defined. Then at least it is present as a property from initialisation. What is your view on this?
• While retrieving my objects back from the server, I noticed that they are always a Base object, also the nested objects. Will this be standard from now on? I don’t see any problems with this, it will not be an issue to create the corresponding Beam/Line object, but more out of curiosity.

Thanks again for the quick answers and fixes in the repo!

[izzylys (izzy)]

Glad you were successful in getting the properties to work and that the Base object in commits is more clear to you now!

Tbh the speckle_type vs speckleType bug is an open issue in speckle-server but I’ve added a temporary cheeky fix in pyspeckle for now

On to your questions:

Nested Base Objects
I think this depends on what you’re wanting to do. If the nested element doesn’t need to be detached, you don’t need to use the @attribute syntax and you could initialise this as normal without using any getters or setters.

If you do need it to be detachable and you do want the attribute to be initialised, your idea of using getters and setters and initialising the backing property (as a private _baseLine not just baseline since private attributes are not serialised) I think is a good one. However, now that I think about it this might be worth an addition in the Base class.

Currently, the only way to detach a prop is dynamically using the @ syntax. However, there should probably be a way for class authors to mark attributes within the class as detachable despite not being able to prepend attribute names with @. There is a simple way to do this in .NET, but PySpeckle doesn’t yet have this. Maybe it should be implemented as a decorator or something simpler a la how chunks are currently defined? Only downside here is that it wouldn’t be obvious to a user if any of their attributes are going to be detached.

Receiving From The Server
Currently, the serialiser looks at all the classes in speckle.objects to find the correct class and construct it during deserialisation. If you’ve defined your classes externally, it will not know what it should use and will default to the regular Base. I’ve been thinking of a way to be able to pass your custom classes or point to serialiser to a folder somewhere to look for existing Base sub classes, but I haven’t quite worked out what the best user experience is for this. If you have a view on how you would like/expect this to work, do share your thoughts!

For now, you could just drop your class definitions into the objects dir along with the base class to see the deserialisation in action.

[Rob]

Had a look into both topics:

Nested Base objects
I now tested getters and setters that get/set the detachable attribute, allowing the user to set these attributes as usual, e.g. speckle_beam.material = speckle_timber, which looks like this:

This actually seems to work quite neat. From the Decomposition API topic I see that implementing a detachable attribute in .NET is indeed a lot simpler compared to Python. I guess it would be nice to have something similar implemented though. Maybe adding something like a decorator @detach before initalising the private _material attribute that adds similar functionality as the property above. Just sharing my thoughts here

Receiving from server
Putting my subclasses within the objects folder indeed did the trick, thanks! What I did notice is that you can only put other Python modules directly within the folder (if you want the classes to be included). You cannot add a nested folder to maintain some structure like objects/custom/beam.py, as these nested modules aren’t included in the search for subclasses. That would definitely be a nice addition. Also, having some functionality to include any subclass somewhere in your project into the deserialization process would be nice. For Speckle 1.0, we have a class that registers Speckle classes during import, which is later used for proper (de)serialization. We can register each Speckle class with a decorator that is defined within that “RegistryClass”, e.g. @RegistryClass.register. Maybe that is something to consider.

Thanks for your answers and hope my input helps!

[izzylys (izzy)]

Heya @Rob !

Soz for the delay - I meant to reply to this earlier, but I wanted to let you know that with the help of @gergo we now have a much slicker way of recognising custom base classes. They are registered automatically, so all you need to do is make sure they’ve been imported and objects should be deserialised correctly! You can see Gergő’s example here.

I’ve also added some helper method for flagging chunkable and detachable attributes. Check out the updated docs here so you can update your classes

[Rob]

Hi @izzylys,

Was suffering from some login issues, but thanks to Dimitrie I’m online again.
Did some testing, and it’s working really neat, happy with the additional functionality!
Also glad to see the addition of the Geometry classes in the PySpeckle client.
That brings me to my question:

• Will you also implement the ‘BuiltElements’ classes that are within the C# Library on the Github repo?

Think this would be really nice, I see a lot of potential there. Going back to the Beam example (structural engineers like beams), we would then be able to easily subclass the (BuiltElements) Beam class into a StructuralBeam class. For this class we can write a conversion method to create a RevitBeam, or alternatively, add chunkable properties that can be used when retrieving this StructuralBeam in Revit. I do have to dive a bit more into the chunkable properties to see how we can really benefit from these, but any feedback on the above ideas is of course already welcome. I will do some more tests with it in the coming weeks.

I also tried to generate some Speckle 2.0 classes on the fly in Python, which is actually very easy with the pydantic.create_model() functionality. Happy to see that this is available through Pydantic.

[dimitrie]

@Rob

I do have to dive a bit more into the chunkable properties to see how we can really benefit from these,

Chunkable attributes are important for list properties that are unbound in size. In other words, if I have an array of number/strings/other objects that can start counting in the 100000s, then you’d ideally flag it as chunkable. We’re using this for mesh vertices and faces, and other array props that can get out of hand - this is what makes it now possible to send a mesh with 10 million vertices or more. I would see them useful in your case for things like structural analysis results, etc.

@izzylys will follow up re the rest - i’m not qualified to answer

[izzylys (izzy)]

So glad you’re enjoying the updates, Rob!

Re BuiltElements, this is coming but ideally through code generation which we’ve put on hold for now. If there is a need for them sooner though, we could def look into implementing them manually!

With how easy it is to create subclasses though, you could also whip up the classes you need for your particular case or create them based on a received object. I’d be interested to hear more about the workflow you’re envisioning using pyspeckle to see how all these pieces might fit together…