Alright, this issue is solved!
I had to reference the Objects.dll from \AppData\Roaming\Speckle\Kits\Objects in my own project and also needed to make sure that these files are not copied to my output folder.
Then there was still an issue because my plugin couldn’t find the right .dll if the Objects.dll was not loaded already by the Speckle Revit plugin. So then I used the Assembly.LoadFrom Method in my code to make sure it does the job.