We’ve been thinking a bit about a testing strategy for our v2 deployment and the v2 connector development we’re planning to do (ex. GSA connector), and we’re keen to get your thoughts on a few things:
To make it easier to set up a consistent set of automated tests for each connector, we’re interested in integrating connector-specific runners into our testing strategy. These runners would have the relevant software and required licenses installed (ex. a Rhino runner, a GSA runner). We’d build out a set of tests for each connector that could be executed on the relevant runner. We’re thinking to start by adding an initial set of tests that check if a connector can be properly loaded in its respective application. Longer term though, we would like to develop more complex, perhaps closer to end to end, tests that could be automated via use of the runner. We are also thinking to start with setting up Rhino runners first, while we look into how we might implement this for the other connectors. Thoughts on this approach? Would this be something that aligns well with overall testing strategy for Speckle as whole? Ideally these tests could sit in monorepo for the existing connectors, and we could update the existing CI/CD workflow to execute these tests as well.
For community-developed connectors, is there a minimum required level of testing that we should all be aiming for? If so, what should it include/cover?
Is CircleCI the platform that the team is planning to use going forward? (if it is what you plan to use long-term, we will look to setup our CI/CD with CircleCI as well)
On a slightly different vein - has anyone done any tests or benchmarking exercises that compare the performance of v1 vs v2? Ex. comparing speed of sending/receiving, comparing speed based on number of objects
Hey Janessa, welcome to the forum & thanks for bringing this up!
I can tell you what we have at the moment and then we can work together to improve things
In speckle-sharp, we have Core with plenty of unit and integration tests, but for the Connectors only Revit has some Tests which have to be manually run atm. The main challenges we have found when looking into automating connector testing are:
asides from Rhino 7, no other host software can be run headlessly
VMs with each software installed have to be spun up
it is quite hard to do any UI testing
We would definitely support the development of runners for automated connector testing, I know @AlanRynne had started looking into creating Tests for Rhino.
For community-developed and official connectors alike, we should guarantee they load properly in their host application, that basic conversions work as expected and that core functionalities such as sending and receiving work before each release.
CircleCI is the platform we have been using so far, but we’re not tied to it!
In regards to performance, @dimitrie did a lot of benchmarking in the early stages of v2, he can tell you more about that!
I’m unfortunately short on numbers right now But overall, it was on par or faster in key places. The big thing to note is that Speckle is literally now an object graph storage system, which is a huge evolutionary step from the “everything must be a flat list or else” setup in 1.0.
There’s several headline performance improvements:
no more object size limitations (combo of my pet decomposition api and a sprinkling of chunking magic)
api queries are much faster on large sets (300k+ objects)
no more limitations on “stream” (now commit) size
thanks to cristi, stability and memory performance on the server is now becoming
To note, some things - such as breps - may be slower, but it’s the price we need to pay for fancy direct shapes and free form elements in revit.
We still have a lot of room for optimizations which are slowly making their way in, for example, diffing was introduced just several weeks ago. There’s also this pr from @cristi that we need to finalise on the end user side; that shaves off another few seconds from deserialisation. We’re looking at memoisation in converters, some caching in deserialisation & reducing reliance on reflection in our hot paths, etc.
Great to see we can work together on this one. It is worth noting that we are also interesting on manual testing, where we cannot automate. On V1 there were a few .gh files that people were supposed to run one by one and take notes, maybe we can do this a bit more organised for key clients for v2!
I was going to do a quick test with Rhino 7 GH plugin but if Alan is working on it but I can wait! I am off on holidays either way
@AlanRynne could you share your WIP when you are ready?
Hey @Stam, I did my initial explorations creating unit tests with Rhino.Inside on this branch:
It was a bit outdated so I just rebased it from the ‘latest’ master. Not sure of the limitations of this approach, initially I was thinking to just test the converter, but I imagine other more complex tests can be performed…
For now, the test project is just a proof of concept with a single conversion test. It runs on Rhino.Inside so it requires Rhino 7 in your machine to work.
But overall, it was on par or faster in key places. The big thing to note is that Speckle is literally now an object graph storage system, which is a huge evolutionary step from the “everything must be a flat list or else” setup in 1.0.
