Understanding the structure of streams from Revit

[henkon (Henning)]

Hi, I am quite new to Speckle. Just playing around with it for now when I have the time.
I am using the new Speckle 2 Alpha Connector for Revit (2021) as-is, v.2.3.0.
I tried to create a stream where I send the entire Revit model (“all”). The model I am working with, is the Revit RAC_sample_project.
What I find difficult, based on trying to retrieve information using graphql, or just browse the stream on speckle on-line, is what is sent, how it is classified and structured in the process and then how to query it.
If I browse the model on-line, I see that for example there are two windows under grouping “windows” with no geometry and little other information. To find all other windows, they are found under elements for each wall for example. So far, so good I guess.
However, if I query the stream, I will manage to find all windows with its parameters without going through the walls themselves, which is one thing I am trying to do. I can query by applying a filter for Category = “Windows” (the revit category) (windows seems to get speckle_type “Objects.BuiltElements.Revit.FamilyInstance”

For Walls however, the model seems more confusing. If I do the same thing, querying for Category = “Walls”, I get back some walls, but no parameters. I manage to find them when exploring on-line as well. If I do a more generic query, or browse the model on-line to find all walls, it seems I have to query for speckle_type = “Objects.BuiltElements.Wall:Objects.BuiltElements.Revit.RevitWall” to get the actual walls that has detailed information like parameters, levels, etc… (These wall elements do not have a parameter “category” with value “Walls”), thus are not returned in my graphql query for category = walls.
Also, the wall “structure” does not seem to be part of the wall - by that I mean the materials/layers of the wall. I guess there are just some constraints in the plug-in itself and what is sent?

For doors, well, they just are not sent to Speckle when sending the model (“all”). They are not even a choice if I try to send Family and Types or categories. Only way to send a door that I have found, is by manual selection.

So, I am trying to find a “unified way” to query the model and process the response. I would think that I could query different categories - Revit categories, but unfortunately it is not that easy. I am not very used to graphql, and the way Speckle structures the data. I wanted to find “one query fits all”.

To process the returning json, I played around with .NET dynamics and newtonsoft to not tie myself to serialize the response to classes, since I at first was un-sure if I could standardize what I query and the response I would get. That was no fun. Any good resources, examples, experience would be welcomed.

That was a lot, I might get an “F” in how to create a good topic

[dimitrie]

Hi @henkon, apologies for the late reply! No worries about the long post

Some of your remarks sound like potential bugs from our end. Specifically:

@henkon

there are two windows under grouping “windows” with no geometry and little other information.

Either those two windows are not hosted anywhere, or we’re fumbling something up.

@henkon

For doors, well, they just are not sent to Speckle when sending the model (“all”). They are not even a choice if I try to send Family and Types or categories. Only way to send a door that I have found, is by manual selection.

This one too. “All” should really mean all

@henkon

the wall “structure” does not seem to be part of the wall - by that I mean the materials/layers of the wall.

This is something that was requested in this thread - so it should make it in our todos.

@henkon

So, I am trying to find a “unified way” to query the model and process the response. I would think that I could query different categories - Revit categories, but unfortunately it is not that easy. I am not very used to graphql, and the way Speckle structures the data. I wanted to find “one query fits all”.

This is defintively something we should enable and/or write a tutorial about, as it’s not that straightforward, especially because we have two APIs right now through which you can do it (you probably noticed that ) - one is the GraphQL way, which you have tried out, and that supports queries, and the other one is the REST way, which we use for sending and receiving (and that doesn’t support queries).

In .NET, To resolve the “not fun” part of the dynamics and such, i would actually forego the graphql way, and go for receiving the objects the classic way the connectors do and writing a traversal function to isolate what object categories you need. Would you need some help with that? Someone from the team could give it a shot!

I’m recommending this way mostly because we have https://github.com/specklesystems/speckle-server/issues/340 in the backlog since a while, and we plan to tackle it sooner or later! (will add support for querying objects on the REST endpoint too).

[henkon (Henning)]

Hi @dimitrie
thanks for the reply, I didn’t find it late at all.

For the windows issue:
I did not check in Revit if there are two un-hosted windows, could be. The two windows reported in speckle on-line has no geometry.

For the issue with the doors:
Great. I have only tested the basic example model from Revit, but seems like a bug maybe

For the issue with wall structure:
Great, it sounds good, as it is important

For the “how to treat responses from graphql”, I will look into going in the non-graphql way then. I will try to find documentation on the REST API instead and see how that works out. If I need help, I will let you know by replying on the thread. Maybe I will find some good examples as well. If you have some examples/resources from your knowledge available, I would welcome them of course Happy weekend!

[dimitrie]

Hi @henkon, just had some fun today hacking a quick Flatten function + some LINQ filtering. I’ve posted the full spiel in our docs right over here (we try to document valid questions and responses!)

Long story short, add this extension method somewhere to your project:

public static class Extensions
{
  // Flattens a base object into all its constituent parts.
  public static IEnumerable<Base> Flatten(this Base obj)
  {
    yield return obj;

    var props = obj.GetDynamicMemberNames();
    foreach (var prop in props)
    {
      var value = obj[prop];
      if (value == null) continue;

      if (value is Base b)
      {
        var nested = b.Flatten();
        foreach (var child in nested) yield return child;
      }

      if (value is IDictionary dict)
      {
        foreach (var dictValue in dict.Values)
        {
          if (dictValue is Base lb)
          {
            foreach (var lbChild in lb.Flatten()) yield return lbChild;
          }
        }
      }

      if (value is IEnumerable enumerable)
      {
        foreach (var listValue in enumerable)
        {
          if (listValue is Base lb)
          {
            foreach (var lbChild in lb.Flatten()) yield return lbChild;
          }
        }
      }
    }
  }
}

Afterwards, you’ll be able to do things such as:

      var data = Helpers.Receive("https://speckle.xyz/streams/0d3cb7cb52/commits/681cdd572c").Result;
      var flatData = data.Flatten().ToList();

      var windows = flatData.FindAll(obj => (string)obj["category"] == "Windows");
      var timberWalls = flatData.FindAll(obj => obj is Objects.BuiltElements.Revit.RevitWall wall && wall.type == "Wall - Timber Clad");
      var rooms = flatData.FindAll(obj => obj is Objects.BuiltElements.Room);
      var levels = flatData.FindAll(obj => obj is Objects.BuiltElements.Level).Cast<Objects.BuiltElements.Level>().GroupBy(level => level.name).Select(g => g.First()).ToList();

      Console.WriteLine($"Found {windows.Count} windows.");
      Console.WriteLine($"Found {timberWalls.Count} timber walls.");
      Console.WriteLine($"Found {rooms.Count} rooms.");
      Console.WriteLine($"Found {levels.Count} levels.");
      
      var elementsByLevel = flatData.FindAll(obj => obj["level"] != null).GroupBy(obj => ((Base)obj["level"])["name"]);
      foreach(var grouping in elementsByLevel) {
        Console.WriteLine($"On level {grouping.Key} there are {grouping.Count()} elements.");
      }

Have fun!

[henkon (Henning)]

Hi again @dimitrie, I really appreciate the help. I tried to replicate your joy from yesterday. hehe, I hit a snag testing it in a console app, but at least got it working from a windows forms app. Thanks

[dimitrie]

Right, I forgot to publish the repo For anyone else reading this in the future, I’ve pushed it here.

[henkon (Henning)]

@dimitrie may I challenge you to return all parameters for a window?
First, LINQ is not my strong side yet, and secondly I get in the same trouble as I did with dynamics when trying. I am unable to get the parameters without accessing them by named index, and I do not know them by name up front

[dimitrie]

Ah, that’s easy. You can get all dynamic prop names from a base object using GetDynamicMembers(), then you can simply iterate through them and access the object via [] notation.

Here’s a snippet i cooked up just now:

var myWindowParams = ((Base)windows[0])["parameters"] as Base;
var propNames = myWindowParams.GetDynamicMembers();
foreach(var prop in propNames)
{
  var param = myWindowParams[prop] as Objects.BuiltElements.Revit.Parameter;
  Console.WriteLine($"{prop}: {param.value} (Name: {param.name}, Unit: {param.applicationUnit})");
}

Edit: note, i’m doing this for a single window only. Output below. Note, some params come with nulls or empty values, so you should always act defensively there.

ALL_MODEL_MARK: 22 (Name: Mark, Unit: )
CASEWORK_WIDTH: 1118 (Name: Width, Unit: DUT_MILLIMETERS)
OMNICLASS_CODE: 23.30.20.21.14 (Name: OmniClass Number, Unit: )
UNIFORMAT_CODE:  (Name: Assembly Code, Unit: )
WINDOW_TYPE_ID: 33 (Name: Type Mark, Unit: )
CASEWORK_HEIGHT: 1169.9999999999998 (Name: Height, Unit: DUT_MILLIMETERS)
HOST_AREA_COMPUTED: 2.7291125068598747 (Name: Area, Unit: DUT_SQUARE_METERS)
ALL_MODEL_TYPE_NAME: 1180 x 1170mm (Name: Type Name, Unit: )
HOST_VOLUME_COMPUTED: 0.04199918304494646 (Name: Volume, Unit: DUT_CUBIC_METERS)
ALL_MODEL_FAMILY_NAME: M_Skylight (Name: Family Name, Unit: )
OMNICLASS_DESCRIPTION: Roof Windows (Name: OmniClass Title, Unit: )
UNIFORMAT_DESCRIPTION:  (Name: Assembly Description, Unit: )
ALL_MODEL_TYPE_COMMENTS:  (Name: Type Comments, Unit: )
INSTANCE_OFFSET_POS_PARAM: False (Name: Moves With Nearby Elements, Unit: )
STRUCTURAL_FAMILY_CODE_NAME:  (Name: Code Name, Unit: )
ANALYTICAL_THERMAL_RESISTANCE: 0.27110556850837714 (Name: Thermal Resistance (R), Unit: DUT_SQUARE_METER_KELVIN_PER_WATT)
ANALYTIC_CONSTRUCTION_GBXML_TYPEID: GSP4R (Name: Construction Type Id, Unit: )
ANALYTIC_CONSTRUCTION_LOOKUP_TABLE: 1/8 in Pilkington single glazing (Name: Analytic Construction, Unit: )
ANALYTICAL_HEAT_TRANSFER_COEFFICIENT: 3.6886 (Name: Heat Transfer Coefficient (U), Unit: DUT_WATTS_PER_SQUARE_METER_KELVIN)
ANALYTICAL_VISUAL_LIGHT_TRANSMITTANCE: 0.9 (Name: Visual Light Transmittance, Unit: DUT_GENERAL)
ANALYTICAL_SOLAR_HEAT_GAIN_COEFFICIENT: 0.78 (Name: Solar Heat Gain Coefficient, Unit: DUT_GENERAL)
ANALYTICAL_DEFINE_THERMAL_PROPERTIES_BY: 1 (Name: Define Thermal Properties by, Unit: )

[henkon (Henning)]

hehe, many thanks @dimitrie , I have tried a few combinations that is so close to your snippet, as this was how I imagined it. But didn’t manage to cast things correctly.

Cheers

[dimitrie]

Yes, i feel you. We’re trying to allow for full flexibility in a strongly typed language, and sometimes the compromises feel s bit off. As we progress, i hope we will have better patterns emerge and/or document things properly

[dirksliepenbeek]

Hi there,

This is somehow related to my issue, so I just drop my question here @dimitrie.

I’ve got a stream created in Grasshopper with Revit elements in it. Now I try to use the Flatten extension method described above to get out the objects in my @data. This does give back a list with the @data in it again. Double (or tripple) flattening does not do the job. I noticed my stream structure is quite different from the one in the Flattening docs, but I am a bit lost on how to receive objects and use them further in the code. I’m using C# with the Revit API.

Grasshopper 2.5.2 (2.6 gave other issues described here by someone else)
Revit 2.6

Speckle Stream

Thanks in advance

[AlanRynne]

Hi @dirksliepenbeek,

so you’re trying to flatten a Grasshopper commit in Revit?

As far as I’m aware, the flatten function should work as expected in 2.5.2. As for 2.6.0, we’ve modified the structure of Grasshopper’s data trees to be a Base object instead of nested lists, but it should still be compatible with the flatten function.

I’m going to do some tests and i’ll get back to you.

What other issues did you encounter in 2.6?

[dirksliepenbeek]

Hi @AlanRynne ,

Thanks for the response: Speckle, I was unable to receive this stream in my plugin.

Before we used the @SpeckleSchema detached property to actually map the elements to it’s C# object, that seems to not be possible anymore. Then I started looking what would be the right way and I bumped into that Flatten functionality which seemed quite useful for us.

What we are trying to do is get the family name and type from the SpeckleRevit object, and change it in the Speckle stream through a UI if that’s what the user want. Then we update the stream. If there is a more easy way to that then the mapping to the SpeckleRevitObj I would be keen to hear that.

[AlanRynne]

So I’ve had a chat with @dimitrie, and it turns out the Flatten function provided in the docs is more intended as an example (and was designed to work with Revit commits mostly).

I’ve played around a bit and came up with a new Flatten function I’d love you to try. This one no longer takes in a Base object, but rather a generic object, and returns a Enumerable<Base> just like the previous one.

    public static class Extensions
    {
        public static IEnumerable<Base> Flatten(object obj, HashSet<string> existingCache = null)
        {
            var cache = existingCache ?? new HashSet<string>();
            switch (obj)
            {
                case Base @base:
                {
                    if (cache.Contains(@base.id)) break;
                    cache.Add(@base.id);
                    yield return @base;
                    foreach (var prop in @base.GetDynamicMemberNames())
                    foreach (var child in Flatten(@base[prop], cache))
                        yield return child;
                    break;
                }
                case IDictionary dict:
                    foreach (var dictValue in dict.Values)
                    foreach (var child in Flatten(dictValue, cache))
                        yield return child;
                    break;
                case IEnumerable enumerable:
                    foreach (var listValue in enumerable)
                    foreach (var child in Flatten(listValue, cache))
                        yield return child;
                    break;
            }
        }
    }

then just use it like this

var obj = Speckle.Core.Api.Helpers.Receive("YOUR_STREAM_URL").Result;
var flat = Extensions.Flatten(obj).ToList();

I tested it on your commit and some other more complex ones with some success. I’m also preventing duplicate objects from being added to the flat list multiple times.

@dirksliepenbeek

Before we used the @SpeckleSchema detached property to actually map the elements to it’s C# object, that seems to not be possible anymore.

As for this, not sure what you’re referring to, I’m not aware of any changes in the behaviour of the speckleSchema tags atm, could you give us a bit more detail about this? Feel free to open a different thread as it seems non-related to this directly

Anyway, let me know if the new function works as expected!

[dirksliepenbeek]

Thanks for getting back so quickly.

var commit = await AppState.Client.CommitGet(AppState.Stream.id, AppState.Stream.commit.id);
var serverTransport = new ServerTransport(AppState.Client.Account, AppState.Stream.id);
var res = await Operations.Receive(commit.referencedObject, serverTransport);
var properties = res.GetDynamicMembers().ToList();
var objList = new List<Base>() { };
var objects = properties.SelectMany(prop =>
{
  var listOfLists = res[prop] as List<object>;
  return listOfLists?.Select(l => l) ?? new List<object>();

}).ToList().FirstOrDefault() as List<object>;

var schemaObjects = objects.OfType<dynamic>()
  .Select(l => l["@SpeckleSchema"] as dynamic).Where(lb => lb != null)
  .ToList();

foreach (var schema in schemaObjects)
{
  switch (schema)
  {
      case RevitBeam revitBeam:
          objList.Add(revitBeam);
          break;
      case RevitColumn column:
          objList.Add(column);
          break;
      case RevitWall wall:
          objList.Add(wall);
          break;
      case RevitFloor floor:
          objList.Add(floor);
          break;
      case Beam beam:
          objList.Add(beam);
          break;
      case Column column:
          objList.Add(column);
          break;
      case Wall wall:
          objList.Add(wall);
          break;
      case Floor floor:
          objList.Add(floor);
          break;
      default:
          break;
  }
}

This is a snippet from the previous codebase, we used this stream from testing: Speckle.

In this way we could easily access the objects properties later in the flow of the programme.

I’ll give the new flatten code a try tomorrow, thanks for creating it!

[dirksliepenbeek]

Hi Alan,

I just gave it a try, and it is doing the flattening indeed :), also tried different streams and all worked.

But what I’m currently struggling with is casting the Base objects to the right SpeckleRevit object:

var rooms = flatData.FindAll(obj => obj is Objects.BuiltElements.Room);
Trying above for RevitBeam did not work as the object is of type Base, while in the stream you can see that the property “speckle_type” is of RevitBeam.

Then I tried to cast the Base object to the RevitBeam in this way, Revit just froze:

var flat = Extensions.Flatten(res).ToList();
var beams = flat.Where(lb => lb.applicationId != null);
beams = beams.Where(obj => obj.applicationId.Contains("RevitBeam")).Cast<RevitBeam>().ToList();

But that’s also crashing.

Maybe some relevant background:

var obj = Speckle.Core.Api.Helpers.Receive("YOUR_STREAM_URL").Result;
var flat = Extensions.Flatten(obj).ToList();

I was not able to use the Helpers.Receive to receive the stream, so I did it like this:

var commit = await AppState.Client.CommitGet(AppState.Stream.id, AppState.Stream.commit.id);
var serverTransport = new ServerTransport(AppState.Client.Account, AppState.Stream.id);
var res = await Operations.Receive(commit.referencedObject, serverTransport);

Not sure if that would affect it as well.

[AlanRynne]

@dirksliepenbeek

var rooms = flatData.FindAll(obj => obj is Objects.BuiltElements.Room);
Trying above for RevitBeam did not work as the object is of type Base, while in the stream you can see that the property “speckle_type” is of RevitBeam.

I think there may be a bit of a misconception here. When I receive them, all the objects are properly instantiated as an instance of their specific class:

I think what’s happening here is that your script/project/plugin does not have access to the Objects.dll, hence you cannot cast that into a more specific class.

As an example, take the Revit connector:

• It is a Revit plugin that is in charge of Send/Receive operations mainly but knows nothing about the objects it receives.
• It also has a Converter Project that knows everything about the available speckle objects, as well as the Revit objects, so it acts as a “bridge” between the 2.

This was done to allow for extending the object model, or even replacing it for a different one alltogether.

What I’d do in your case

Depending on where you’re running this:

• If its a Grasshopper/Dynamo script, you can just reference our Objects.dll from the Kits folder (%appdata%/speckle/kits/objects) and it should work alright.
• If its a Revit or GH plugin (or any other plugin), you could do the same as above to ensure you’re always running the same version of objects,
• If your project is designed to work “standalone”, in theory you could just use our Objects nuget package, but you’d have to deal with upgrading the versions as we do, or you might face some nasty errors.

As for not using the Helpers, its fine. I think what is throwing your code off is the final .Result at the end, which some platforms like and others don’t But what you’re doing looks fine.

As for filtering, what you’re doing seems correct:

In order to cast it:

but you can make it even shorter:

var beams = flat.OfType<Objects.BuiltElements.Beam>().ToList();

As long as Objects.dll can be found at runtime this should do the trick!!

If you could give us more details on where and how is this running we may be able to give you some tips on how to set it up.

[dirksliepenbeek]

Appreciate the elaborate answer, I’ll have a more detailed look tomorrow morning. Thanks!

To give you a bit of context on what we are building…

Speckle Library Manager

Hi JdB, Thanks for tagging me! We’ve been doing some work on that one yes, as you know. It’s similar to the process you are describing but for us it connects to our inhouse developed library which is stored in .json format. The workflow is as follows: Select stream, branch, commit Plug-in will read all the specific unique types Based on the “mapping json” (basically the key is the structural type from our library, the value is the Revit family name / type name from our library), it auto sele…

Then this is how we solved some issues with the different versions of the objects.dll earlier

Speckle for Revit plugin assembly/version conflicts

Alright, this issue is solved! I had to reference the Objects.dll from \AppData\Roaming\Speckle\Kits\Objects in my own project and also needed to make sure that these files are not copied to my output folder. Then there was still an issue because my plugin couldn’t find the right .dll if the Objects.dll was not loaded already by the Speckle Revit plugin. So then I used the Assembly.LoadFrom Method in my code to make sure it does the job.

[AlanRynne]

Nice! Thanks for cross referencing, I hadn’t joined the dots by myself!

So I think the solution for this would be do the latter, and call Assembly.LoadFrom to ensure the Objects.dll is actually loaded, if you’re not doing this already If you are… then we may need to look deeper into this.

The way this currently works is that the Kit manager will inspect the Objects.dll for available types, but it won’t actually load the assembly until its needed, i.e. until a conversion is performed.

So you can’t really count on objects being loaded really, as you’ve already discovered but they will get loaded at some point…

I just messaged Dim to see if we could add some mechanism on our side to make your life easier, and rely on the KitManager to do this for you (and us). But for now, I think the solution that you found is good enough as long as you reference the Objects.dll in the Kits folder which you were already doing.

Anyway, I’m intrigued about the solution, but also how we could make your life as a developer a bit easier Do keep the questions coming! And if you have any suggestions we’d be happy to hear them too.

[dirksliepenbeek]

I actually dived into the RevitSpeckle plugin to investigate how the objects are being mapped and I noticed you were calling the Kit manager over there as well. I tried to implement this one in my code also, but without success (also did not try super hard ).

I will do some testing tomorrow, maybe it will work if I first receive some objects with the RevitSpeckle plugin. This won’t be a solution ofcourse, but it will give more insight in the problem.

I’ll expect to drop more questions later. Thanks a for the support!