QGIS Connector WIP

I totally agree! Although I’m not sure about how this would be supported in other programs that don’t really “understand” height maps (thinking of things like Revit?), which is the main reason they were left out of the initial development :slight_smile:

Although I suspect just being able to extract the raw data may be useful in itself too! Would definitely love to hear more from the community to see how this could be used in different workflows!

Looks like I am running into issues sending 100MB strings…
According to Decomposition API | Speckle Docs I need to tell my property that it´s chunkable. Does that work with a string?

public class RasterLayer : Base
{
    [DetachProperty]
    [Chunkable(10000)] // Can a string be chunked?
    public string raster { get; set; }
}

Or rather use a list?

public class RasterLayer : Base
{
    [DetachProperty]
    [Chunkable(10000)] 
    public List<string> raster { get; set; } = new List<string>();
}

Is there an example around for specklepy sending stuff in chunks?

1 Like

Strings are funny because they’re arrays, but we don’t want to chunk them by default, as it doesn’t make sense for “hello world”, which is the most usecases :smiley: We basically treat them as any other primitive (which they’re not really).

The first option won’t chunk anything; the second option will chunk the list itself, but not the strings inside the list - so you’ll need to do your own “pre-chunking” of your a massive string into the list itself. Does this make sense? I’m assuming you have one huuuuuuge string here that you want split up in parts.

2 Likes

Yes! you can find one in our Developer guide, here :point_right:t3: Examples | Speckle Docs

1 Like

The first option won’t chunk anything; the second option will chunk the list itself, but not the strings inside the list - so you’ll need to do your own “pre-chunking” of your a massive string into the list itself. Does this make sense? I’m assuming you have one huuuuuuge string here that you want split up in parts.

That totally makes sense :slight_smile:

In specklepy that would probably look something like this?

class RasterLayer(Base, speckle_type="Objects.Geometry." + "RasterLayer", chunkable={"Raster": 10000}, detachable={"Raster"}):
    Raster: List[str] = None

    @classmethod
    def from_list(cls, args: List[Any]) -> "RasterLayer":
        return cls(
            Raster=args,
        )

    def to_list(self) -> List[Any]:
        encoded = []
        for line in self.Raster:
            encoded.append(line)
        return encoded

Although it looks like we need to change some server settings first.

Upload server response: File size too large (97716266 > 52428800)
Error sending batch of objects to server: SpeckleException: Could not save the object to the server - status code 400
2 Likes

In case someone is interested, I created a pull request for sending raster layers around: WIP: send raster layers by tlmnrnhrdt · Pull Request #13 · specklesystems/speckle-qgis · GitHub

I am creating 1mb chunks if possible as discussed here: Uploads Max File Size - why is that value chosen? - #5 by dimitrie

3 Likes

Hey @tlmn this is great :slight_smile:

I’ll be doing some QGIS work myself this week to improve the UI so I’ll also check this out! Had a quick peek now and it looks good :+1:t3:

I’m interested to explore how this would play out with other connectors (specially Grasshopper/Dynamo) and hear about what workflows (if any) would this enable that were not possible before.

As I understand it, we may need to have some sort of conversion logic for it to “understand” and display what this Raster data is… similar to how we deal with BIM objects. So there’s a deeper discussion to be had here on supporting GIS data across Speckle and how would that look like.

2 Likes

Currently we are sending a height map (raster data with some real heights attached to it) as a string in the native format. The user takes that string in GH and converts it into a terrain mesh (reducing the resolution).

We could also imagine displaying that data in GH as an image (as done in QGIS), but then we would need some kind of raster band(not sure if that´s the right term?!) with a color <-> height (or any other raster data) mapping.

So I would see two possibilities atm:

  1. Sending generic raster data with 1…n bands (?) to be displayed as an image in GH
  2. Sending some speckle object that specifically represents a height map that can be directly converted into a mesh (which might be ending up way to big without reducing the resolution)

Maybe solution 1.) with the additional information that it´s about heights might be reasonable?

2 Likes

Hi everyone,

since I saw this post on twitter lately (https://twitter.com/SpeckleSystems/status/1465753964230721551) I wanted to ask if there has been some recent development that hasn´t made into the repository yet?

Is somebody working on new features in general and what are the plans for the near future?

We thought that it could be quite useful to add the functionality to pick a specific branch for sending receiving.

1 Like

Hey @tlmn! Nice to hear from you

I’m actually wrapping up some things to finalise a “stable release” of QGIS https://github.com/specklesystems/admin/issues/216

I’ve been implementing the Receiving part of things, and updated the UI with better controls (including send/receive to/from a specific branch).

This was already doable before, just by pasting the URL of the branch, but now it will be easier to manage.

I also just commented on your PR (sorry I had it a bit abandoned…) to check if there was anything else pending to move it out of WIP status, if not, i’ll be happy to merge it in right away so it will be available on the following release :slight_smile:

Once we have sending/receiving working, plans will depend on community feature requests :wink:

2 Likes

Sounds awesome!
I am sure we can provide you with feature requests in the future… :wink:

2 Likes

Do keep them coming!! We also accept “hate mail” when things don’t work :wink: :raised_hands:t3: Thx for the great contribution by the way!

1 Like

Hi there, thanks for you great work!

I admit I still have to properly test the connector (especially the latest update), but I would like to share my use case.

I’m using QGIS as a base to prepare/collect data in urban environments (buildings, structures, utilities) near a planned tunnel excavation; the data is then exported (via CSV=>Excel with WKT geometry :fearful: ) and used in an internal python tool that performs a Building Risk Assessment and outputs Excel, SHP and DXF files for further analysis and reports.

The option to move away from Excel data exchange has been in the to do list for ages, and I planned to build a qgis plugin for this. Now that I found you :heart:, I suppose I can exploit your work (and help you as much as I can, of course!)

This is how the connector can be useful:

  • import the alignment from Civil3D data (or other CAD software, the important thing is to know the CRS used);
  • Import available CAD drawings geometries (mostly buildings - now it’s a manual task using QGIS dwg/dxf import)
  • Integrate Excel data: usually my geotechnical colleagues are not fond of working in QGIS, and are happier editing the exported Excel file to update structures information (building heights, foundations elevation, vulnerability classes, and so on), but these results in data drift.
    Our current excel “data model” is to have one layer (type of structure) per excel sheet (since they have slightly different parameters to populate) - we usually have the code attribute to uniquely identify a structure (and allow us to do data joining).

We also planned to implement other workflows to bring GIS and BIM data together and have better sense of the project constraints; I suppose they will mostly use the sender part of the connector, but a 2 way data flow is not to exclude. I will report new use cases as soon as I discover them.

From my first tests I ran into some troubles (QGIS 3.20):

  • I tried to receive the alignment sent by Civil 3D connector in an empty QGIS project, but nothing comes up (speckle log says "successfully received "). I suppose creating new layers/features is not yet supported, isn’t it?
  • I created a new branch in the web app to send some QGIS feature, but the (empty) branch doesn’t show in the branch dropdown, even after clicking “Reload” multiple times - I had to close and reopen QGIS
  • QGIS geometries are successfully received by Civil3D connector, but layer names are somewhat cryptic
    • linestring features are in the first layer
    • (multi)polygons are both in $boundary and $displayValue layers (as closed polylines)
  • I tinkered a bit with the QGIS <-> Excel round trip without success, but then I saw this discussion and realized that the excel connector is not made for property updates! I’ll try to force my colleagues to use QGIS for the time being :stuck_out_tongue:

That said, keep up the good work!

7 Likes

Welcome to our community and thanks a lot for the feedback @sanzoghenzo !

It’s really helpful when people explain their use cases in detail so we can tailor our testing and future planning to their needs.

I have to admit the QGIS <> Civil3D is not a flow that we have tested much - if not at all, but it sounds like the issues you’ve found might be simple fixes on our end! @AlanRynne and @clrkng can track these internally and we’ll let you know when they’re fixed.

In regards to Excel, that thread is quite Revit specific and it might be indeed possible to update some QGIS properties. Do you have a very simple model or stream for us to test with?

3 Likes

Thanks @teocomi for you reply!

Here I created a stream with some buildings of my town downloaded from OSM.

Somehow I’ve managed to mess the features attributes, but it can be a good example of the use case: being able to make edits to the attributes in excel and send it back to QGIS.

Unfortunately the data chunks and __closure fields don’t play well with this kind of task!

For the Civil 3D → QGIS part, if I understood the current code correctly, the qgis plugin only looks for Layer objects (and its children) from the stream, so it only works with its own data (and indeed I was able to copy the polygons to an empty qgis project).

For my specific use case, I don’t care much if the received alignment(s) will be placed in a new or existing layer, but I can imagine some cases where, e.g., CAD layers need to be transposed to qgis layers - if they have only a single geometry type!

1 Like

Hello,

I am trying to install the speckle plugin for QGIS.
Running into the same issue than @tlmn
I did pip install specklepy. Is it because my version of QGIS is too recent ? Currently installing 3.16.13

2 Likes

Hi @pauldotnet

Are you trying to install the connector via the package manager from QGIS? or directly from code?

We added a dependency installer that should check if specklepy is installed upon load and if it isn’t it should install it itself (or upgrade it if needed)

Not sure about the QGIS version, I use 3.22 but I’m aware Kateryna does use 3.16 so in theory it should be fine with both.

If you’re installing the connector directly from the github repo, try installing the latest main branch.

EDIT: Also, what system are you running QGIS from?

thanks @AlanRynne for the instant reply. I tried to install from QGIS’ package manager. Will try from the github repo then, hopefully the dependency installer with solve this.

If you mean Mac or Windows, I am using Windows

I’ll try to debug on your specific version of QGIS to check if it has anything to do with that.

The latest version in the QGIS package manager should have the dependency installer… so I’m leaning towards considering this an unknown bug for now. Thanks for bringing it up.

On a sidenote, just to try to pinpoint the issue a bit more. In windows there are several ways to install QGIS afaik, I know of:

  • Using the Standalone Version installer
  • Using the OSGEO4W installer
  • Maybe there’s more?

Which one did you use? This will help me reproduce your steps and hopefully get the same issue

Thank you for investigating further. I installed QGIS from this Standalone

2 Likes