QGIS Connector WIP

I tried both now:

  1. fresh install (v0.0.3-beta)
  2. v0.0.2-beta overwritten by v0.0.3-beta

You are referring to the second method right? :wink:
Still getting a similar error message:

1 Like

Thanks! Not sure what could be going on there. I’ll clean my QGIS install and try to reproduce your error :wink:

1 Like

Thanks Alan!

Just tried with QGIS 3.20.1 and getting the same results…

1 Like

Just as a follow up while I check this, how are you installing QGIS? Through the normal installer or the OSGeo4W one?

Could you confirm what version of python it comes with?

I was using the latter (https://qgis.org/downloads/QGIS-OSGeo4W-3.20.3-1.msi) and it comes with Python 3.9

1 Like

Hey @tlmn, I could not manage to reproduce the error, but sometime this afternoon I’ll be releasing a new version that will automatically install the dependencies for you on the first load of the plugin. Hopefully this will fix the issue you are having that reports a missing dependency. :+1:t3: I’ll ping you here once it’s ready to try out.

This will also mean that soon we’ll have the Speckle plugin published in the marketplace as experimental

3 Likes

Hi @AlanRynne, now that the speckle connector works I was trying to do some debugging in vs code, but I get the following error when I try to attach the process: 'PyDB' object has no attribute 'dispose_and_kill_all_pydevd_threads'

Any ideas what might be going wrong here?
The debugvs plugin for QGIS is up and running and I created a virtual environment in vs code.

Actually I have also trouble installing specklepy in the QGIS python environment:

C:\Program Files\QGIS 3.20.3\apps\Python39>python.exe -m pip install specklepy
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Requirement already satisfied: specklepy in c:\users\tilman.reinhardt\appdata\roaming\python\python39\site-packages (2.3.5)
Requirement already satisfied: appdirs<2.0.0,>=1.4.4 in c:\users\tilman.reinhardt\appdata\roaming\python\python39\site-packages (from specklepy) (1.4.4)
Requirement already satisfied: ujson<5.0.0,>=4.1.0 in c:\users\tilman.reinhardt\appdata\roaming\python\python39\site-packages (from specklepy) (4.2.0)
Requirement already satisfied: pydantic<2.0.0,>=1.7.3 in c:\users\tilman.reinhardt\appdata\roaming\python\python39\site-packages (from specklepy) (1.8.2)
Requirement already satisfied: gql[all]>=3.0.0a6 in c:\users\tilman.reinhardt\appdata\roaming\python\python39\site-packages (from specklepy) (3.0.0b0)
Requirement already satisfied: graphql-core<3.2,>=3.1.5 in c:\users\tilman.reinhardt\appdata\roaming\python\python39\site-packages (from gql[all]>=3.0.0a6->specklepy) (3.1.6)
Requirement already satisfied: yarl<2.0,>=1.6 in c:\users\tilman.reinhardt\appdata\roaming\python\python39\site-packages (from gql[all]>=3.0.0a6->specklepy) (1.7.0)
Requirement already satisfied: urllib3>=1.26 in c:\users\tilman.reinhardt\appdata\roaming\python\python39\site-packages (from gql[all]>=3.0.0a6->specklepy) (1.26.7)
Requirement already satisfied: aiohttp<3.8.0,>=3.7.1 in c:\users\tilman.reinhardt\appdata\roaming\python\python39\site-packages (from gql[all]>=3.0.0a6->specklepy) (3.7.4.post0)
Requirement already satisfied: requests-toolbelt<1,>=0.9.1 in c:\users\tilman.reinhardt\appdata\roaming\python\python39\site-packages (from gql[all]>=3.0.0a6->specklepy) (0.9.1)
Requirement already satisfied: requests<3,>=2.26 in c:\users\tilman.reinhardt\appdata\roaming\python\python39\site-packages (from gql[all]>=3.0.0a6->specklepy) (2.26.0)
Requirement already satisfied: websockets<10,>=9 in c:\users\tilman.reinhardt\appdata\roaming\python\python39\site-packages (from gql[all]>=3.0.0a6->specklepy) (9.1)
Requirement already satisfied: typing-extensions>=3.6.5 in c:\users\tilman.reinhardt\appdata\roaming\python\python39\site-packages (from aiohttp<3.8.0,>=3.7.1->gql[all]>=3.0.0a6->specklepy) (3.10.0.2)
Requirement already satisfied: chardet<5.0,>=2.0 in c:\program files\qgis 3.20.3\apps\python39\lib\site-packages (from aiohttp<3.8.0,>=3.7.1->gql[all]>=3.0.0a6->specklepy) (3.0.4)
Requirement already satisfied: multidict<7.0,>=4.5 in c:\users\tilman.reinhardt\appdata\roaming\python\python39\site-packages (from aiohttp<3.8.0,>=3.7.1->gql[all]>=3.0.0a6->specklepy) (5.2.0)
Requirement already satisfied: attrs>=17.3.0 in c:\users\tilman.reinhardt\appdata\roaming\python\python39\site-packages (from aiohttp<3.8.0,>=3.7.1->gql[all]>=3.0.0a6->specklepy) (21.2.0)
Requirement already satisfied: async-timeout<4.0,>=3.0 in c:\users\tilman.reinhardt\appdata\roaming\python\python39\site-packages (from aiohttp<3.8.0,>=3.7.1->gql[all]>=3.0.0a6->specklepy) (3.0.1)
Requirement already satisfied: idna<4,>=2.5 in c:\program files\qgis 3.20.3\apps\python39\lib\site-packages (from requests<3,>=2.26->gql[all]>=3.0.0a6->specklepy) (2.10)
Requirement already satisfied: certifi>=2017.4.17 in c:\program files\qgis 3.20.3\apps\python39\lib\site-packages (from requests<3,>=2.26->gql[all]>=3.0.0a6->specklepy) (2020.6.20)
Requirement already satisfied: charset-normalizer~=2.0.0 in c:\users\tilman.reinhardt\appdata\roaming\python\python39\site-packages (from requests<3,>=2.26->gql[all]>=3.0.0a6->specklepy) (2.0.7)
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping

Looks like it is related to windows and this issue:

Edit: I think I solved this myself.

  1. specklepy was already available…
    Requirement already satisfied: specklepy in c:\users\..

  2. …and when I use the OSGeo4W shell with admin rights I am allowed to install specklepy or other modules…

I also seem to have problems sending any objects at all (sorry for spamming this thread today).

I have selected elements in the view and picked the according layers, but the stream seems to be always empty: example Speckle stream

Is there another way than debugging in vs code to check what might go wrong in this case?

Hi @tlmn! Sorry for the late reply, I was out for a long weekend :slight_smile:

Wow! No need to apologise for this :slight_smile: All error reports are super helpful!

So, lets go one at a time:

‘PyDB’ object has no attribute ‘dispose_and_kill_all_pydevd_threads’

Are you trying to debug following the instructions in our docs? I have a pending task to update them this week, and release a more comprehensive guide on how the QGIS connector works :slight_smile: Anyway, I refactored the debug functionality into it’s own function and commented it out so “normal” users wouldn’t have that extra dependency automatically installed, but this really only applies to Mac users (as you can pick the process in windows directly).

I’m fairly certain the “Mac way” should also work on windows, have you tried that one. Just uncomment one this line (if you haven’t already found it by yourself :wink: ) in the _init_.py file

That should take care of setting up ptvsd, then you’d just have to start your launch.json configuration as specified in the guide (the mac one, which specifies a port to connect to, instead of connecting to a process). I hope that works but I’ll look into what could possibly be going on there.

Actually I have also trouble installing specklepy in the QGIS python environment:

Happy you solved it! But let me ask you a couple of questions just for future reference.

  1. Where you trying to install specklepy manually, or did this happen automatically when opening QGIS? In order to ensure users had pipeinstalled, I added the get_pip.py module. If it’s giving trouble I could look into alternative solutions.
  2. I assume you solved by accessing the OSGeo shell with admin rights and manually installing specklepy? Once you did that, did you get any other errors from speckle-qgis?

I also seem to have problems sending any objects at all (sorry for spamming this thread today).

:thinking: not sure what could be going on there, seems like the entire conversion is being skipped. What type of layers are they?

Sorry for the long reply! And thanks again for reporting all of this! I’m hoping by the end of the week to have a specific section in our docs for QGIS and if possible, uploaded to the QGIS plugin marketplace (as experimental for now)

1 Like

Hi @AlanRynne, thx for the long answer, really appreciated! :slight_smile:

Just uncomment one this line (if you haven’t already found it by yourself :wink: ) in the _init_.py file

The “Mac way” does indeed work for windows too, although I didn´t need to uncomment that line, actually I get an error when I do so…

CRITICAL    Traceback (most recent call last):
              File "C:\PROGRA~1/QGIS32~1.3/apps/qgis/./python\qgis\utils.py", line 335, in _startPlugin
              plugins[packageName] = package.classFactory(iface)
              File "C:\Users\tilman.reinhardt\git\speckle-qgis\__init__.py", line 43, in classFactory
              enable_remote_debugging()
             NameError: name 'enable_remote_debugging' is not defined

I also pointed QGIS_PLUGINPATH to my QGIS git repo, not sure if that is necessary.

  1. Where you trying to install specklepy manually, or did this happen automatically when opening QGIS? In order to ensure users had pipeinstalled, I added the get_pip.py module. If it’s giving trouble I could look into alternative solutions.

I think I installed specklepy somewhere on the way globally, removed it later and re-installed it in the OSGeo4W shell with admin rights.

  1. I assume you solved by accessing the OSGeo shell with admin rights and manually installing specklepy? Once you did that, did you get any other errors from speckle-qgis?

Yes, but actually I think speckle-qgis initially took care of all the dependencies successfully, I was just confused because I got this ssl error when I tried installing from the QGIS_PYTHON_PATH (and debugging didn´t work and I couldn´t send anything… :wink: )

:thinking: not sure what could be going on there, seems like the entire conversion is being skipped. What type of layers are they?

Now that debugging works I can see that my layers are missing an attribute fields. I will investigate and report back here.

1 Like

Oh… from my initial research I had assumed that all layers in QGIS do have fields with data (even if empty), as usually all entities had an id that would be given back. Maybe this is not the case! It would be super helpful if you could share with us the layer you’re testing (if this is a possibility) so I can include it in further testing before releases.

Regarding the debug error, I think the final cleanup removed the “unused” import of the enable_remote_debug function. I’ll add that too in the future to the commented section so nobody forgets. But try adding this at the top of your file (or right before the enable_remote_debugging call)

from .speckle.utils import enable_remote_debugging

Although if you have it working without that call, I’d love to know how! The current debugging solution we found is not as elegant as we would like (it’s just the one we got to work) :slight_smile:

Regarding specklepy, if you go for the venv route, I think you need to install it in the venv as well as having it installed in QGIS (which is now handled automatically). The venv install would be used by VSCode for autocompletion, while the QGIS one will be the one that actually runs.

Any better/simpler solutions for dev setup are welcome to simplify/unify the whole process for everyone.

I kind of only was trying to follow the steps in the docs :wink:

  1. QGIS_PYTHON_PATH pointing at git repo (maybe that makes the difference?),
  2. virtual environment in vs code with specklepy and ptvsd installed (as you pointed out above),
  3. debugvs plugin in QGIS (instead of Remote Debugger)
  4. and using the “Mac way” to connect vs code with QGIS

Oh… from my initial research I had assumed that all layers in QGIS do have fields with data (even if empty), as usually all entities had an id that would be given back. Maybe this is not the case! It would be super helpful if you could share with us the layer you’re testing (if this is a possibility) so I can include it in further testing before releases.

I am not sure if I can share the layer publicly, but I will check if that is possible.

I think one problem is that my project.layerTreeRoot().children() contain groups not layers, so I have to dig further to get to the layers.

def getLayers(tree, parent):
    children = parent.children()
    layers = []
    for node in children:
        isLayer = tree.isLayer(node)
        if isLayer:
            layers.append(node)
            continue
        isGroup = tree.isGroup(node)
        if isGroup:
            layers.extend(getLayers(tree, node))
    return layers

I also created a pull request for that: WIP: recursively get children by tlmnrnhrdt · Pull Request #12 · specklesystems/speckle-qgis · GitHub

Can you point me to some documentation regarding accessing the “fields”? So far I only came across the python API (Welcome to the QGIS Python API documentation project)

Hi @Community,

at the moment the QGIS connector only converts objects from Class: QgsVectorLayer to speckle (which it does pretty great btw).

But we would also want to convert Class: QgsRasterLayer objects!

For example something like the following height map…

…which is basically stored as a very long list of heights (3500 x 3500 = ~12 million/~100MB in this case) in an ascii file:

ncols        3500
nrows        3500
xllcorner    301000.000000000000
yllcorner    192000.000000000000
cellsize     2.000000000000
NODATA_value  -9999.000
 288.329 289.472 290.618 293.196 295.113 296.552 ..
 287.940 289.254 290.376 292.500 294.467 296.406 ..
 ..

So I guess we would need some kind of raster layer object in speckle too?! What do you think?
Also do you see issues with the size of the stream?

1 Like

I totally agree! Although I’m not sure about how this would be supported in other programs that don’t really “understand” height maps (thinking of things like Revit?), which is the main reason they were left out of the initial development :slight_smile:

Although I suspect just being able to extract the raw data may be useful in itself too! Would definitely love to hear more from the community to see how this could be used in different workflows!

Looks like I am running into issues sending 100MB strings…
According to Decomposition API | Speckle Docs I need to tell my property that it´s chunkable. Does that work with a string?

public class RasterLayer : Base
{
    [DetachProperty]
    [Chunkable(10000)] // Can a string be chunked?
    public string raster { get; set; }
}

Or rather use a list?

public class RasterLayer : Base
{
    [DetachProperty]
    [Chunkable(10000)] 
    public List<string> raster { get; set; } = new List<string>();
}

Is there an example around for specklepy sending stuff in chunks?

1 Like

Strings are funny because they’re arrays, but we don’t want to chunk them by default, as it doesn’t make sense for “hello world”, which is the most usecases :smiley: We basically treat them as any other primitive (which they’re not really).

The first option won’t chunk anything; the second option will chunk the list itself, but not the strings inside the list - so you’ll need to do your own “pre-chunking” of your a massive string into the list itself. Does this make sense? I’m assuming you have one huuuuuuge string here that you want split up in parts.

2 Likes

Yes! you can find one in our Developer guide, here :point_right:t3: Examples | Speckle Docs

1 Like

The first option won’t chunk anything; the second option will chunk the list itself, but not the strings inside the list - so you’ll need to do your own “pre-chunking” of your a massive string into the list itself. Does this make sense? I’m assuming you have one huuuuuuge string here that you want split up in parts.

That totally makes sense :slight_smile:

In specklepy that would probably look something like this?

class RasterLayer(Base, speckle_type="Objects.Geometry." + "RasterLayer", chunkable={"Raster": 10000}, detachable={"Raster"}):
    Raster: List[str] = None

    @classmethod
    def from_list(cls, args: List[Any]) -> "RasterLayer":
        return cls(
            Raster=args,
        )

    def to_list(self) -> List[Any]:
        encoded = []
        for line in self.Raster:
            encoded.append(line)
        return encoded

Although it looks like we need to change some server settings first.

Upload server response: File size too large (97716266 > 52428800)
Error sending batch of objects to server: SpeckleException: Could not save the object to the server - status code 400
2 Likes