Creating a terrain from Google Satellite is hanging/very slow

[JoostGevaert]

Hey @Kateryna,
I’ve been trying to apply the Google Satellite on the Mapzen DEM + 3D buildings tutorial for an imaginary site in Hong Kong.
I created a 21.3 MB Google Satellite .tif with a resolution of 1m, and a 2.3 MB Mapzen DEM .tif with a resolution of 3 m. Both layers have the same CRS: EPSG:2326.
Sending this to Speckle is taking ages. It seems to be stuck on the ‘Converting Layer’ step:

Any ideas on how to solve this?

Maybe reducing the resolution of the Google Satellite .tif would help?
With a resolution of 2m the size already reduces to 2MB, but then the image is also considerably “shittier”, and I thought there weren’t really size limitations on Speckle?

[jonathon]

Is it reporting activity if you open the Python console and/or the Speckle console log?

It could be a slow process.

@JoostGevaert

Sending this to Speckle is taking ages.

Do you have a frame of reference for “ages”?

[jan-casas (LUIS CASAS)]

I suspect it’s a similar issue. Reducing the resolution significantly (~5m territorial scale with a resolution of 144 dpi) solved the problem for me.
Raster Image in QGIS - Help - Speckle Community

[JoostGevaert]

Thanks for the quick reply Jonathon,
I tried again just now. This time I’m using a google satellite image with a 2m resolution and a Digital Elevation Model (DEM) with a resolution of 3m.
I don’t get any error messages, and am receiving some progress updates, which I didn’t get last time:

Getting to 40% progress of the layer conversion took 15 minutes, which I think is very long.

Is there a way to preprocess this, and save the result?
Or would there be a way to do this with Speckle automate, rather than inside QGIS on my machine?

[JoostGevaert]

In total it took an hour:

But maybe it has something to do with these crazy spikes:

?

How can I best repair those spikes?
I’ll ask ChatGPT, but some advice from real experts would be much appreciated as well

https://speckle.xyz/streams/013aaf06e7/commits/e1223ae9ce

[jonathon]

Interesting, yes, it is currently not a quick process. It has been introduced as a POC to see if users would find it helpful. We can add a task to see where the processing can improve. Not sure about preprocessing.

I’m speculating, but removing those spikes could be algorithmically checked for - clearly, some are errors or sparse data points, and some smoothing via outlier checking could be implemented, but it would add another step in the process.

As with your other report, I can raise it, but I cannot guarantee it will be implemented quickly.

---

It could well be a good candidate for moving from the Connector to Automate. If you haven’t already registered as a beta programme candidate, perhaps you’d like to give that a whirl, fully armed with Chad Jippity.

[JoostGevaert]

The preprocessing question and especially the DEM outlier removal question were more QGIS questions, rather than Speckle questions, as I’m quite a QGIS noob. I didn’t mean to suggest them as Speckle features.

I’ll register for Speckle Automate beta

[jonathon]

Nevertheless, all good suggestions. Automate, too, it’s a good call and could improve the experience a lot.

[Kateryna]

These spikes look to me like No-data pixels… Would you mind sharing the original .tif for the elevation or share the screenshot of the Stats of the raster layer, showing no-data values for each band (in Layer Properties)? The resolution makes the “texturing” a long-running operation (not the spikes), but I’ll see if it can be optimized in any way

[JoostGevaert]

Hey @Kateryna,

Thanks for your reply!
Here is the .tif with 3m resolution I exported from the Mapzen Global Terrain:
KaiTak_Context_DEM_3m.tif (2.3 MB)

I don’t find the “Stats”. Are these these the layer properties you are asking for?

Is it the resolution of the DEM or the resolution of the satellite image that makes the texturing a long-running process?

[Kateryna]

These are the Bands statistics, tbh I was expecting (hoping) to see the “-1051” value in a No-Data column, then it would be clear which values to filter out In this case it’s a source data issue,

For now, I’d suggest to clean the data before sending, e.g. using Raster-> Raster Calculator:

P.S. The higher is the resolution of the satellite (texture) image, the longer it will take, in this case the size is over 1000x1000 cells, so more than a million little mesh faces.

[JoostGevaert]

Thanks @Kateryna!
Good to know it’s the image’s resolution that makes the texturing slow.
That raster calculator is a great trick, and I can indeed just set everything that’s below 0 to 0, because I’m not interested in the bathymetry anyway.

[JoostGevaert]

Pretty awesome stuff . I’m really amazed by this Speckle functionality:

https://speckle.xyz/streams/013aaf06e7/commits/dfbec71408?overlay=651e3058c7,a8f162b744,95b8e790dc&c=[837172.41513,820445.32437,351.26766,838430.67256,820130.32974,23.54064,0,1]