Hi,
I’m currently running Speckle app on my local server (I’ve set it up with docker-compose as per Docs).
My version is 2.7.1 but I would like to upgrade it to most current version - is there a way to do the upgrade without losing data? (i.e. stream data and user accounts)
I suspect that’s this guide then.
Don’t believe a word I say before the devops team confirms (@iainsproat/@gjedlicska), but if I am not mistaken DB data will persist throughout the update, even if postgres is deployed as part of the docker compose file.
1 Like
It should be preserved, but for safety it’s always good practice to back up the data first.
In Step 2 of the guide, the docker-compose file mounts a local volume for storing the postgres data. (see below exerpt).
volumes:
- ./postgres-data:/var/lib/postgresql/data/
When upgrading the database, docker should mount the same volume to the new container and the data should be preserved.
If you wish, you can back up the volume, ./postgres-data, on your local server before performing any actions. I’d also recommend regularly backing up this data if you have any business-critical information stored in Speckle. Docker has more guidance on volumes and backups in their documentation.
2 Likes
Thanks Guys,
I missed that volume mounting on the Postgres service - with that in place it should be OK
(I understand it’s only Postgres service that stores the data).
Hi @iainsproat
I’m just trying to deploy the new version of the server following the guide you mentioned above, but for login page (/authn/login) on the frontend I get “Invalid credentials”, when I then go to register (/authn/register?token) it responds with either “failed to fetch” or “Email taken. Try logging in?”
I’m running the server on http (non-ssl)
When I check frontend service logs it shows (the “python-request” bit seems to keep pinging the server all the time):
192.168.10.157 - - [05/Oct/2022:15:23:55 +0000] "GET /graphql HTTP/1.1" 101 87 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"
192.168.10.157 - - [05/Oct/2022:15:24:06 +0000] "GET /graphql HTTP/1.1" 101 87 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"
172.29.24.1 - - [05/Oct/2022:15:24:13 +0000] "GET / HTTP/1.1" 200 4662 "-" "python-requests/2.25.0"
Hey @simplynail
could you attach the logs from the speckle-server container too? That would help us diagnosing the issue.
Hi,
looks like there has been some issue with container manager on my NAS, and the data storage was actually removed.
I managed to get that data restored, it looks like frontend is to some extent capable of getting data from the database (it now shows on frontend login page that server is “invite only” which is not default behavior but I have changed that previously on my instance to be invite only so it must be getting that info from restored postgres(?) ).
But now I get “failed to fetch” on login page when I try to login
frontend logs:
192.168.10.139 - - [06/Oct/2022:08:06:23 +0000] "GET /graphql HTTP/1.1" 101 87 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"
192.168.10.139 - - [06/Oct/2022:08:06:31 +0000] "GET /graphql HTTP/1.1" 101 87 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"
192.168.10.139 - - [06/Oct/2022:08:06:31 +0000] "POST /auth/local/login?challenge=5rwcgwm1i8 HTTP/1.1" 302 99 "http://192.168.10.177:81/authn/login?token" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"
192.168.10.139 - - [06/Oct/2022:08:06:32 +0000] "POST /auth/local/login?challenge=5rwcgwm1i8 HTTP/1.1" 302 99 "http://192.168.10.177:81/authn/login?token" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"
192.168.10.139 - - [06/Oct/2022:08:06:33 +0000] "GET /graphql HTTP/1.1" 101 87 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"
192.168.10.139 - - [06/Oct/2022:08:06:34 +0000] "GET /graphql HTTP/1.1" 101 87 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"
172.29.32.1 - - [06/Oct/2022:08:06:37 +0000] "GET / HTTP/1.1" 200 4662 "-" "python-requests/2.25.0"
server container logs (nothing extra shows after the startup, even when I make requests from frontend):
[INFO wait] docker-compose-wait - Everything's fine, the application can now start!
[INFO wait] --------------------------------------------------------
2022-10-06T07:51:37.940Z speckle:db-startup Loaded knex conf for production
2022-10-06T07:51:45.039Z speckle:modules 💥 Init core module
2022-10-06T07:51:45.251Z speckle:modules 🔑 Init auth module
2022-10-06T07:51:45.380Z speckle:modules 💅 Init graphql api explorer module
2022-10-06T07:51:45.381Z speckle:modules 📧 Init emails module
2022-10-06T07:51:45.382Z speckle:modules 📧 Email provider is not configured. Server functionality will be limited.
2022-10-06T07:51:45.392Z speckle:modules ♻️ Init pwd reset module
2022-10-06T07:51:45.401Z speckle:modules 💌 Init invites module
2022-10-06T07:51:45.429Z speckle:modules 📸 Init object preview module
2022-10-06T07:51:45.430Z speckle:modules 📄 Init FileUploads module
2022-10-06T07:51:45.431Z speckle:modules 🗣 Init comments module
2022-10-06T07:51:45.441Z speckle:modules 📦 Init BlobStorage module
2022-10-06T07:51:45.561Z speckle:modules 📞 Init notifications module
2022-10-06T07:51:45.561Z speckle:modules 📞 Initializing notification queue consumption...
2022-10-06T07:51:46.677Z speckle:modules 🤺 Init activity module
2022-10-06T07:51:46.686Z speckle:modules 🔐 Init access request module
2022-10-06T07:51:47.850Z speckle:startup 🚀 My name is Speckle Server, and I'm running at 0.0.0.0:3000
I checked also Speckle Manager - it looks like it can communicate again with server with no problem based on old credentials (it shows count of streams and commits for that server under Accounts tab)
Just to drop here a note if anybody has similar issue:
I needed to expose Speckle over different port (than standard 80).
to do that I changed the following:
speckle-frontend:
...
ports:
- "0.0.0.0:81:80" # changed exposed port to 81 from standard "0.0.0.0:80:80"
however if you do that change, you also need to change the canonical url to include port as well:
speckle-server:
...
environment:
# TODO: Change this to the URL of the speckle server, as accessed from the network
CANONICAL_URL: "http://192.168.10.177:81" # note ":81" added after the IP/host url
One thing extra I learned is after you spin things up, all the communication you will be then doing with your Speckle instance will be through port 81 (other ports from containers are not exposed).
So regardless if you need to:
- interact with Speckle via its frontend,
- do speckle-py api calls or
- connect your Speckle Manager,
it always needs to go via CANONICAL_URL (with port added at the end for my case).
After the upgrade, the new instance is able to use persisted data from previous setup (as long as the data is still present under the same folder)
thanks @gjedlicska for taking your time to get that solved with me !
3 Likes
Since this discussion we have written a guide on how to backup, upgrade, and restore the database: Database backup, upgrade, and restore | Speckle Docs
I hope this more detailed guide helps anyone who needs to do this in the future.
2 Likes
Hi Iain,
following your documentation about db backup I had some problems.
We use a manual installation of speckle, which is up and running.
The file docker-compose-pgadmin.yml from step 3 throws these errors:
- while scanning a simple key in “./docker-compose-pgadmin.yml”, line 8, column 7
- could not find expected ‘:’ in “./docker-compose-pgadmin.yml”, line 9, column 7
Since a sheer "admin@localhost.com makes no sense in this position I commented out this line and get: - Service ‘pgadmin’ depends on service ‘postgres’ which is undefined.
Then I tried to load both yml-files:
docker-compose -f docker-compose.yml -f docker-compose-pgadmin.yml up --detach
and docker compose complains about different versions of those two files.
Last try was:
delete the section depends_on in docker-compose-pgadmin.yml and start this container:
docker-compose -f docker-compose-pgadmin.yml up --detach
Now it works and I can load pgadmin via tunnel from local machine!
Bernhard
1 Like
From pgadmin it was not possible, to connect to the db-server. The server name could not be resolved.
After I put all docker-compose settings in one single file, pgadmin could connect to the server.
Bernhard
1 Like
Thanks for the feedback @bernhard, it’s extremely useful!
I think the depends_on section can be removed, which should allow it to be in a separate file. I’ll try and verify this later.
depends_on:
- postgres
The additional admin@localhost.com is strange. You are correct in saying it should not be there. It doesn’t appear in the source code so is possibly a rendering artefact when the html is built. I’ll have to investigate this more thoroughly. In the meantime, you will have to manually remove it as you mentioned.
1 Like
The documentation is now updated, with the depends_on removed and the duplicate admin@localhost.com fixed.
Thank you again for reporting these problems @bernhard
2 Likes
Some more annotations:
- The backup should have the Options “Include CREATE DATABASE statement” and “Include DROP DATABASE statement” ckecked. This makes restore easier.
- Without these two options set, a restore brings up lot’s of errors because some elements already exist. The database should be deleted before restore.
- It’s not possible to delete or restore a database as long as it is the “Maintenance database” for pgAdmin.
Workaround in pgAdmin:
– Create another (may be empty) database.
– Disconnect from server.
– Edit server properties and on the connection tab set the other database as “Maintenance database”.
– Connect to server.
Now one can restore the speckle backup.
4 Likes
Thanks again for the excellent feedback @bernhard , this is incredibly helpful.
I’ve updated the documentation to include your comments.
2 Likes