Let me know if you would like to see any additional logs and thanks for your help …
*I’m unable to attach front-end log because it exceeds number of allowable characters.
`speckle-server’ log:
[INFO wait] --------------------------------------------------------
[INFO wait] docker-compose-wait 2.8.0
[INFO wait] ---------------------------
[DEBUG wait] Starting with configuration:
[DEBUG wait] - Hosts to be waiting for: [postgres:5432, redis:6379]
[DEBUG wait] - Timeout before failure: 30 seconds
[DEBUG wait] - TCP connection timeout before retry: 5 seconds
[DEBUG wait] - Sleeping time before checking for hosts availability: 0 seconds
[DEBUG wait] - Sleeping time once all hosts are available: 0 seconds
[DEBUG wait] - Sleeping time between retries: 1 seconds
[DEBUG wait] --------------------------------------------------------
[INFO wait] Checking availability of postgres:5432
[INFO wait] Host postgres:5432 is now available!
[INFO wait] --------------------------------------------------------
[INFO wait] Checking availability of redis:6379
[INFO wait] Host redis:6379 is now available!
[INFO wait] --------------------------------------------------------
[INFO wait] docker-compose-wait - Everything's fine, the application can now start!
[INFO wait] --------------------------------------------------------
2022-08-15T15:26:16.603Z speckle:db-startup Loaded knex conf for production
2022-08-15T15:26:16.924Z speckle:www Error: The migration directory is corrupt, the following files are missing: 20220803104832_ts_test.js
at validateMigrationList (/speckle-server/node_modules/knex/lib/migrations/migrate/Migrator.js:566:11)
at Migrator.latest (/speckle-server/node_modules/knex/lib/migrations/migrate/Migrator.js:69:7)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async exports.init (/speckle-server/packages/server/app.js:98:3)
[INFO wait] --------------------------------------------------------
[INFO wait] docker-compose-wait 2.8.0
[INFO wait] ---------------------------
[DEBUG wait] Starting with configuration:
[DEBUG wait] - Hosts to be waiting for: [postgres:5432, redis:6379]
[DEBUG wait] - Timeout before failure: 30 seconds
[DEBUG wait] - TCP connection timeout before retry: 5 seconds
[DEBUG wait] - Sleeping time before checking for hosts availability: 0 seconds
[DEBUG wait] - Sleeping time once all hosts are available: 0 seconds
[DEBUG wait] - Sleeping time between retries: 1 seconds
[DEBUG wait] --------------------------------------------------------
[INFO wait] Checking availability of postgres:5432
[INFO wait] Host postgres:5432 is now available!
[INFO wait] --------------------------------------------------------
[INFO wait] Checking availability of redis:6379
[INFO wait] Host redis:6379 is now available!
[INFO wait] --------------------------------------------------------
[INFO wait] docker-compose-wait - Everything's fine, the application can now start!
[INFO wait] --------------------------------------------------------
2022-08-17T01:26:22.662Z speckle:db-startup Loaded knex conf for production
2022-08-17T01:26:23.094Z speckle:www Error: The migration directory is corrupt, the following files are missing: 20220803104832_ts_test.js
at validateMigrationList (/speckle-server/node_modules/knex/lib/migrations/migrate/Migrator.js:566:11)
at Migrator.latest (/speckle-server/node_modules/knex/lib/migrations/migrate/Migrator.js:69:7)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async exports.init (/speckle-server/packages/server/app.js:98:3)
this is probably a mixup between different versions of the server container. If you haven’t added any data to the server instance, I suggest deleting the postgresql database. You can do that, by either deleting the docker volumes attached with docker-compose down -v
Or you can use any database management frontend to delete and recreate the speckle database.
I do have data in the DB I need to keep. Is there another way around this? This server is just a sandbox mostly for staging and testing, but I do have an active server being used on a project that I do not have the luxury of killing the data in the DB.
Are you using the same database or the same database server?
I highly recommend not sharing the databases between different server instances. See this tutorial for some context.
If you are sharing the same database, these sorts of problems can happen, if a newer version migrates the database schema, the older (most probably, your production server) wont be able to run anymore.
If you are not sharing the database between installations, you can just create a new database for the staging/testing server specifically and point your test server to that new DB.
~TOP SECRET~
There is actually a way to do it, but its a bit more involved and risky.
WARNING WARNING WARNING
if the operation is not done right, it can break your app state.
The knex_migrations table stores the successful migration file names, that ran on the DB.
If you delete the row, that contains 20220803104832_ts_test.js, your app will be able to run.
Note, this is only true, to this specific case, cause this migration is not actually changing anything in the DB.
I’m also facing a similar problem now when I upgrade to the latest version of speckle I get this error in the server log :
[INFO wait] --------------------------------------------------------
[INFO wait] docker-compose-wait 2.8.0
[INFO wait] ---------------------------
[DEBUG wait] Starting with configuration:
[DEBUG wait] - Hosts to be waiting for: [postgres:5432, redis:6379]
[DEBUG wait] - Timeout before failure: 30 seconds
[DEBUG wait] - TCP connection timeout before retry: 5 seconds
[DEBUG wait] - Sleeping time before checking for hosts availability: 0 seconds
[DEBUG wait] - Sleeping time once all hosts are available: 0 seconds
[DEBUG wait] - Sleeping time between retries: 1 seconds
[DEBUG wait] --------------------------------------------------------
[INFO wait] Checking availability of postgres:5432
[INFO wait] Host postgres:5432 not yet available...
[INFO wait] Host postgres:5432 not yet available...
[INFO wait] Host postgres:5432 is now available!
[INFO wait] --------------------------------------------------------
[INFO wait] Checking availability of redis:6379
[INFO wait] Host redis:6379 is now available!
[INFO wait] --------------------------------------------------------
[INFO wait] docker-compose-wait - Everything's fine, the application can now start!
[INFO wait] --------------------------------------------------------
2022-11-04T17:32:44.473Z speckle:db-startup Loaded knex conf for production
2022-11-04T17:32:48.646Z speckle:www Error: The migration directory is corrupt, the following files are missing: 20220727091536_blobs-id-length-removal.js, 20220825123323_usernotificationpreferences.js, 20220829102231_add_server_access_requests_table.js, 20220921084935_fix_branch_nullability.js, 20220929141717_scheduled_tasks.js
at validateMigrationList (/speckle-server/node_modules/knex/lib/migrations/migrate/Migrator.js:566:11)
at Migrator.latest (/speckle-server/node_modules/knex/lib/migrations/migrate/Migrator.js:69:7)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async init (/speckle-server/packages/server/dist/app.js:128:5)
How can this be resolved without deleting data from the current DB?
Strangely enough - I managed to fix it. I think this is what happened:
I upgraded to the latest version of speckle by setting the docker-compose.yml to :latest on both front-end and server (2.9). When I checked, the site did not work (but I did not check the logs).
In panic, I downgraded to the previous version we had installed 2.5, this did not work either.
Here is where I checked the logs and posted above declaring a Knex Corruption.
So. I’ve upgraded again to the latest (2.9) and logged again - this time I’m getting a S3_ACCESS_KEY error!
I’ve added the all the variables in the server environment section in the docker-compose.yml, like so and everything seems to be working now.
Glad you fixed it @dvalent - thanks for writing up how you resolved it.
As a slightly more detailed explanation of what happened ‘under the hood’. On upgrading from 2.5 to latest, Speckle undertook some database schema changes (migrations) and recorded the filenames that it used to undertake these migrations in the database. When you downgraded, the migration records are retained but the files don’t exist in the previous version which results in the errors you saw.
The S3_ACCESS_KEY error is because we added, or changed, the configuration or its defaults between versions.
As you probably already realised, for future upgrades, you could cross-reference your configuration against changes in the example file to identify potential issues. I’d also recommend keeping an eye on the logs as it is deploying. It can also be helpful to test the upgrade process first, by deploying a second Speckle server with the current version you have (2.5 in your case) and practice the upgrade process separately from the server & database you have your actual data in.
