Is speckle container unhealthy?

Hi everyone, I’m facing an issue related to manual server deployment with docker compose up :crying_cat_face:

Here are the specificities:

  • Objective: Running my own instance of Speckle server on a Google Cloud Virtual Machine (Ubuntu 20.04.6 LTS, 20GB storage, 16GB RAM, 4 core CPU) with docker compose up as you can see here. My Docker version is 24.0.5 and my docker-compose version is v2.27.0
  • Issue: Everything is fine until docker has to run speckle-server container. The logs are:
WARN[0000] /opt/speckle/docker-compose.yml: `version` is obsolete 
[+] Running 9/9
 ✔ Container speckle-server-speckle-frontend-2-1  Started                                                                                                                                   0.9s 
 ✔ Container speckle-server-minio-1               Healthy                                                                                                                                   6.2s 
 ✔ Container speckle-server-redis-1               Healthy                                                                                                                                   6.7s 
 ✔ Container speckle-server-speckle-ingress-1     Started                                                                                                                                   1.1s 
 ✔ Container speckle-server-postgres-1            Healthy                                                                                                                                   6.2s 
 ✘ Container speckle-server-speckle-server-1      Error                                                                                                                                    91.0s 
 ✔ Container speckle-server-preview-service-1     Created                                                                                                                                   0.0s 
 ✔ Container speckle-server-fileimport-service-1  Created                                                                                                                                   0.0s 
 ✔ Container speckle-server-webhook-service-1     Created

And, when I inspect what went wrong with docker log speckle-server-speckle-server-1, I can see, repeated over and over, this error:

{"@l":"Information","phase":"db-startup","@t":"2024-06-06T09:00:04.206Z","@mt":"Loaded knex conf for production"}
{"@l":"Information","phase":"startup","@t":"2024-06-06T09:00:05.308Z","@mt":"🖼️  Serving for frontend-2..."}
{"@l":"Error","component":"redis","err":{"type":"Error","message":"connect ETIMEDOUT","stack":"Error: connect ETIMEDOUT\n    at Socket.<anonymous> (/speckle-server/node_modules/ioredis/built/Redis.js:168:41)\n    at Object.onceWrapper (node:events:631:28)\n    at Socket.emit (node:events:517:28)\n    at Socket.emit (node:domain:489:12)\n    at Socket._onTimeout (node:net:598:8)\n    at listOnTimeout (node:internal/timers:569:17)\n    at process.processTimers (node:internal/timers:512:7)","errorno":"ETIMEDOUT","code":"ETIMEDOUT","syscall":"connect"},"@t":"2024-06-06T09:00:15.347Z","@x":"Error: connect ETIMEDOUT\n    at Socket.<anonymous> (/speckle-server/node_modules/ioredis/built/Redis.js:168:41)\n    at Object.onceWrapper (node:events:631:28)\n    at Socket.emit (node:events:517:28)\n    at Socket.emit (node:domain:489:12)\n    at Socket._onTimeout (node:net:598:8)\n    at listOnTimeout (node:internal/timers:569:17)\n    at process.processTimers (node:internal/timers:512:7)","@mt":"Redis encountered an error."}

/speckle-server/packages/server/dist/modules/shared/redis/redis.js:18
                throw new errors_1.EnvironmentResourceError('Redis encountered an error.', err); //FIXME backoff and retry?
                ^
EnvironmentResourceError: Redis encountered an error.: connect ETIMEDOUT
    at EventEmitter.<anonymous> (/speckle-server/packages/server/dist/modules/shared/redis/redis.js:18:23)
    at EventEmitter.emit (node:events:517:28)
    at EventEmitter.emit (node:domain:489:12)
    at EventEmitter.silentEmit (/speckle-server/node_modules/ioredis/built/Redis.js:460:30)
    at /speckle-server/node_modules/ioredis/built/redis/event_handler.js:189:14
    at Socket.<anonymous> (/speckle-server/node_modules/ioredis/built/Redis.js:175:61)
    at Object.onceWrapper (node:events:631:28)
    at Socket.emit (node:events:517:28)
    at Socket.emit (node:domain:489:12)
    at Socket._onTimeout (node:net:598:8) {
  jse_shortmsg: 'Redis encountered an error.',
  jse_cause: Error: connect ETIMEDOUT
      at Socket.<anonymous> (/speckle-server/node_modules/ioredis/built/Redis.js:168:41)
      at Object.onceWrapper (node:events:631:28)
      at Socket.emit (node:events:517:28)
      at Socket.emit (node:domain:489:12)
      at Socket._onTimeout (node:net:598:8)
      at listOnTimeout (node:internal/timers:569:17)
      at process.processTimers (node:internal/timers:512:7) {
    errorno: 'ETIMEDOUT',
    code: 'ETIMEDOUT',
    syscall: 'connect'
  },
  jse_info: { code: 'ENVIRONMENT_RESOURCE_ERROR' },
  cause: [Function: ve_cause]
}

I suspect that the error is coming from the CANONICAL_URL and all the related URLs I set in the docker-compose.yml file: I tried both with the external IP of my Google Cloud VM and with a custom domain name (the latter can be find in the example I’ll provide), but neither of them seem to work.

  • Example: This is my docker-compose.yml file:
version: "2.3"
name: "speckle-server"

services:
  ####
  # Speckle Server dependencies
  #######
  postgres:
    image: "postgres:14.5-alpine"
    restart: always
    environment:
      POSTGRES_DB: speckle
      POSTGRES_USER: speckle
      POSTGRES_PASSWORD: speckle
    volumes:
      - postgres-data:/var/lib/postgresql/data/
    healthcheck:
      # the -U user has to match the POSTGRES_USER value
      test: ["CMD-SHELL", "pg_isready -U speckle"]
      interval: 5s
      timeout: 5s
      retries: 30

  redis:
    image: "redis:7-alpine"
    restart: always
    volumes:
      - redis-data:/data
    ports:
      - "127.0.0.1:6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "--raw", "incr", "ping"]
      interval: 5s
      timeout: 5s
      retries: 30

  minio:
    image: "minio/minio"
    command: server /data --console-address ":9001"
    restart: always
    volumes:
      - minio-data:/data
    healthcheck:
      test: ["CMD", "mc", "ready", "local"]
      interval: 5s
      timeout: 5s
      retries: 5

  ####
  # Speckle Server
  #######
  speckle-ingress:
    image: speckle/speckle-docker-compose-ingress:2
    restart: always
    ports:
      - "0.0.0.0:80:8080"
    environment:
      FILE_SIZE_LIMIT_MB: "100"
      NGINX_ENVSUBST_OUTPUT_DIR: "/etc/nginx"

  speckle-frontend-2:
    image: speckle/speckle-frontend-2:2
    restart: always
    environment:
      NUXT_PUBLIC_SERVER_NAME: "local"
      # TODO: Change NUXT_PUBLIC_API_ORIGIN to the URL of the speckle server, as accessed from the network. This is the same value as should be used for the CANONICAL_URL in the server section below.
      NUXT_PUBLIC_API_ORIGIN: "http://speckledomain"
      NUXT_PUBLIC_BACKEND_API_ORIGIN: "http://speckle-server:3000"
      # TODO: Change NUXT_PUBLIC_BASE_URL to the URL of the speckle frontend, as accessed from the network. This is the same value as should be used for the CANONICAL_URL in the server section below.
      NUXT_PUBLIC_BASE_URL: "http://speckledomain"
      NUXT_PUBLIC_LOG_LEVEL: 'warn'
      NUXT_REDIS_URL: "redis://redis"

  speckle-server:
    image: speckle/speckle-server:2
    restart: always
    healthcheck:
      test:
        [
          "CMD",
          "node",
          "-e",
          "try { require('node:http').request({headers: {'Content-Type': 'application/json'}, port:3000, hostname:'127.0.0.1', path:'/graphql?query={serverInfo{version}}', method: 'GET', timeout: 2000 }, (res) => { body = ''; res.on('data', (chunk) => {body += chunk;}); res.on('end', () => {process.exit(res.statusCode != 200 || body.toLowerCase().includes('error'));}); }).end(); } catch { process.exit(1); }",
        ]
      interval: 10s
      timeout: 3s
      retries: 30

    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
      minio:
        condition: service_healthy
    environment:
      # TODO: Change this to the URL of the speckle server, as accessed from the network
      CANONICAL_URL: "http://speckledomain"
      SPECKLE_AUTOMATE_URL: "http://speckledomain:3030"

      REDIS_URL: "redis://redis"

      S3_ENDPOINT: "http://minio:9000"
      S3_ACCESS_KEY: "minioadmin"
      S3_SECRET_KEY: "minioadmin"
      S3_BUCKET: "speckle-server"
      S3_CREATE_BUCKET: "true"

      FILE_SIZE_LIMIT_MB: 100

      # TODO: Change this to a unique secret for this server
      SESSION_SECRET: "my_secret" #This is just a placeholder, the real secret is a SHA256 hash

      STRATEGY_LOCAL: "true"
      DEBUG: "speckle:*"

      POSTGRES_URL: "postgres"
      POSTGRES_USER: "speckle"
      POSTGRES_PASSWORD: "speckle"
      POSTGRES_DB: "speckle"
      ENABLE_MP: "false"

      USE_FRONTEND_2: "true"
      # TODO: Change this to the URL of the speckle server, as accessed from the network
      FRONTEND_ORIGIN: "http://speckledomain"

  preview-service:
    image: speckle/speckle-preview-service:2
    restart: always
    depends_on:
      speckle-server:
        condition: service_healthy
    mem_limit: "1000m"
    memswap_limit: "1000m"
    environment:
      DEBUG: "preview-service:*"
      PG_CONNECTION_STRING: "postgres://speckle:speckle@postgres/speckle"

  webhook-service:
    image: speckle/speckle-webhook-service:2
    restart: always
    depends_on:
      speckle-server:
        condition: service_healthy
    environment:
      DEBUG: "webhook-service:*"
      PG_CONNECTION_STRING: "postgres://speckle:speckle@postgres/speckle"
      WAIT_HOSTS: postgres:5432

  fileimport-service:
    image: speckle/speckle-fileimport-service:2
    restart: always
    depends_on:
      speckle-server:
        condition: service_healthy
    environment:
      DEBUG: "fileimport-service:*"
      PG_CONNECTION_STRING: "postgres://speckle:speckle@postgres/speckle"
      WAIT_HOSTS: postgres:5432

      S3_ENDPOINT: "http://minio:9000"
      S3_ACCESS_KEY: "minioadmin"
      S3_SECRET_KEY: "minioadmin"
      S3_BUCKET: "speckle-server"

      SPECKLE_SERVER_URL: "http://speckle-server:3000"

networks:
  default:
    name: speckle-server

volumes:
  postgres-data:
  redis-data:
  minio-data:

I would be very grateful if someone could help! :smiling_face_with_three_hearts: :pleading_face:

Thanks and have a nice day :))

Hi @AstraBert

Try to deploy the latest version of speckle-server, by putting the tag to "latest ".

@AstraBert - it appears that the Speckle server cannot connect to Redis. I do not believe this is related to CANONICAL_URL.

Can you view the logs of Redis? Has it logged any errors?

Are you able to connect to Redis using a Redis client Redis CLI | Docs, or from the web app available from the Redis Insight container? Our docker compose files provide an example of how to use Redis Insight: speckle-server/docker-compose-deps.yml at f5007daf895949fd4ecbda5dc8d34e625643e584 · specklesystems/speckle-server · GitHub

Iain

Hi @iainsproat and @ala_eddine!

Thanks for the help, your solutions work well and Speckle server is started!

Now I am facing another problem: I access it within my VM external IP, and when I try to sign up for the service, it returns “ERROR: Failed to fetch”… Is there any particular requirement my server is missing? Here’s the docker-compose.yml:

version: "2.3"
name: "speckle-server"

services:
  ####
  # Speckle Server dependencies
  #######
  postgres:
    image: "postgres:14.5-alpine"
    restart: always
    environment:
      POSTGRES_DB: speckle
      POSTGRES_USER: speckle
      POSTGRES_PASSWORD: speckle
    volumes:
      - postgres-data:/var/lib/postgresql/data/
    healthcheck:
      # the -U user has to match the POSTGRES_USER value
      test: ["CMD-SHELL", "pg_isready -U speckle"]
      interval: 5s
      timeout: 5s
      retries: 30

  redis:
    image: "redis:7-alpine"
    restart: always
    volumes:
      - redis-data:/data
    ports:
      - "127.0.0.1:6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "--raw", "incr", "ping"]
      interval: 5s
      timeout: 5s
      retries: 30
  
  redis_insight:
    image: redislabs/redisinsight:latest
    restart: always
    volumes:
      - redis_insight-data:/db
    ports:
      - '127.0.0.1:8001:8001'
    depends_on:
      - redis

  minio:
    image: "minio/minio"
    command: server /data --console-address ":9001"
    restart: always
    volumes:
      - minio-data:/data
    healthcheck:
      test: ["CMD", "mc", "ready", "local"]
      interval: 5s
      timeout: 5s
      retries: 5

  ####
  # Speckle Server
  #######
  speckle-ingress:
    image: speckle/speckle-docker-compose-ingress:2
    restart: always
    ports:
      - "0.0.0.0:80:8080"
    environment:
      FILE_SIZE_LIMIT_MB: "100"
      NGINX_ENVSUBST_OUTPUT_DIR: "/etc/nginx"

  speckle-frontend-2:
    image: speckle/speckle-frontend-2:2
    restart: always
    environment:
      NUXT_PUBLIC_SERVER_NAME: "local"
      # TODO: Change NUXT_PUBLIC_API_ORIGIN to the URL of the speckle server, as accessed from the network. This is the same value as should be used for the CANONICAL_URL in the server section below.
      NUXT_PUBLIC_API_ORIGIN: "http://speckledomain"
      NUXT_PUBLIC_BACKEND_API_ORIGIN: "http://speckle-server:3000"
      # TODO: Change NUXT_PUBLIC_BASE_URL to the URL of the speckle frontend, as accessed from the network. This is the same value as should be used for the CANONICAL_URL in the server section below.
      NUXT_PUBLIC_BASE_URL: "http://speckledomain"
      NUXT_PUBLIC_LOG_LEVEL: 'warn'
      NUXT_REDIS_URL: "redis://redis"

  speckle-server:
    image: speckle/speckle-server:2
    restart: always
    healthcheck:
      test:
        [
          "CMD",
          "node",
          "-e",
          "try { require('node:http').request({headers: {'Content-Type': 'application/json'}, port:3000, hostname:'127.0.0.1', path:'/graphql?query={serverInfo{version}}', method: 'GET', timeout: 2000 }, (res) => { body = ''; res.on('data', (chunk) => {body += chunk;}); res.on('end', () => {process.exit(res.statusCode != 200 || body.toLowerCase().includes('error'));}); }).end(); } catch { process.exit(1); }",
        ]
      interval: 10s
      timeout: 3s
      retries: 30

    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
      minio:
        condition: service_healthy
    environment:
      # TODO: Change this to the URL of the speckle server, as accessed from the network
      CANONICAL_URL: "http://speckledomain"
      SPECKLE_AUTOMATE_URL: "http://speckledomain:3030"

      REDIS_URL: "redis://redis"

      S3_ENDPOINT: "http://minio:9000"
      S3_ACCESS_KEY: "minioadmin"
      S3_SECRET_KEY: "minioadmin"
      S3_BUCKET: "speckle-server"
      S3_CREATE_BUCKET: "true"

      FILE_SIZE_LIMIT_MB: 100

      # TODO: Change this to a unique secret for this server
      SESSION_SECRET: "f293cafdcf84041280fd3c5ccb77054bc1f7485e5122f421e93ae425a407959d"

      STRATEGY_LOCAL: "true"
      DEBUG: "speckle:*"

      POSTGRES_URL: "postgres"
      POSTGRES_USER: "speckle"
      POSTGRES_PASSWORD: "speckle"
      POSTGRES_DB: "speckle"
      ENABLE_MP: "false"

      USE_FRONTEND_2: "true"
      # TODO: Change this to the URL of the speckle server, as accessed from the network
      FRONTEND_ORIGIN: "http://speckledomain"

  preview-service:
    image: speckle/speckle-preview-service:2
    restart: always
    depends_on:
      speckle-server:
        condition: service_healthy
    mem_limit: "1000m"
    memswap_limit: "1000m"
    environment:
      DEBUG: "preview-service:*"
      PG_CONNECTION_STRING: "postgres://speckle:speckle@postgres/speckle"

  webhook-service:
    image: speckle/speckle-webhook-service:2
    restart: always
    depends_on:
      speckle-server:
        condition: service_healthy
    environment:
      DEBUG: "webhook-service:*"
      PG_CONNECTION_STRING: "postgres://speckle:speckle@postgres/speckle"
      WAIT_HOSTS: postgres:5432

  fileimport-service:
    image: speckle/speckle-fileimport-service:2
    restart: always
    depends_on:
      speckle-server:
        condition: service_healthy
    environment:
      DEBUG: "fileimport-service:*"
      PG_CONNECTION_STRING: "postgres://speckle:speckle@postgres/speckle"
      WAIT_HOSTS: postgres:5432

      S3_ENDPOINT: "http://minio:9000"
      S3_ACCESS_KEY: "minioadmin"
      S3_SECRET_KEY: "minioadmin"
      S3_BUCKET: "speckle-server"

      SPECKLE_SERVER_URL: "http://speckle-server:3000"

networks:
  default:
    name: speckle-server

volumes:
  postgres-data:
  redis-data:
  minio-data:
  redis_insight-data:

Do I need maybe need to specify volumes?

Thanks in advance for the help! :pleading_face: :smiling_face_with_three_hearts:

Astra

This type of error message typically indicates that the server is unreachable or not working. Can you view the logs of the server, and do they show any error message?

Iain

Hi @iainsproat and thanks for the reply!

I’ve looked into the logs and I guess it may have something to do with the initialization steps:

{"@l":"Information","phase":"db-startup","@t":"2024-06-07T13:12:13.398Z","@mt":"Loaded knex conf for production"}
{"@l":"Information","phase":"startup","@t":"2024-06-07T13:12:14.406Z","@mt":"🖼️  Serving for frontend-2..."}
{"@l":"Information","component":"modules","@t":"2024-06-07T13:12:16.197Z","@mt":"💥 Init core module"}
{"@l":"Information","component":"db-notifications","@t":"2024-06-07T13:12:16.239Z","@mt":"🔔 Initializing postgres notification listening..."}
{"@l":"Information","component":"db-notifications","@t":"2024-06-07T13:12:16.239Z","@mt":"Ending connection..."}
{"@l":"Information","component":"db-notifications","@t":"2024-06-07T13:12:16.239Z","@mt":"Attempting to (re-)connect..."}
{"@l":"Information","component":"modules","@t":"2024-06-07T13:12:16.240Z","@mt":"🔑 Init auth module"}
{"@l":"Information","component":"modules","@t":"2024-06-07T13:12:16.306Z","@mt":"💅 Init graphql api explorer module"}
{"@l":"Information","component":"modules","@t":"2024-06-07T13:12:16.306Z","@mt":"📧 Init emails module"}
{"@l":"Warning","component":"modules","@t":"2024-06-07T13:12:16.307Z","@mt":"📧 Email provider is not configured. Server functionality will be limited."}
{"@l":"Information","component":"modules","@t":"2024-06-07T13:12:16.308Z","@mt":"♻️  Init pwd reset module"}
{"@l":"Information","component":"modules","@t":"2024-06-07T13:12:16.308Z","@mt":"💌 Init invites module"}
{"@l":"Information","component":"modules","@t":"2024-06-07T13:12:16.351Z","@mt":"📸 Init object preview module"}
{"@l":"Information","component":"db-notifications","eventName":"preview_generation_update","@t":"2024-06-07T13:12:16.352Z","@mt":"Registering postgres event listener for {eventName}"}
{"@l":"Information","component":"modules","@t":"2024-06-07T13:12:16.352Z","@mt":"📄 Init FileUploads module"}
{"@l":"Information","component":"db-notifications","eventName":"file_import_update","@t":"2024-06-07T13:12:16.352Z","@mt":"Registering postgres event listener for {eventName}"}
{"@l":"Information","component":"db-notifications","eventName":"file_import_started","@t":"2024-06-07T13:12:16.352Z","@mt":"Registering postgres event listener for {eventName}"}
{"@l":"Information","component":"modules","@t":"2024-06-07T13:12:16.353Z","@mt":"🗣  Init comments module"}
{"@l":"Information","component":"modules","@t":"2024-06-07T13:12:16.353Z","@mt":"📦 Init BlobStorage module"}
{"@l":"Information","component":"modules","@t":"2024-06-07T13:12:16.395Z","@mt":"📞 Init notifications module"}
{"@l":"Information","component":"modules","@t":"2024-06-07T13:12:16.395Z","@mt":"📞 Initializing notification queue consumption..."}
{"@l":"Information","component":"modules","@t":"2024-06-07T13:12:16.403Z","@mt":"🤺 Init activity module"}
{"@l":"Information","component":"modules","@t":"2024-06-07T13:12:16.403Z","@mt":"🔐 Init access request module"}
{"@l":"Information","component":"modules","@t":"2024-06-07T13:12:16.403Z","@mt":"🎣 Init webhooks module"}
{"@l":"Information","component":"modules","@t":"2024-06-07T13:12:16.408Z","@mt":"🔄️ Init cross-server-sync module"}
{"@l":"Information","component":"modules","@t":"2024-06-07T13:12:16.408Z","@mt":"🤖 Init automations module"}
{"@l":"Information","component":"cross-server-sync","@t":"2024-06-07T13:12:16.556Z","@mt":"⬇️  Ensuring base onboarding stream asynchronously..."}
{"@l":"Information","component":"cross-server-sync","@t":"2024-06-07T13:12:16.557Z","@mt":"Ensuring onboarding project is present..."}
{"@l":"Information","component":"cross-server-sync","@t":"2024-06-07T13:12:16.557Z","@mt":"No base onboarding stream configured through env vars..."}
{"@l":"Information","phase":"startup","@t":"2024-06-07T13:12:17.350Z","@mt":"🚀 My name is Speckle Server, and I'm running at 0.0.0.0:3000"}

And maybe with the “e-mail provider” portion(?)

Thank you so much for your help!

Cheers,
Astra

That looks healthy, I think we can eliminate this hypothesis.

Can you please provide the full error message details, including a screenshot of the browser’s console and network tab output (these can be found in your browser’s developer tools)?

Iain