Server Deployment Uploads Not Working

Hi,

we recently deployed our own Speckle server on a Ubuntu server 22.04.3 using your documentation: Deploying a Server - manual setup.
Our deployment is using the integrated PostgreSQL and Redis installations, but AWS S3 instead of the included minio. We have added the TLS certificate from Let’s Encrypt and serve everything over HTTPS.

While most features seem to work (including sending/receiving Stream objects via Blender Connector), we noticed that we were unable to upload files directly in the frontend with the drag’n’drop feature. It always shows this error:

Upload failed with code 404 - no response

Checking the browser console log shows us that the API returns status 404 response for this request: https://[OUR_SPECKLE_SERVER_ADDRESS]/api/file/autodetect/491ac3f1fb/test and the /graphql WebSocket connection is not working:

Failed to load resource: the server responded with a status of 404 ()
WebSocket connection to 'wss://[OUR_SPECKLE_SERVER_ADDRESS]/graphql' failed
WebSocket connection to 'wss://[OUR_SPECKLE_SERVER_ADDRESS]/graphql' failed: WebSocket is closed before the connection is established.

Our docker-compose.yml looks like this:

version: '2.3'
name: 'speckle-server'

services:
  ####
  # Speckle Server dependencies
  #######
  postgres:
    image: 'postgres:14.5-alpine'
    restart: always
    environment:
      POSTGRES_DB: speckle
      POSTGRES_USER: speckle
      POSTGRES_PASSWORD: speckle
    volumes:
      - postgres-data:/var/lib/postgresql/data/
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U speckle"]
      interval: 5s
      timeout: 5s
      retries: 30

  redis:
    image: 'redis:7-alpine'
    restart: always
    volumes:
      - redis-data:/data
    ports:
      - '127.0.0.1:6379:6379'
    healthcheck:
      test: [ "CMD", "redis-cli", "--raw", "incr", "ping" ]
      interval: 5s
      timeout: 5s
      retries: 30

  reverse-proxy:
    image: traefik:v2.10
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.myresolver.acme.tlschallenge=true"
      - "--certificatesresolvers.myresolver.acme.email=[CONFIDENTIAL]"
      - "--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json"

    ports:
      - "443:443"
      - "8080:8080"
    volumes:
      - "./letsencrypt:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  ####
  # Speckle Server
  #######
  speckle-ingress:
    image: speckle/speckle-docker-compose-ingress:2
    restart: always
    ports: []
    environment:
      FILE_SIZE_LIMIT_MB: '100'
      NGINX_ENVSUBST_OUTPUT_DIR: '/etc/nginx'
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.speckle-ingress.rule=Host(`[OUR_SPECKLE_SERVER_ADDRESS]`)"
      - "traefik.http.routers.speckle-ingress.entrypoints=websecure"
      - "traefik.http.routers.speckle-ingress.tls.certresolver=myresolver"
      - "traefik.http.services.speckle-ingress.loadbalancer.server.port=8080"

  speckle-frontend-2:
    image: speckle/speckle-frontend-2:2
    restart: always
    environment:
      NUXT_PUBLIC_SERVER_NAME: 'local'
      NUXT_PUBLIC_API_ORIGIN: 'https://[OUR_SPECKLE_SERVER_ADDRESS]'
      NUXT_PUBLIC_BACKEND_API_ORIGIN: 'http://speckle-server:3000'

  speckle-server:
    image: speckle/speckle-server:2
    restart: always
    healthcheck:
      test: ["CMD", "node", "-e", "try { require('node:http').request({headers: {'Content-Type': 'application/json'}, port:3000, hostname:'127.0.0.1', path:'/graphql?query={serverInfo{version}}', method: 'GET', timeout: 2000 }, (res) => { body = ''; res.on('data', (chunk) => {body += chunk;}); res.on('end', () => {process.exit(res.statusCode != 200 || body.toLowerCase().includes('error'));}); }).end(); } catch { process.exit(1); }"]
      interval: 10s
      timeout: 3s
      retries: 30

    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    environment:
      CANONICAL_URL: 'https://[OUR_SPECKLE_SERVER_ADDRESS]'
      SPECKLE_AUTOMATE_URL: 'http://127.0.0.1:3030'

      SESSION_SECRET: '[CONFIDENTIAL]'

      STRATEGY_LOCAL: "true"
      DEBUG: "speckle:*"

      REDIS_URL: 'redis://redis'

      S3_ENDPOINT: 'https://s3.eu-central-1.amazonaws.com'
      S3_ACCESS_KEY: '[CONFIDENTIAL]'
      S3_SECRET_KEY: '[CONFIDENTIAL]'
      S3_BUCKET: '[CONFIDENTIAL]'
      S3_CREATE_BUCKET: 'true'

      FILE_SIZE_LIMIT_MB: 100

      EMAIL: "true"
      EMAIL_HOST: "email-smtp.eu-central-1.amazonaws.com"
      EMAIL_FROM: "[CONFIDENTIAL]"
      EMAIL_PORT: "2587"
      EMAIL_USERNAME: "[CONFIDENTIAL]"
      EMAIL_PASSWORD: "[CONFIDENTIAL]"

      POSTGRES_URL: 'postgres'
      POSTGRES_USER: 'speckle'
      POSTGRES_PASSWORD: 'speckle'
      POSTGRES_DB: 'speckle'
      ENABLE_MP: 'false'

      USE_FRONTEND_2: 'true'
      FRONTEND_ORIGIN: '[OUR_SPECKLE_SERVER_ADDRESS]'

  preview-service:
    image: speckle/speckle-preview-service:2
    restart: always
    depends_on:
      speckle-server:
        condition: service_healthy
    mem_limit: '1000m'
    memswap_limit: '1000m'
    environment:
      DEBUG: 'preview-service:*'
      PG_CONNECTION_STRING: 'postgres://speckle:speckle@postgres/speckle'

  webhook-service:
    image: speckle/speckle-webhook-service:2
    restart: always
    depends_on:
      speckle-server:
        condition: service_healthy
    environment:
      DEBUG: 'webhook-service:*'
      PG_CONNECTION_STRING: 'postgres://speckle:speckle@postgres/speckle'
      WAIT_HOSTS: postgres:5432

  fileimport-service:
    image: speckle/speckle-fileimport-service:2
    restart: always
    depends_on:
      speckle-server:
        condition: service_healthy
    environment:
      DEBUG: 'fileimport-service:*'
      PG_CONNECTION_STRING: 'postgres://speckle:speckle@postgres/speckle'
      WAIT_HOSTS: postgres:5432

      S3_ENDPOINT: 'https://s3.eu-central-1.amazonaws.com'
      S3_ACCESS_KEY: '[CONFIDENTIAL]'
      S3_SECRET_KEY: '[CONFIDENTIAL]'
      S3_BUCKET: '[CONFIDENTIAL]'

      SPECKLE_SERVER_URL: 'http://speckle-server:3000'

networks:
  default:
    name: speckle-server

volumes:
  postgres-data:
  redis-data:

Now we would like to know:

  • We don’t really need the drag’n’drop feature. Is this problem an indication for another, more severe problem or can we ignore it as long as we don’t want to use it?
  • Can this be caused by the fact that we use AWS S3 instead of minio? We noticed the server isn’t using the S3 bucket. In fact, it’s not even using the AWS IAM account credentials we provided in the docker-compose.yml at all. What is the S3 object storage even used for?

Thanks for your help!

Best regards
Sven

1 Like

Hi @vsx-sieber

Welcome to Speckle’s community, and sorry that you’re first experience with running the server wasn’t as smooth as we’d like. If you haven’t already, please do introduce yourself in our introductions discussion if you wish - we’d love to know more about what you hope to achieve with Speckle.

It appears that perhaps it is the websocket connection that is broken, this is causing the 404 error message. As a result perhaps the file is not actually being uploaded, so the code that stores the file in S3 is never run.

Are there any relevant logs in the speckle-server container that is running in docker? (you can also retrieve them from the Docker desktop dashboard if you are using that instead of the command line).

Are you also receiving websocket errors when you attempt to add comments to a model? We had a similar error reported earlier this month related to this, which was solved by changes to the ingress.

Iain

1 Like

Hi @iainsproat
thanks for your quick response.

So the models are stored in S3? If so how is it possible that I can see and open the model (the one I sent through the Blender Connector) but the AWS management console tells me, that the S3 Bucket I wanted to use is empty?

I indeed found the following suspicous log entry:

AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-central-1'
    at throwDefaultError (/speckle-server/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)
    at /speckle-server/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:18:39
    at de_PutObjectCommandError (/speckle-server/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:5701:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /speckle-server/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /speckle-server/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:14:20
    at async /speckle-server/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46
    at async /speckle-server/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/flexibleChecksumsMiddleware.js:58:20
    at async /speckle-server/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26
    at async Promise.all (index 0) {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 400,
    requestId: '[REMOVED]',
    extendedRequestId: '[REMOVED]',
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Code: 'AuthorizationHeaderMalformed',
  Region: 'eu-central-1',
  RequestId: '[REMOVED]',
  HostId: '[REMOVED]'
}

It seems that the request to AWS S3 via the AWS SDK node package you’re using is not properly setting the region based on the provided AWS S3 endpoint. My endpoint specifically sets the eu-central-1 region (because my Bucket is located there), but Speckle server is using us-east-1 region for making the request.

Adding a text comment works without any issues, but adding a file to a discussion indeed causes a similiar 404 error response from the API (to https://[OUR_SPECKLE_SERVER_ADDRESS]/api/stream/491ac3f1fb/blob), followed by the WebSocket closing error.

Thanks again!

Sven

You will need to add S3_REGION: eu-central-1 to the environment variables of the speckle-server service in the docker compose files.

Let me know if this solves the problem, including the websocket errors.

The file (Speckle currently supports IFC, OBJ, or STL format) is stored in S3, then converted to a Speckle model which is stored in the postgres database.

Iain

1 Like

This resolved all our issues. Thank you very much!

Sven

2 Likes