Server Issue after update to version 2.7.1

Hello Specklers,

after finally setting up everything right including figuring out how to set up the reverse proxy with SSL, I thought it would be a good idea to also update my just set-up server from 2.7.0 to 2.7.1 … this resulted after the Docker-Compose pull and the Docker-Compose Up --detach at first to an error of the Client_Max_Body_size (Your internal NGINX, I believe)… the frontend could not be initiated. In my frustration I cleaned up the docker by removing the containers and setting up speckle again … maybe not the right approach :roll_eyes:.

After that this is the error protocol:

azureuser@speckle2-dev:~/opt/speckle$ sudo docker-compose up -d
Creating network "speckle_default" with the default driver
Creating speckle_speckle-server_1 ...
Creating speckle_speckle-frontend_1 ...
Creating speckle_speckle-server_1     ... done
Creating speckle_redis_1            ...
Creating speckle_postgres_1           ... done
Creating speckle_redis_1              ... done
Creating speckle_preview-service_1    ... done
Creating speckle_minio_1              ... done
Creating speckle_webhook-service_1    ... done
Creating speckle_fileimport-service_1 ... done

ERROR: for speckle-frontend  a bytes-like object is required, not 'str'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 261, in _raise_for_status
    response.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.22/containers/b3fe870a47890c0c3ef78a0ddcbea7190e11246f70704fd345a437271622edfe/start

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/compose/service.py", line 625, in start_container
    container.start()
  File "/usr/lib/python3/dist-packages/compose/container.py", line 241, in start
    return self.client.start(self.id, **options)
  File "/usr/lib/python3/dist-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/docker/api/container.py", line 1095, in start
    self._raise_for_status(res)
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 263, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/lib/python3/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("b'driver failed programming external connectivity on endpoint speckle_speckle-frontend_1 (9a0af439d8fcd46b801644e89745a406c4adcb2e11cfd6ee09fe244693b6fff3): Error starting userland proxy: listen tcp4 0.0.0.0:80: bind: address already in use'")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/docker-compose", line 11, in <module>
    load_entry_point('docker-compose==1.25.0', 'console_scripts', 'docker-compose')()
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 72, in main
    command()
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 128, in perform_command
    handler(command, command_options)
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 1107, in up
    to_attach = up(False)
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 1088, in up
    return self.project.up(
  File "/usr/lib/python3/dist-packages/compose/project.py", line 565, in up
    results, errors = parallel.parallel_execute(
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 112, in parallel_execute
    raise error_to_reraise
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 210, in producer
    result = func(obj)
  File "/usr/lib/python3/dist-packages/compose/project.py", line 548, in do
    return service.execute_convergence_plan(
  File "/usr/lib/python3/dist-packages/compose/service.py", line 545, in execute_convergence_plan
    return self._execute_convergence_create(
  File "/usr/lib/python3/dist-packages/compose/service.py", line 460, in _execute_convergence_create
    containers, errors = parallel_execute(
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 112, in parallel_execute
    raise error_to_reraise
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 210, in producer
    result = func(obj)
  File "/usr/lib/python3/dist-packages/compose/service.py", line 465, in <lambda>
    lambda service_name: create_and_start(self, service_name.number),
  File "/usr/lib/python3/dist-packages/compose/service.py", line 457, in create_and_start
    self.start_container(container)
  File "/usr/lib/python3/dist-packages/compose/service.py", line 627, in start_container
    if "driver failed programming external connectivity" in ex.explanation:
TypeError: a bytes-like object is required, not 'str'
type or paste code here

Do you have an idea, where the issue might be? If it is on my side, I would be very happy for any hint :slight_smile:

Best,
Alex

1 Like

Hey Alex,

sorry for pushing a breaking change on you and thanks for reporting it. I’ll fix this asap. In the meantime, you can add the missing env variable FILE_SIZE_LIMIT_MB for the frontend container and the server container.

See an example here

Let me know how it goes.
Gergő

1 Like

Hey @gjedlicska,

no worries, we are only testing :slight_smile:.

Environment Variables helped, so that it could be built.

Now it seems that the webpage tries to refresh a lot … the blue cube is jumping around

speckle-frontend_1    | 172.18.0.1 - - [04/Aug/2022:09:26:59 +0000] "GET /css/chunk-1da8f299.4f8486b6.css HTTP/1.1" 200 1062 "https://xxxx/authn/login" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.134 Safari/537.36 Edg/103.0.1264.77"
speckle-frontend_1    | 172.18.0.1 - - [04/Aug/2022:09:26:59 +0000] "GET /css/chunk-25a92ccb.b59695da.css HTTP/1.1" 200 279 "https://xxxx" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.134 Safari/537.36 Edg/103.0.1264.77"
speckle-frontend_1    | 172.18.0.1 - - [04/Aug/2022:09:26:59 +0000] "GET /css/chunk-24bfd9b3.00cf68e1.css HTTP/1.1" 200 2425 "https://xxxx/authn/login" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.134 Safari/537.36 Edg/103.0.1264.77"

That could be caused by browser cache or the backend not being accessible. So try hard refreshing first and maybe check on the logs of the backend.

Thanks @gjedlicska :slight_smile:. Cleaning up the cache helped and showed the login page … sadly without input fields. After that I pruned the docker and reinstalled …

did not work as well

The speckle-server has itstelf an issue like this

[ioredis] Unhandled error event: Error: connect ECONNREFUSED 127.0.0.1:6379
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1187:16)

but we are not in a hurry :slight_smile:

1 Like

Check your server config variables, port 6379 is the default port for Redis, and its not reachable from the server.

Did not change anything about the initial configuration so far

Hmm I don’t have all the info based on this. Your config looks valid. What does the redis container logs say?

Redis looks like this:

azureuser@speckle2-dev:~/opt/speckle$ sudo docker container logs speckle_redis_1
1:C 04 Aug 2022 12:30:33.642 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 04 Aug 2022 12:30:33.642 # Redis version=6.0.16, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 04 Aug 2022 12:30:33.642 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
1:M 04 Aug 2022 12:30:33.664 * Running mode=standalone, port=6379.
1:M 04 Aug 2022 12:30:33.664 # Server initialized
1:M 04 Aug 2022 12:30:33.664 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
1:M 04 Aug 2022 12:30:33.665 * Loading RDB produced by version 6.0.16
1:M 04 Aug 2022 12:30:33.665 * RDB age 5105 seconds
1:M 04 Aug 2022 12:30:33.665 * RDB memory usage when created 0.77 Mb
1:M 04 Aug 2022 12:30:33.665 * DB loaded from disk: 0.000 seconds
1:M 04 Aug 2022 12:30:33.665 * Ready to accept connections

The history of why this issues happened:
a) I was not aware of the new environment variables with version 2.7.1
b) This lead me to the believe that I maybe have to simplify the setting … maybe this messed something up
c) punching on it in the hopes it works
d) I messed up my configuration of the reverse proxy in nginx
e) after receiving the environment variables today, it was obvious that the installation could not work …

  • my reverse proxy has two servers one on port 443 where speckle lives and is forwarded on from port 81 and one where http requests are forwarded from port 80 to 443 …
  • after modifying the docker compose file yesterday late evening I put Speckle back on port 80:80

f) Nginx and the Containers blocked each other at docker-compose up --detach
→ Nginx had the port 80 already in use which Speckle needed

I did the following to resolve it:

  • recreate the docker-compose file with my settings and port 81:80 → 81 as internal port on the vm
  • stop nginx
  • prune all containers
  • docker-compose up -d
  • clean up the config-file of nginx
    → Proxy Pass to frontend IP address with port 81
  • start nginx

I hope this will help others … thank you again @gjedlicska for your support :slight_smile:

7 Likes

The real solution lies here.

6 Likes

Thank you Alex for the detailed posthumous.
We’ve added some default config values, so even with just pulling the latest release version, things work just fine.

1 Like