S3_ACCESS_KEY is missing when using Azure Blob Storage

Help
,
1

Hi, please may I have your help with the following.
I’m deliberately using Azure and Azure Blob Storage which I am aware is not S3 compatible. This might not be workable but wanted to see what is the limitation. I’ve installed Speckle on Kubernetes and the server pod is erroring with the following error. The error message is S3_ACCESS_KEY is missing please see below. I’ve followed the instructions
Deploying a Server - Kubernetes | Speckle Docs
and have assigned s3_secret_key and also tried s3_access_key to my K8s secrets but got the same error. I’ve set the value of these secrets to the Azure Blob Store SAS url in the following format
https://strldddevuksouth.blob.core.windows.net/?sv=2021-12-02&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2023-03-20T21:30:16Z&st=2023-03-20T13:30:16Z&spr=https&sig=*signature*
Not sure which connection string I should be using
If anyone has tried this please can you share your experience Thank you for your help

PS C:\Users\shiangj> kubectl logs pod/speckle-server-5d86cb95bc-gwplx
{“level”:“info”,“time”:“2023-03-20T13:45:57.461Z”,“phase”:“db-startup”,“msg”:“Loaded knex conf for production”}
{“level”:“info”,“time”:“2023-03-20T13:45:58.847Z”,“component”:“modules”,“msg”:“:boom: Init core module”}
{“level”:“info”,“time”:“2023-03-20T13:45:58.980Z”,“component”:“modules”,“msg”:“:key: Init auth module”}
{“level”:“info”,“time”:“2023-03-20T13:45:59.006Z”,“component”:“modules”,“msg”:“:nail_care: Init graphql api explorer module”}
{“level”:“info”,“time”:“2023-03-20T13:45:59.007Z”,“component”:“modules”,“msg”:“:e-mail: Init emails module”}
{“level”:“warn”,“time”:“2023-03-20T13:45:59.007Z”,“component”:“modules”,“msg”:“:e-mail: Email provider is not configured. Server functionality will be limited.”}
{“level”:“info”,“time”:“2023-03-20T13:45:59.008Z”,“component”:“modules”,“msg”:“:recycle: Init pwd reset module”}
{“level”:“info”,“time”:“2023-03-20T13:45:59.008Z”,“component”:“modules”,“msg”:“:love_letter: Init invites module”}
{“level”:“info”,“time”:“2023-03-20T13:45:59.013Z”,“component”:“modules”,“msg”:“:camera_flash: Init object preview module”}
{“level”:“info”,“time”:“2023-03-20T13:45:59.013Z”,“component”:“modules”,“msg”:“:page_facing_up: Init FileUploads module”}
{“level”:“info”,“time”:“2023-03-20T13:45:59.013Z”,“component”:“modules”,“msg”:“:speaking_head: Init comments modu le”}
{“level”:“info”,“time”:“2023-03-20T13:45:59.014Z”,“component”:“modules”,“msg”:“:package: Init BlobStorage module”}
{“level”:“error”,“time”:“2023-03-20T13:45:59.014Z”,“err”:{“type”:“Error”,“message”:“Config value S3_ACCESS_KEY is missing”,“stack”:“Error: Config value S3_ACCESS_KEY is missing\n at getS3Config (/speckle-server/packages/server/dist/modules/blobstorage/objectStorage.js:10:19)\n at getObjectStorage (/speckle-server/packages/server/dist/modules/blobstorage/objectStorage.js:39:26)\n at ensureStorageAccess (/speckle-server/packages/server/dist/modules/blobstorage/objectStorage.js:86:46)\n at ensureConditions (/speckle-server/packages/server/dist/modules/blobstorage/index.js:18:15)\n at exports.init (/speckle-server/packages/server/dist/modules/blobstorage/index.js:42:11)\n at exports.init (/speckle-server/packages/server/dist/modules/index.js:66:22)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async init (/speckle-server/packages/server/dist/app.js:193:5)”},“msg”:“Config value S3_ACCESS_KEY is missing”}

2

Hi @shiangoli

The API for Azure’s blob storage is not compatible with S3, so won’t be workable with Speckle.

However, this particular error is likely due to misconfiguration in your values.yaml or secret manifest. It seems that the secret is not being correctly mounted to the container, as the value cannot be read as an environment variable.

The relevant section of your values.yaml file should be like so:

s3:
  endpoint: "https://s3-compatible-blob-storage.example.org"
  bucket: "example-bucket"
  access_key: "EXAMPLEACCESSKEY"

In your Kubernetes Secret resource, you should have a key s3_secret_key. Section 3.b describes how this should be created.

Iain

3

You could only try to put flexify.io in between @shiangoli … I decided against it and put my S3 on a separate VM.

Microsoft Azure Marketplace

1 Like
4

Hi Iain
The following message got me excited “My name is Speckle Server, and I’m running at 0.0.0.0:3000” but alas not quite :slight_smile:
I take its not worth pursuing azure blob store since the server is unworkable? If so I will have a look at Alex’s option

PS C:\Users\shiangj> kubectl logs pod/speckle-server-74959898cc-4dxx5
{“level”:“info”,“time”:“2023-03-20T15:27:57.265Z”,“phase”:“db-startup”,“msg”:“Loaded knex conf for production”}
{“level”:“info”,“time”:“2023-03-20T15:27:58.651Z”,“component”:“modules”,“msg”:“:boom: Init core module”}
{“level”:“info”,“time”:“2023-03-20T15:27:58.787Z”,“component”:“modules”,“msg”:“:key: Init auth module”}
{“level”:“info”,“time”:“2023-03-20T15:27:58.813Z”,“component”:“modules”,“msg”:“:nail_care: Init graphql api explorer module”}
{“level”:“info”,“time”:“2023-03-20T15:27:58.814Z”,“component”:“modules”,“msg”:“:e-mail: Init emails module”}
{“level”:“warn”,“time”:“2023-03-20T15:27:58.814Z”,“component”:“modules”,“msg”:“:e-mail: Email provider is not configured. Server functionality will be limited.”}
{“level”:“info”,“time”:“2023-03-20T15:27:58.815Z”,“component”:“modules”,“msg”:“:recycle: Init pwd reset module”}
{“level”:“info”,“time”:“2023-03-20T15:27:58.815Z”,“component”:“modules”,“msg”:“:love_letter: Init invites module”}
{“level”:“info”,“time”:“2023-03-20T15:27:58.820Z”,“component”:“modules”,“msg”:“:camera_flash: Init object preview module”}
{“level”:“info”,“time”:“2023-03-20T15:27:58.821Z”,“component”:“modules”,“msg”:“:page_facing_up: Init FileUploads module”}
{“level”:“info”,“time”:“2023-03-20T15:27:58.821Z”,“component”:“modules”,“msg”:“:speaking_head: Init comments modu le”}
{“level”:“info”,“time”:“2023-03-20T15:27:58.821Z”,“component”:“modules”,“msg”:“:package: Init BlobStorage module”}
{“level”:“error”,“time”:“2023-03-20T15:27:58.916Z”,“err”:{“type”:“S3ServiceException”,“message”:“Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:dae52b08-b01e-0054-3040-5b143c000000\nTime:2023-03-20T15:27:58.9068335Z”,“stack”:“AuthenticationFailed: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:dae52b08-b01e-0054-3040-5b143c000000\nTime:2023-03-20T15:27:58.9068335Z\n at throwDefaultError (/speckle-server/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)\n at deserializeAws_restXmlCreateBucketCommandError (/speckle-server/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:3165:51)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /speckle-server/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24\n at async /speckle-server/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:14:20\n at async /speckle-server/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46\n at async /speckle-server/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:5:22\n at async ensureStorageAccess (/speckle-server/packages/server/dist/modules/blobstorage/objectStorage.js:98:17)\n at async ensureConditions (/speckle-server/packages/server/dist/modules/blobstorage/index.js:18:9)\n at async exports.init (/speckle-server/packages/server/dist/modules/blobstorage/index.js:42:5)”,“name”:“AuthenticationFailed”,“$fault”:“client”,“$metadata”:{“httpStatusCode”:403,“attempts”:1,“totalRetryDelay”:0},“Code”:“AuthenticationFailed”},“msg”:“Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:dae52b08-b01e-0054-3040-5b143c000000\nTime:2023-03-20T15:27:58.9068335Z”}
{“level”:“info”,“time”:“2023-03-20T15:27:58.917Z”,“component”:“modules”,“msg”:“:telephone_receiver: Init notifications module”}
{“level”:“info”,“time”:“2023-03-20T15:27:58.917Z”,“component”:“modules”,“msg”:“:telephone_receiver: Initializing notification queue consumption…”}
{“level”:“info”,“time”:“2023-03-20T15:27:59.241Z”,“component”:“modules”,“msg”:“:person_fencing: Init activity module”}
{“level”:“info”,“time”:“2023-03-20T15:27:59.245Z”,“component”:“modules”,“msg”:“:closed_lock_with_key: Init access request module”}
{“level”:“info”,“time”:“2023-03-20T15:27:59.245Z”,“component”:“modules”,“msg”:“:fishing_pole_and_fish: Init webhooks module”}
{“level”:“info”,“time”:“2023-03-20T15:27:59.925Z”,“phase”:“startup”,“msg”:“:rocket: My name is Speckle Server, and I’m running at 0.0.0.0:3000”}
{“level”:“error”,“time”:“2023-03-20T15:28:13.360Z”,“component”:“redis”,“err”:{“type”:“Error”,“message”:“read ECONNRESET”,“stack”:“Error: read ECONNRESET\n at TCP.onStreamRead (node:internal/stream_base_commons:217:20)”,“errno”:-104,“code”:“ECONNRESET”,“syscall”:“read”},“msg”:“Redis encountered an error.”}

/speckle-server/packages/server/dist/modules/shared/redis/redis.js:18
throw new errors_1.EnvironmentResourceError(‘Redis encountered an error.’, err); //FIXME backoff and retry?
^
EnvironmentResourceError: Redis encountered an error.: read ECONNRESET
at EventEmitter. (/speckle-server/packages/server/dist/modules/shared/redis/redis.js:18:23)
at EventEmitter.emit (node:events:513:28)
at EventEmitter.emit (node:domain:489:12)
at EventEmitter.silentEmit (/speckle-server/node_modules/ioredis/built/Redis.js:460:30)
at Socket. (/speckle-server/node_modules/ioredis/built/redis/event_handler.js:189:14)
at Object.onceWrapper (node:events:628:26)
at Socket.emit (node:events:525:35)
at Socket.emit (node:domain:489:12)
at emitErrorNT (node:internal/streams/destroy:151:8)
at emitErrorCloseNT (node:internal/streams/destroy:116:3) {
jse_shortmsg: ‘Redis encountered an error.’,
jse_cause: Error: read ECONNRESET
at TCP.onStreamRead (node:internal/stream_base_commons:217:20) {
errno: -104,
code: ‘ECONNRESET’,
syscall: ‘read’
},
jse_info: { code: ‘ENVIRONMENT_RESOURCE_ERROR’ },
cause: [Function: ve_cause]
}

Node.js v18.14.1

5

Hi Alex, what did you install on your VM?

6

Hi @shiangoli - glad you got the s3 credentials configured correctly!

It appears that Redis is still broken for you. You’ll need to fix Redis first as it’s mandatory for the functioning of Speckle. (Speckle can hobble on without S3, though it will be limited in functionality - so I’d look at Redis first).

The Redis issue might possibly be either a misconfiguration issue, such as misconfigured credentials, or a networking issue, such as a firewall.

Have you been able to connect to your Redis cluster without Speckle, such as with telnet or a similar tool? There is some documentation on how to run Redis commands, such as monitor via telnet on Redis’ site.

Iain

1 Like
7

Hi Iain,
Yes well spotted :slight_smile: , unfortunately this is still a problem :frowning: I have tried using the cli but to no avail, please see below. This might be a firewall issue from my client to the azure service but its difficult to tell. Hence the issue below may have nothing to do with the connection issue I’m getting on the server pod
C:\Program Files\Redis>redis-cli -h speckle-lld.redis.cache.windows.net -p 6380 -a redisaccesskey
speckle-lld.redis.cache.windows.net:6380> ping
Error: An existing connection was forcibly closed by the remote host.

8

I just put MinIO on the VM with this YAML of the Specklers
speckle-server/docker-compose-deps.yml at main · specklesystems/speckle-server (github.com)

I removed the Postgres and Redis from it in my case as I used the managed service for these services.
The MinIO managed service on Azure was cost-wise too expensive for me with 2400 EUR/month

1 Like
9

Thank you Alex for sharing. Which flavour of redis are you using on Azure?

10

They appear to be similar issues with Redis; the ECONNRESET error message from Speckle suggests the network was closed by the Redis host (or a firewall/loadbalancer in between). This is similar to the An existing connection was forcibly closed by the remote host. error message you received from the Redis CLI.

It’s possible you’re using a Redis url with an incorrect protocol (redis:// when it is expecting rediss://), though unfortunately there are many causes to this type of error message.

11

Hi Iain
I was using the protocol “redis” rather than “rediss”. I changed it to rediss and error is different, invalid username/password pair. That username is complete arbitrary I couldn’t find an option to create a user on azure redis cache it would appear the password is all that is required which I found a bit odd.

PS C:\Users\shiangj> kubectl logs pod/speckle-server-74959898cc-h96bp
{“level”:“info”,“time”:“2023-03-20T16:21:25.468Z”,“phase”:“db-startup”,“msg”:“Loaded knex conf for production”}
{“level”:“error”,“time”:“2023-03-20T16:21:26.115Z”,“component”:“redis”,“err”:{“type”:“ReplyError”,“message”:“WRONGPASS invalid username-password pair”,“stack”:“ReplyError: WRONGPASS invalid username-password pair\n at parseError (/speckle-server/node_modules/redis-parser/lib/parser.js:179:12)\n at parseType (/speckle-server/node_modules/redis-parser/lib/parser.js:302:14)”,“command”:{“name”:“auth”,“args”:[“specklelld”,“password”]}},“msg”:“Redis encountered an error.”}

/speckle-server/node_modules/standard-as-callback/built/index.js:6
throw e;
^
EnvironmentResourceError: Redis encountered an error.: WRONGPASS invalid username-password pair
at EventEmitter. (/speckle-server/packages/server/dist/modules/shared/redis/redis.js:18:23)
at EventEmitter.emit (node:events:513:28)
at EventEmitter.emit (node:domain:489:12)
at EventEmitter.silentEmit (/speckle-server/node_modules/ioredis/built/Redis.js:460:30)
at EventEmitter.recoverFromFatalError (/speckle-server/node_modules/ioredis/built/Redis.js:500:14)
at /speckle-server/node_modules/ioredis/built/redis/event_handler.js:34:30
at tryCatcher (/speckle-server/node_modules/standard-as-callback/built/utils.js:12:23)
at /speckle-server/node_modules/standard-as-callback/built/index.js:33:51
at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
jse_shortmsg: ‘Redis encountered an error.’,
jse_cause: ReplyError: WRONGPASS invalid username-password pair
at parseError (/speckle-server/node_modules/redis-parser/lib/parser.js:179:12)
at parseType (/speckle-server/node_modules/redis-parser/lib/parser.js:302:14) {
command: {
name: ‘auth’,
args: [ ‘specklelld’, ‘password’ ]

12

I have used the Azure Cache for Redis for now. Mostly because it is in the favor of our IT department to keep it with Microsofts managed Services.

13

That’s a great step forward, as you’ve been able to make the network connection to Redis.
As you mention, Redis on Azure doesn’t have/require a username.

If your password is yourpassword, and your host is yourazurehost, and the port 6380, I would expect your Redis connection string to look something similar to:

rediss://:yourpassword@yourazurehost:6380

Notice the : prior to yourpassword indicating no username between the rediss:// and the :.

Hope this helps

1 Like
14

Woo hoo!! we have lift off!! :slight_smile: the redis connection issue is sorted. Once I get the storage sorted we should be ok. I thought I would be able to browse to the service using the ingress-nginx-controller ip address but alas I get message this site can’t be reached https://ipaddress

PS C:\Users\shiangj> kubectl logs pod/speckle-server-74959898cc-vmdmq
{“level”:“info”,“time”:“2023-03-20T17:41:06.280Z”,“phase”:“db-startup”,“msg”:“Loaded knex conf for production”}
{“level”:“info”,“time”:“2023-03-20T17:41:07.614Z”,“component”:“modules”,“msg”:“:boom: Init core module”}
{“level”:“info”,“time”:“2023-03-20T17:41:07.761Z”,“component”:“modules”,“msg”:“:key: Init auth module”}
{“level”:“info”,“time”:“2023-03-20T17:41:07.787Z”,“component”:“modules”,“msg”:“:nail_care: Init graphql api explorer module”}
{“level”:“info”,“time”:“2023-03-20T17:41:07.787Z”,“component”:“modules”,“msg”:“:e-mail: Init emails module”}
{“level”:“warn”,“time”:“2023-03-20T17:41:07.787Z”,“component”:“modules”,“msg”:“:e-mail: Email provider is not configured. Server functionality will be limited.”}
{“level”:“info”,“time”:“2023-03-20T17:41:07.788Z”,“component”:“modules”,“msg”:“:recycle: Init pwd reset module”}
{“level”:“info”,“time”:“2023-03-20T17:41:07.788Z”,“component”:“modules”,“msg”:“:love_letter: Init invites module”}
{“level”:“info”,“time”:“2023-03-20T17:41:07.796Z”,“component”:“modules”,“msg”:“:camera_flash: Init object preview module”}
{“level”:“info”,“time”:“2023-03-20T17:41:07.796Z”,“component”:“modules”,“msg”:“:page_facing_up: Init FileUploads module”}
{“level”:“info”,“time”:“2023-03-20T17:41:07.796Z”,“component”:“modules”,“msg”:“:speaking_head: Init comments modu le”}
{“level”:“info”,“time”:“2023-03-20T17:41:07.797Z”,“component”:“modules”,“msg”:“:package: Init BlobStorage module”}
{“level”:“error”,“time”:“2023-03-20T17:41:07.873Z”,“err”:{“type”:“S3ServiceException”,“message”:“Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:35f4224f-201e-005a-2653-5bf837000000\nTime:2023-03-20T17:41:07.8687652Z”,“stack”:“AuthenticationFailed: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:35f4224f-201e-005a-2653-5bf837000000\nTime:2023-03-20T17:41:07.8687652Z\n at throwDefaultError (/speckle-server/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)\n at deserializeAws_restXmlCreateBucketCommandError (/speckle-server/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:3165:51)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /speckle-server/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24\n at async /speckle-server/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:14:20\n at async /speckle-server/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46\n at async /speckle-server/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:5:22\n at async ensureStorageAccess (/speckle-server/packages/server/dist/modules/blobstorage/objectStorage.js:98:17)\n at async ensureConditions (/speckle-server/packages/server/dist/modules/blobstorage/index.js:18:9)\n at async exports.init (/speckle-server/packages/server/dist/modules/blobstorage/index.js:42:5)”,“name”:“AuthenticationFailed”,“$fault”:“client”,“$metadata”:{“httpStatusCode”:403,“attempts”:1,“totalRetryDelay”:0},“Code”:“AuthenticationFailed”},“msg”:“Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:35f4224f-201e-005a-2653-5bf837000000\nTime:2023-03-20T17:41:07.8687652Z”}
{“level”:“info”,“time”:“2023-03-20T17:41:07.874Z”,“component”:“modules”,“msg”:“:telephone_receiver: Init notifications module”}
{“level”:“info”,“time”:“2023-03-20T17:41:07.874Z”,“component”:“modules”,“msg”:“:telephone_receiver: Initializing notification queue consumption…”}
{“level”:“info”,“time”:“2023-03-20T17:41:08.193Z”,“component”:“modules”,“msg”:“:person_fencing: Init activity module”}
{“level”:“info”,“time”:“2023-03-20T17:41:08.213Z”,“component”:“modules”,“msg”:“:closed_lock_with_key: Init access request module”}
{“level”:“info”,“time”:“2023-03-20T17:41:08.213Z”,“component”:“modules”,“msg”:“:fishing_pole_and_fish: Init webhooks module”}
{“level”:“info”,“time”:“2023-03-20T17:41:08.860Z”,“phase”:“startup”,“msg”:“:rocket: My name is Speckle Server, and I’m running at 0.0.0.0:3000”}

2 Likes
15

:tada: - great news regarding Redis!

For security reasons, the Speckle server will not accept connections from your browser (or any other client) which do not match the value you set as the domain in your Helm Chart values.

For example, our Speckle Server is configured with:

domain: speckle.xyz

You can then only visit our Speckle server if you use https://speckle.xyz as the address in your browser. The loadbalancer IP will not work.

So you will need a Domain Name, which matches the value you set as domain, and configure a DNS A record to point at your loadbalancer.

16

The error message in the logs has a different root cause than the domain name configuration of my previous reply. This error is related to Azure’s blob storage not being S3 compliant. Our S3 client is sending S3-compatible requests which are not handled by Azure, resulting in this error.

1 Like
17

Hi Iain, Alex
we are looking at the various options

  1. Fexify.IO
  2. Minio managed service, too expensive, I agree
  3. Minio on a vm cost will be just the running the VM because Minio is fully FOSS

Looks like option 3 atm

1 Like