Troubleshooting Speckle File Uploads via MinIO in Kubernetes

Speckle Help Hosting Speckle
,
1

I have minio running in the same cluster as Speckle. Both are installed from official Helm charts.

Speckle works and is talking to a database and everything provided file uploads are turned off. Minio is up and I can talk to it from my laptop, create buckets, upload files etc. via portforwarding. Minio is running on port 9000 without tls.

However, if I enable file uploads, the speckle-server and speckle-objects pods no longer start. They just get to Init BlobStorage module and hang, with k8s restarting them every ten minutes or so. No errors.

The result is the same whether or not I specify a port in the endpoint, whether or not the port is set in the s3.networkPolicy section of my values and whether I set the protocol to http or https. BUT there does appear to be some sort of connection, at least if I change the endpoint to something that doesn’t exist, I get a relevant error and a container crash.

I’m wondering if this is due to some sort of network policy that’s needed? I’m not too well-versed in that part of Kubernetes, but I have filled out the sections in the values.yaml:

podSelector: { matchLabels: { "statefulset.kubernetes.io/pod-name": "minio-speckle-dev-pool-0-0" } }
namespaceSelector: { matchLabels: { "kubernetes.io/metadata.name": "minio-tenant" } }

Any pointers would be appreciated.

2

(No network policy object gets created.)

3

Hi @nngaiup

Network policies are not created by default, so these are unlikely to be the cause of this issue.

It is likely that this is a networking issue between Minio and the running pod in Kubernetes.

Have you tried deploying a pod with diagnostic tools to your kubernetes pod? You can shell in to the pod then attempt to verify the connection & configuration values. e.g. can DNS be resolved, can a telnet connection be established, can a TLS connection be established, and can S3/Minio http requests be sent using your configuration?

Hope this helps,

Iain

3 Likes
4

Thanks, I got a lot further today, it was mostly my lack of understanding of network names and which strings to put the port numbers in and such.

My IFC files are still not being correctly imported but I’ll have to look at that after my summer holiday :slight_smile: Take it easy and thanks for the reply.

1 Like