For some reason we are not able to receive a stream in Speckle Rhino. Speckle sender is working.
Heya @Sunayna! Some questions:
- where are you sending the stream from?
- what is the error looking like - screenshots would help!
Last, if you want to, can you send us in a private message your streamId (& server, if it’s not hestia) to check it out?
It’s not just the receiver, @dimitrie–the sender is pooched too. Odd thing here is that server can actually send/receive via Rhino sometimes; although I do get a web_socket error there too.
Current Server.
Current Client.
Nothing special going on there. Just upgraded server and viewer this week.
Perms are good. PM2 status good. No reason why this should not work.
Yep, sorry guys for disappearing on this one. If the same problem happens when sending through hestia, then i can try to debug; if not it sounds like a deployment problem.
Been trying to determine what this is. Nothing has changed with server. Nadda. Yet now we’re getting this websocket-disconnected error. Tried on another server and everything is good. Really don’t want to kill the server if I don’t have to.
Gotcha. @markcichy, when you updated, what was the process? usually this comes from possibly incorrectly configured upstream proxys - ie, nginx, apache, or aws gateways/loadbalancers, etc. - you need to make sure they correctly handle websocket upgrades.
For nginx, this would look something like this, under /location in your block:
#ws support
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
Already doing that @dimitrie
server {
# ssl
listen 443 ssl;
listen [::]:443 ssl;
server_name speckle.domain.ca;
root /speckle/SpeckleServer;
location / {
# forward http trafic
proxy_pass http://IPADDR:3000;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# ws support
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
ssl_certificate /etc/letsencrypt/live/speckle.domain.ca/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/speckle.domain.ca/privkey.pem;
}
server {
listen 80;
listen [::]:80;
server_name speckle.domain.ca;
return 301 https://$server_name$request_uri;
}Funny thing is, @dimitrie–
This exact setup works flawlessly on three other servers. I think there must be some kind of other issue clipping the server. But am talking to IT and they say they haven’t changed anything.
**EDIT: okay, this may actually be a client issue, not sure–after testing several permutations I’ve found that connecting via internal IP is totally fine. It’s when hitting the domain (through client only-admin viewer FINE) externally via sender/receiver that things break.
@dimitrie can you explain some causes for this??
Why the websocket may terminate? Network related? File issue? Proxy misconfiguration?
If you can provide some insight, then I’ll take a look—it may be DNS related? Though, as I said: the only change has been an update to Speckle. Need a starting point.
Came across this article: though I’m not sure if it is relevant…
Hello Mark, I’ve tested latest speckle clients with latest server version just now and can confirm there’s no problem - updates are fine, and websocket connections are going through properly.
Are there any errors in the logs? (pm2 logs <app>, run pm2 ls beforehand to see which one). Try streaming the logs and creating a receiver/sender and see if any red pops around.
Otherwise, off the top of my head, don’t really know where to start. It depends on your cloud provider as well, and other deployment details.
- how are you pointing your server domain name to the VMs ip?
- are you using https?
- are there any load balancers/gateways forwarding the traffic to your VM?
- noticed the domain name is
speckle.domain.ca- are you using their nameserves/services?
I am getting a certificate error when trying to access the website via https, and a cgi error on non https.
Perhaps you’re fiddling with it right now, but getting rid of these errors and setting up a proper https cert should be the first steps!
PM2 Log Output
[TAILING] Tailing last 15 lines for [0] process (change the value with --lines option)
/home/speckle/.pm2/logs/server-error.log last 15 lines:
0|server | at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)
0|server | errno: 'ECONNREFUSED',
0|server | code: 'ECONNREFUSED',
0|server | syscall: 'connect',
0|server | address: '127.0.0.1',
0|server | port: 6379 }
0|server | { Error: Redis connection to localhost:6379 failed - connect ECONNREFUSED 127.0.0.1:6379
0|server | at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)
0|server | errno: 'ECONNREFUSED',
0|server | code: 'ECONNREFUSED',
0|server | syscall: 'connect',
0|server | address: '127.0.0.1',
0|server | port: 6379 }
0|server | (node:1586) DeprecationWarning: current Server Discovery and Monitoring engine is deprecated, and will be removed in a future version. To use the new Server Discover and Monitoring engine, pass option { useUnifiedTopology: true } to the MongoClient constructor.
0|server | (node:1586) DeprecationWarning: collection.ensureIndex is deprecated. Use createIndexes instead.
/home/speckle/.pm2/logs/server-out.log last 15 lines:
0|server |
0|server | 2020-06-03 11:49:10 debug: Flushing redis database.
0|server | 2020-06-03 11:49:11 debug: Speckle worker 1586 is now online.
0|server |
0|server | 2020-06-03 11:49:11 info: Version: v1.9.9
0|server | 2020-06-03 11:49:11 debug: Scanning for speckle plugins...
0|server | 2020-06-03 11:49:11 warn: specified plugin directory does not exist: ./node_modules/@speckle
0|server | 2020-06-03 11:49:11 warn: No plugin manifest file found in ./plugins.
0|server | 2020-06-03 11:49:11 debug: Found 2 plugin(s): Speckle Admin, Sample Plugin
0|server | 2020-06-03 11:49:12 debug: Initialising redis in radio tower.
0|server | 2020-06-03 11:49:14 debug: Speckle worker process 1586 now running on port 3000.
0|server | 2020-06-03 11:49:14 debug: 1586 connected to redis.
0|server | 2020-06-03 11:49:14 debug: Connected to mongo.
0|server | 2020-06-03 11:49:14 debug: connected to mongoose at mongodb://localhost:27017/speckle
0|server | 2020-06-03 11:49:14 debug: getaddrinfo EAI_AGAIN CHANGE_ME
Answers to your previous bullets:
- Am pointing to the internal IP (on a discreet subnet, but this has not changed)
- Yes, server block automatically redirects to
https - There are definitely a series gates that the traffic needs to go through – although the, as far as I know the reverse proxy is in front of the network in the DMZ (should be good)
- we spoke privately about the domain
Server back up–no errors. It appears as though this has to be infrastructure related.
Output
0|server | 2020-06-03 12:26:54 info: Failed to initialise route based telemetry.
0|server | 2020-06-03 12:26:54 info: HTTP GET /api/accounts/5cc70c3d5094ca6bf9e3dcf5 304 4ms
0|server | 2020-06-03 12:26:54 info: Failed to initialise route based telemetry.
0|server | 2020-06-03 12:26:54 info: HTTP GET /api/accounts/5cc70c3d5094ca6bf9e3dcf5 304 4ms
0|server | 2020-06-03 12:26:54 info: Failed to initialise route based telemetry.
0|server | 2020-06-03 12:26:54 info: HTTP GET /api/accounts/5cc70c3d5094ca6bf9e3dcf5 304 3ms
0|server | 2020-06-03 12:26:54 info: Failed to initialise route based telemetry.
0|server | 2020-06-03 12:26:54 info: HTTP GET /api/accounts/5cc70c3d5094ca6bf9e3dcf5 304 4ms
0|server | 2020-06-03 12:32:20 info: Failed to initialise route based telemetry.
0|server | 2020-06-03 12:32:20 info: HTTP GET /api/accounts 200 5ms
0|server | 2020-06-03 12:32:20 debug: Bulk obj save: 0 received
0|server | 2020-06-03 12:32:20 info: HTTP POST /api/streams 200 36ms
0|server | 2020-06-03 12:32:20 info: HTTP POST /api/clients 200 37ms
0|server | 2020-06-03 12:32:29 debug: Patching stream wQWEDAPAs
0|server | 2020-06-03 12:32:29 info: HTTP PUT /api/streams/wQWEDAPAs 200 11ms