Speckle Server: Remote end closed connection without response

jagannath · 16 April 2020 11:47

We are uploading a Speckle object to Speckle server by cloning an existing stream and updating it with new Speckle object. This works fine with a small file. But when doing the same with a bigger file (slightly upwards of 5 MB) the client (Python Script) fails with the following message

“http.client.RemoteDisconnected: Remote end closed connection without response”

Although the client receives this error, the object does get updated in Speckle server.

Logs from Speckle below, notice the time difference in saving the object. For smaller file, of course this time difference doesn’t exist and the connection is not closed.

The Setup:
This error is from my local deployment via Docker. No proxy in between.

We faced this issue in our server setup and we always got HTTP 502 error from the Nginx proxy. To troubleshoot the issue I ran a local Docker-based Speckle server and connected the script directly to the application without a proxy in between.

We are running the latest version of Speckle server at this point.

dimitrie · 16 April 2020 19:25

@jagannath - welcome to the forum! I don’t have any experience in python whatsoever, but off the top of my head:

regarding nginx as a proxy: increase the connection timeout. this is how it looks in hestia’s nginx configuration file:

	location / {
		#allow for longer requests (5 minutes)
		proxy_read_timeout 300s;
	}

It might be that the timeout is a default configuration on the python client lib. If stuff actually gets updated, it’s probably this. Unfortunately I’ve got no experience in using python, so I’ll ping @TomSvilans out of his self-isolation slumber. It might help him/others if you add a wee bit of code to see what’s going on

TomSvilans · 17 April 2020 08:38

Hi @jagannath can you post the full console readout of the error? I suspect it might be a server response issue… How big is the payload? Can you provide sample data so I can try it from my end?

@dimitrie my quick Google snooping finds similar problems where the server doesn’t complete a good response such as here with the end_headers() function described here.

Are there any details about what the client should expect to hear back from the Speckle server?

jagannath · 20 April 2020 14:34

Thanks for your replies and hope you are sound and safe. Apologies for my late reply

@dimitrie 1. Initially, I got timeout and max request size issues with Nginx as the proxy. I increased the timeout and the request size limit then I got the “Remote end closed connection without response”. To remove the proxy from the picture, I ran a local docker container for Speckle and connected to it directly without Nginx in between, got the same error.
2. It could be a timeout on the Python library but the error is received just after the object is saved in Speckle, the error is received on the Python as shown in the image in the question. It very well could be the response (or empty response) from Speckle. The expected result would be to receive the new stream ID. Below is the snippet of the code which does that.

#Speckle client
client = SpeckleClient()

# Attempt to access the Speckle Server
client.login()

# Variables used for creating and/or cloning the Speckle Stream.
stream_name = "Test"
stream_description = "Test"

# Create a stream if it's non existant, or create a new version for the stream with the same name.
# Objects are uploaded along the new stream creation. The Store().objects contains the payload
old_stream, new_stream = client.stream_clone_or_create(
    stream_name=stream_name, stream_description=stream_description, objects=Store().objects)

# Speckle objects in meters
new_stream = client.stream_set_base_property(new_stream, "units", "meters")

# Print the new stream information in the Console Logger.
if old_stream != None:
    print(f"The old stream has been cloned and given a new streamId:")
    print(f"{client._server_base_url}/#/streams/{old_stream.streamId}")
    print(f"----------------------------------------------------------------")

print(f"The new stream is now available at:")
print(f"{client._server_base_url}/#/streams/{new_stream.streamId}")
print(f"{len(new_stream.objects)} objects have been uploaded.")

@TomSvilans,
Below is the full console readout. I think the payload is ~7MB.

Traceback (most recent call last):
File “\Development\venv\lib\site-packages\urllib3\connectionpool.py”, line 672, in urlopen
chunked=chunked,
File “\Development\venv\lib\site-packages\urllib3\connectionpool.py”, line 421, in _make_request
six.raise_from(e, None)
File “”, line 3, in raise_from
File “\Development\venv\lib\site-packages\urllib3\connectionpool.py”, line 416, in _make_request
httplib_response = conn.getresponse()
File “C:\Python37\lib\http\client.py”, line 1336, in getresponse
response.begin()
File “C:\Python37\lib\http\client.py”, line 306, in begin
version, status, reason = self._read_status()
File “C:\Python37\lib\http\client.py”, line 275, in _read_status
raise RemoteDisconnected(“Remote end closed connection without”
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “\Development\venv\lib\site-packages\requests\adapters.py”, line 449, in send
timeout=timeout
File “\Development\venv\lib\site-packages\urllib3\connectionpool.py”, line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File “\Development\venv\lib\site-packages\urllib3\util\retry.py”, line 400, in increment
raise six.reraise(type(error), error, _stacktrace)
File “\Development\venv\lib\site-packages\urllib3\packages\six.py”, line 734, in reraise
raise value.with_traceback(tb)
File “\Development\venv\lib\site-packages\urllib3\connectionpool.py”, line 672, in urlopen
chunked=chunked,
File “\Development\venv\lib\site-packages\urllib3\connectionpool.py”, line 421, in _make_request
six.raise_from(e, None)
File “”, line 3, in raise_from
File “\Development\venv\lib\site-packages\urllib3\connectionpool.py”, line 416, in _make_request
httplib_response = conn.getresponse()
File “C:\Python37\lib\http\client.py”, line 1336, in getresponse
response.begin()
File “C:\Python37\lib\http\client.py”, line 306, in begin
version, status, reason = self._read_status()
File “C:\Python37\lib\http\client.py”, line 275, in _read_status
raise RemoteDisconnected(“Remote end closed connection without”
urllib3.exceptions.ProtocolError: (‘Connection aborted.’, RemoteDisconnected(‘Remote end closed connection without response’))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/Development/Script.py”, line 50, in
stream_name=stream_name, stream_description=stream_description, objects=Store().objects)
File “\Development<Code>\speckle\speckle_core\api_client\mixins\streams.py”, line 103, in stream_clone_or_create
root_stream, objects)
File “\Development<Code>\speckle\speckle_core\api_client\mixins\streams.py”, line 78, in stream_create_clone
self.stream_update(new_root.streamId, stream_obj=dct)
File “\Development<Code>\speckle\speckle_core\api_client\mixins\streams.py”, line 140, in stream_update
return self.put(f’/streams/{stream_id}’, ResponseStream, obj=stream_obj)
File “\Development<Code>\speckle\speckle_core\api_client\client.py”, line 116, in put
req = self._session.put(self._server_api_url + relative_url, body)
File “\Development\venv\lib\site-packages\requests\sessions.py”, line 590, in put
return self.request(‘PUT’, url, data=data, **kwargs)
File “\Development\venv\lib\site-packages\requests\sessions.py”, line 530, in request
resp = self.send(prep, **send_kwargs)
File “\Development\venv\lib\site-packages\requests\sessions.py”, line 643, in send
r = adapter.send(request, **kwargs)
File “\Development\venv\lib\site-packages\requests\adapters.py”, line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: (‘Connection aborted.’, RemoteDisconnected(‘Remote end closed connection without response’))

Process finished with exit code 1

The client expects the new stream ID back from the server. For a small object, there are no issues, the newer stream id is returned. For a slightly bigger object (>~5MB I suppose), the server terminates the connection and presents the error “Remote end closed connection without response”. The error is always received when the “Save Object” takes time during the cloning of the stream.
Could this issue be related to this? https://speckle.systems/docs/developers/dotnet-sdk/, “Create & Save a Stream, Large payload”