Upload object with REST API does not work

ludi81 · 30 June 2023 15:17

Hi,

I tried to write the name of an object with an specific id in a stream. I always receive a 500 error. I am using http requests in webstorm.
5fbd3XXXX is the steram (I changed some chars
728f039911984265ed4513c86dcXXXX the object.

### test speckle write
POST https://speckle.xyz/objects/5fbd3XXXX
Content-Type: application/json
Authorization: Bearer 7361233....

[{
  "id": "728f039911984265ed4513c86dcXXXX",
  "Name": "Terrassentuer11"
}]

I always receive the following error:

HTTP/1.1 500 Internal Server Error
Date: Fri, 30 Jun 2023 15:14:35 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
x-request-id: 9a5bb9e42d0f5e0c0d27b9aadd7f10b9
access-control-allow-origin: *
x-ratelimit-remaining: 499
content-security-policy: default-src 'none'
x-content-type-options: nosniff
strict-transport-security: max-age=15724800; includeSubDomains
CF-Cache-Status: DYNAMIC
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=z8BlAk3RNnVz6xdZYGJKblu9PX1Xrn1ia5zGpd7%2BsmcMFLUMEQ8XR6r023uIg%2BXqI0neuxc3SnROH0k8dwLs95H60qeO4iyXsVqNqk2XC8tHbAtCUSEL%2FlB8g5ZH"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 7df75c188ed524c6-ZRH

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="utf-8">
    <title>Error</title>
</head>
<body>
<pre>Internal Server Error</pre>
</body>
</html>

Response file saved.
> 2023-06-30T171435.500.html

Response code: 500 (Internal Server Error); Time: 241ms (241 ms); Content length: 148 bytes (148 B)

My read statement works fine with the following settings.

### test speckle read
GET https://speckle.xyz/objects/5fbd3XXXX/728f039911984265ed4513c86dcXXXX/single
Authorization: Bearer 7361233....

If I try to read objects with " Downloading a list of objects" I am not able to read the same object.

### test speckle read
POST https://speckle.xyz/api/getobjects/5fbd3XXXX
Authorization: Bearer 7361233....
Content-Type: application/json

["728f039911984265ed4513c86dcXXXX"]

It seams, that something mit my post requests generates the error. Do you have any idea?
Kind regards
Ludi

iainsproat · 30 June 2023 16:09

Hi @ludi81

Welcome to the Speckle community, and thanks for giving Speckle a go!

Our /objects/${streamId} endpoint is not a typical POST, and expects the client to follow the form data specification RFC2388.

Firstly, your Content-Type should be multipart/form-data. Depending on your client (for example node’s fetch), you may actually need to omit this as it may be autogenerated by the client with additional boundary control statement already included.
The content should be contained within the File property of Form Data object.

In Typescript an example would look like this:

  type ObjectToPost = { name: string; id?: string } // subset of properties for brevity
  const objectToPost: ObjectToPost = {
    name: 'name of my object'
  }
  const objectId = crypto
    .createHash('md5')
    .update(JSON.stringify(objectToPost))
    .digest('hex')
  objectToPost.id = objectId
  const form = new FormData() // this is node's builtin FormData https://nodejs.org/docs/latest-v18.x/api/all.html#all_globals_class-formdata
  const blob = new Blob([JSON.stringify([objectToPost])], {
    type: 'application/json'
  })
  form.set('batch0', blob)

  const response = await fetch(`${serverUrl}/objects/${streamId}`, {
    method: 'POST',
    body: form,
    headers: {
      Authorization: `Bearer ${yourSpeckleToken}`
      // 'Content-type': 'multipart/form-data', // this is set automatically by fetch https://github.com/github/fetch/issues/505#issuecomment-293064470
    }
  })

To debug, I’d recommend using an echo server - for example PostMan’s echo server - to inspect what your client has sent. But take care not to send your actual Speckle Tokens when using any third party service!

We do have some known issues with this endpoint where the returned status code and error message is not as helpful as it could be. Please refer to this Pull Request on GitHub for more information. As an open source project we welcome contributions to help improve this.

I hope this helps, happy hacking!

Iain

ludi81 · 30 June 2023 18:05

Hi @iainsproat ,

Thanks a lot for your answer. I tried it with following example:

POST https://postman-echo.com/post
Authorization: Bearer 123123
Content-Type: multipart/form-data; boundary=WebAppBoundary

--WebAppBoundary
Content-Disposition: form-data; name="batch0"
Content-Type: application/json

[
{
  "id": "728f039911984265ed4513c86dcXXXX",
  "Name": "TEST"
}
]
--WebAppBoundary--

the response at https://postman-echo.com/post is:

{
  "args": {},
  "data": {},
  "files": {},
  "form": {
    "batch0": "[\n{\n  \"id\": \"728f039911984265ed4513c86dcXXXX\",\n  \"Name\": \"TEST\",\n}\n]"
  },
  "headers": {
    "x-forwarded-proto": "https",
    "x-forwarded-port": "443",
    "host": "postman-echo.com",
    "x-amzn-trace-id": "Root=1-649f17be-78c716774eb4a5b70e6a184b",
    "content-length": "223",
    "authorization": "Bearer 123123",
    "content-type": "multipart/form-data; boundary=WebAppBoundary",
    "user-agent": "Apache-HttpClient/4.5.14 (Java/17.0.7)",
    "cookie": "sails.sid=s%3ASjrLpVw20g-U_8WkCPEzoaBIX6QTvDma.wbXirRJ6ks6iLsMMzGGGCwAbomMQppiXx2zPhZyKfYs",
    "accept-encoding": "br,deflate,gzip,x-gzip"
  },
  "json": null,
  "url": "https://postman-echo.com/post"
}

As far as I can tell, it seams to be correct, or?
The response from your server is (POST https://speckle.xyz/objects/5fbd3XXXX):

Response code: 201 (Created); Time: 353ms (353 ms); Content length: 0 bytes (0 B)

Why is the return code 201? I would like to update an existing one.

In your example you calculated the objectId. I don’t understand this. Why are you not using the given id? Can you explane the following code part? Do I need this, if I would like to generate a new object?

  const objectId = crypto
    .createHash('md5')
    .update(JSON.stringify(objectToPost))
    .digest('hex')

kind regards
Ludi

iainsproat · 30 June 2023 19:02

Hi @ludi81 - the data needs to be appended as files, not a form. You can see from the source code that we only read the files for the form-data.

Possibly the 201 is another example of unexpected (or, at least, poorly documented) behaviour of this endpoint. It may be responding this way as it had zero files, and thus zero objects, to process.

You may ignore the md5 hash to calculate an id from the example, that is irrelevant for the issue at hand.

Iain

ludi81 · 30 June 2023 20:11

Do you have a postman example? I am not able to generate the given structure in the files attribute.

iainsproat · 30 June 2023 20:23

I’m not personally a PostMan user (other than their very helpful echo server!). However, I think this StackOverflow response may help you.

ludi81 · 30 June 2023 20:34

Do you think that the following request should work?

The speckle.json file contains:

  {
    "id": "728f039911984265ed4513c86dcXXXX",
    "Name": "TEST"
  }

In this case your server returns the following result message:

Error inserting object in the database. Check server logs for details

with error code 400 bad request

iainsproat · 30 June 2023 20:39

Almost, the speckle.json file needs to be an array at the top level, as it represents a batch of one or more objects:

[
  {
    "id": "728f039911984265ed4513c86dcXXXX",
    "Name": "TEST"
  }
]

Hope this resolves it!

iainsproat · 30 June 2023 20:42

FYI, there is another Pull Request in progress to improve the error reporting for this, and similar failure cases, so this experience should be improved in future.

ludi81 · 30 June 2023 21:17

I tried it with the [ ] as well.
In this case I receive the 201 message:
2023-06-30_23-15-19

The name of the object is still not updated.

Should speckle generate a new branch or should I see the change in the original branch?

dimitrie · 1 July 2023 19:38

Hi @ludi81, editing an object in speckle is not as straightforward as it seems - if I understand what you’re trying to achieve correctly.

To change the name - or any other property for that matter - of the object means that:

its hash needs to change (speckle id)
all it’s parent’s hashes need to change (you need to recursively propagate this upwards)
the net result is a new tree (decomposed into its leaves when stored in speckle).

This is trivial with our .NET or PY sdks, e.g. you can change any objects and then just save the parent object again to speckle - we take care of diffing in the background, so only what’s changed gets actually sent.

We currently do not have server bound endpoints for editing objects - and it’s not codified in our transport interface either - for the reasons described above.

Can you give us a better idea of what you’re trying to achieve? I’m sure we could help you resolve it in a potentially different way, or, if not, lodge a strong request for patching objects server side (though i would like to avoid that).

PS: if you’re more interested in the architectural principles around speckle, you can read more here: Architecture | Speckle Docs. Given what you’re trying to do, I’d recommend going through the decomposition api section as well.