Is there a rate limit on POST /objects/:projectId?

vwb · 10 July 2024 11:49

Hi guys,

I am using the @speckle/objectsender typescript library to send some objects to the server. I have lots of data to send and I have been having issues sending the data.

After a while the server stops responding and returns me a 502 error code. I am wondering if there’s a rate limit I should be aware of?

My use case is that I have approx. 8000 objects, each having a displayValue with approx. 16 meshes, with each mesh having approx. 2000 vertices (estimates, obviously).

I have tried different strategies, batching the send, splitting and detaching but now without ideas. Maybe you can help?

jonathon · 10 July 2024 11:54

Which server are you running this against - we could take a look if you are triggering something

vwb · 10 July 2024 11:55

I am running against https://perkinswill.speckle.xyz

dimitrie · 10 July 2024 12:08

Not retired yet, I see?

Anyway, this is a problem we’re looking at on and off as much as we can, whenever we get the info. It’s a ghost in the shell thing.

Can you set us up to reproduce it? Eg., it would be rather important we send the same amount of data in the same format as you, and to a local server ideally. Is a minimal viable repo a viable thing to ask for in this case?

vwb · 10 July 2024 12:16

Of course here’s my entire code base. Note that right now I am batching (its actually working) but I would love to not have to fix closure tables/merge objects later on.

import {
  Base,
  Chunkable,
  Detach,
  send,
  type SendResult,
} from "@speckle/objectsender";
// import {
//   Base,
//   Chunkable,
//   Detach,
//   send,
//   type SendResult,
// } from "./node_modules/@speckle/objectsender/src";

export class Mesh {
  // @Detach()
  // @Chunkable(31250)
  vertices: number[];

  // @Detach()
  // @Chunkable(62500)
  faces: number[];

  constructor(nVertices: number = 15, props?: Record<string, unknown>) {
    // super(props);
    this.vertices = Array(nVertices)
      .fill(0)
      .map(() => Math.random() * 1000);
    this.faces = Array((nVertices / 3) * 4)
      .fill(0)
      .map(() => Math.floor(Math.random() * nVertices));
  }
}

const sendParams = {
  projectId: "13f5f03928",
  token: "ommited",
  serverUrl: "https://perkinswill.speckle.xyz",
};

export class Asset extends Base {
  // @Detach()
  displayValue: Mesh[];

  constructor(meshes: Mesh[], props?: Record<string, unknown>) {
    super(props);
    this.displayValue = meshes;
  }
}

export class Collection<T extends Base> extends Base {
  // @Detach()
  elements: T[];

  constructor(elements: T[], props?: Record<string, unknown>) {
    super(props);
    this.elements = elements;
  }
}

async function main() {
  const t0 = performance.now();
  const numberOfElements = 8000;
  const meshesPerElement = 10;
  const verticesPerMesh = 300;

  const elements = Array(numberOfElements)
    .fill(0)
    .map(
      (v, i) =>
        new Asset(
          Array(meshesPerElement)
            .fill(0)
            .map(() => new Mesh(verticesPerMesh)),
          { name: `Asset ${i}` }
        )
    );

  // split elements and batch send
  const batchSize = 100;
  const elementsBatches = Math.ceil(elements.length / batchSize);

  for (let i = 0; i < elementsBatches; i++) {
    const batch = elements.slice(i * batchSize, (i + 1) * batchSize);
    console.log(
      `Batch ${i + 1}/${elementsBatches} with ${batch.length} elements.`
    );
    const model = new Collection<Asset>(batch);
    const result = await send(model, sendParams);
    console.log(`Batch ${i + 1}/${elementsBatches} sent.`);
  }

  const t1 = performance.now();
  console.log("Main: " + (t1 - t0) / 1000 + "s.");
}

main();

vwb · 10 July 2024 12:45

Here’s a json file with an extract from our database. 50 instances of (vertices + faces). @dimitrie
query_2-2024-07-10_24308.zip (2.6 MB)

iainsproat · 10 July 2024 12:46

Hi @vwb - this is really helpful, thank you.

You mention the above script is actually working. Could you please also send the script that causes the failure?

Could you again run the failing script yourself and let me know the exact time (in UTC time zone) that you received the failure message. Are you given any other information other than the 502 status code? If you can get more details, can you tell me if there a status message or any information in the body of the http response?

This would be very helpful for debugging.

I’ll then add more resources to the server in the hope that this mitigates the problem.

Thank you

Iain

vwb · 10 July 2024 13:07

Thanks Iain,

This version fails with the 502 error code. I received one at 2024-07-10T13:04:18.279Z (ISO)

import {
  Base,
  Chunkable,
  Detach,
  send,
  type SendResult,
} from "@speckle/objectsender";

export class Mesh extends Base {
  @Detach()
  @Chunkable(31250)
  vertices: number[];

  @Detach()
  @Chunkable(62500)
  faces: number[];

  constructor(nVertices: number = 15, props?: Record<string, unknown>) {
    super(props);
    this.vertices = Array(nVertices)
      .fill(0)
      .map(() => Math.random() * 1000);
    this.faces = Array((nVertices / 3) * 4)
      .fill(0)
      .map(() => Math.floor(Math.random() * nVertices));
  }
}

const sendParams = {
  projectId: "13f5f03928",
  token: "ommited",
  serverUrl: "https://perkinswill.speckle.xyz",
};

export class Asset extends Base {
  @Detach()
  displayValue: Mesh[];

  constructor(meshes: Mesh[], props?: Record<string, unknown>) {
    super(props);
    this.displayValue = meshes;
  }
}

export class Collection<T extends Base> extends Base {
  @Detach()
  elements: T[];

  constructor(elements: T[], props?: Record<string, unknown>) {
    super(props);
    this.elements = elements;
  }
}

async function main() {
  const t0 = performance.now();
  const numberOfElements = 8000;
  const meshesPerElement = 10;
  const verticesPerMesh = 900;

  const elements = Array(numberOfElements)
    .fill(0)
    .map(
      (v, i) =>
        new Asset(
          Array(meshesPerElement)
            .fill(0)
            .map(() => new Mesh(verticesPerMesh)),
          { name: `Asset ${i}` }
        )
    );

  const model = new Collection<Asset>(elements);
  try {
    const result = await send(model, sendParams);
  } catch (e) {
    console.log(new Date().toISOString());
    // log utc timestamp
    throw e;
  }

  const t1 = performance.now();
  console.log("Main: " + (t1 - t0) / 1000 + "s.");
}

main();

dimitrie · 10 July 2024 18:38

without sounding like a total noob, where/how do you run this? node project? frontend app? anything to get me setup and remove overhead will help

Plus, i am probably out of touch on the fastest way kids these days setup ts projects they can F5 into…

vwb · 10 July 2024 19:34

I want to add this to our internal backend api. To try things out quickly:

start a node project (pnpm init)
pnpm i dependencies
pnpm i -D typescript and tsx
pnpm tsx index.ts

And, to use decorators I believe you need to enable decorators on a tsconfig.json

vwb · 17 July 2024 17:15

Hi guys, just following up on this. Have you been able to reproduce that? We have been able to send small models, but once we pipe larger models it becomes slow and unreliable.

jonathon · 18 July 2024 07:16

Some of the team have had attention diverted briefly mid investigation.

For a benchmark comparison are you able to repeat the same scenario that fails against latest.speckle and app.speckle

It might offer some insight when it gets picked up again asap.

Sorry for the inconvenience

iainsproat · 18 July 2024 07:30

Hi @vwb - apologies for not providing an update sooner. I’ve taken your script and have been running it against various of our testing servers, but have not yet been able to recreate the exact symptoms.

As @jonathon suggested, please could you try running the same against the following and let us know the outcome (whether it succeeded or failed, and the error message if it failed):

app.speckle.systems
latest.speckle.systems
latest.speckle.dev

That would be incredibly helpful!

Thank you in advance,

Iain

vwb · 18 July 2024 15:43

Hi Iain,

Thanks for the suggestion. I have configured this code sandbox, and submit tasks to all the four servers. PW, Speckle, Speckle Latest, Speckle Latest Dev. Here: Object Sender CodeSandBox

I have made all the projects to be public in all the servers, so hopefully you only have to bring your tokens. To run the tests, please have a look at package.json scripts, OR Codesandbox Tasks if your familiar with that.

1. perkinswill.speckle.xyz

2. app.speckle.systems

3. latest.speckle.systems

4. latest.speckle.dev

I hope that this is useful, and I can’t tell you how much we are looking forward to migrate our data storage to Speckle, but it needs to be able to handle gracefully the amount of data we deal with on our projects.

What should we look into next?

jonathon · 18 July 2024 16:22

100% agree. We’ll definitely compare the profiles and the differences in error responses.

iainsproat · 18 July 2024 17:00

This is really great evidence @vwb , thank you for running the experiments.

From your evidence and the additional server data we have, it strongly suggests that these errors originate with our firewall provider. We’ve made contact with the vendor to try to resolve this.

In the meantime, we believe a more robust retry mechanism is required in the client. Some HTTP and TCP issues would always be expected in any sequence of large volumes of requests. For example, if we send 20 requests every minute for 1 hour duration, we’d expect a 0.1% error rate in the system to cause at least one request to fail.

The client should be expected to retry after receiving a 500-series status code, and this is something we will focus on.

Iain

vwb · 18 July 2024 17:45

Thank you, I have logged the issue here: `objectsender` able to take send payloads · Issue #2520 · specklesystems/speckle-server · GitHub.

Please do reach out if you need help or if this is something you won’t be able to prioritize (that is also fine, it helps me a lot just knowing).

iainsproat · 23 August 2024 12:21

@vwb - it seems I was mistaken and that the objectSender does have a retry mechanism speckle-server/packages/objectsender/src/transports/ServerTransport.ts at e2f2a71b8322309e6d5b6d1bfecad2d03dbf8c6d · specklesystems/speckle-server · GitHub

The transport can be configured to increase retry attempts and backoff between each attempt: speckle-server/packages/objectsender/src/transports/ServerTransport.ts at e2f2a71b8322309e6d5b6d1bfecad2d03dbf8c6d · specklesystems/speckle-server · GitHub

We’ve made some other changes to the server recently to better improve performance when sending. If you have the time, it would be great if you can retry and report back any issues.

Iain

vwb · 24 August 2024 04:07

Hi Iain,

Thanks for taking the time to look into this. We are very excited still waiting for a solution to this issue.

Unfortunately, the mentioned configurations are not exposed by the library. So I cannot test without changing the code.

For reference, the only things exported currently are: speckle-server/packages/objectsender/src/index.ts at e2f2a71b8322309e6d5b6d1bfecad2d03dbf8c6d · specklesystems/speckle-server · GitHub

Let me know if you think we should expose the transport options.

iainsproat · 24 August 2024 11:17

Thanks for pointing this out, I’ve now created a Pull Request which will expose those configuration options: feat(objectsender): expose all options to allow configuration by iainsproat · Pull Request #2751 · specklesystems/speckle-server · GitHub

Iain