Looping rows through an LLM step: why 429s hit and the throttle pattern I use

Discussions
1

I keep hitting the same failure mode on flows that loop over a few hundred rows, send each one to an LLM step to classify and summarize it, then write the result back. Testing with a handful of rows looks fine. On the first real volume run, a chunk of them come back 429’d (rate limited). Worse, some of the retries get written twice, so the sheet ends up with duplicates and gaps at the same time.

What is usually going on:

  1. Requests per minute isn’t the limit, tokens per minute is. Each row’s prompt plus output burns a non-trivial number of tokens. A tight loop over a few hundred rows blows through the TPM budget fast, even when the request count still looks fine.
  2. Retries are all on the same schedule. Every failed call retries after the same fixed delay, so they all hit the API again at the same moment and get 429’d again.
  3. The write step isn’t safe to repeat. A retry after a timeout (the model had answered, the response just came back slow) means the row gets processed twice.

What I change:

  • Batch before the loop. Chunk rows into small groups and send one prompt per chunk that returns a JSON array keyed by row id. That cuts call count hard and drops tokens meaningfully, because the instructions are sent once per chunk instead of once per row.
  • Pace it to a token budget, not a fixed delay. Estimate tokens per chunk (characters / 4 is close enough) and add a delay between chunks so you stay at about 70% of the TPM limit. The headroom matters because other flows share the same key.
  • Backoff with jitter on 429/5xx only. Wait base * 2^attempt + random(0, base), max 4 attempts, and respect Retry-After if the response has it. Don’t retry 400s — those are bad prompts or bad input and retrying just burns tokens.
  • Idempotent writes. Each row carries a stable id, and the write step upserts on that id instead of appending. A double run now overwrites instead of duplicating.
  • Dead-letter, don’t drop. After the last retry, the row goes to a “failed” table with the error and attempt count, and a small scheduled flow re-queues those later.
  • Validate the JSON array. If the chunk response is missing ids or has extras, re-run only the missing ids instead of the whole chunk.

After wiring that in, the next volume run stops losing rows and stops duplicating them. The rows that still fail are usually genuinely malformed input, and they land in the dead-letter table instead of vanishing halfway through the loop.

Question for people running bigger volumes: do you throttle inside the flow like this, or do you push items onto a queue and have a separate flow drain it at a fixed rate? I’m curious where the crossover point is where the queue approach becomes worth the extra moving parts.