Loop piece crashes when it loops through a large array generated by a code piece

I have a code piece where I’m querying a database, paginating the results, and returning a large array of remapped results- e.g. 150-160 items inside the array.

When I start looping on this array, it looks through the first 18-20 items and then crashes with the message “the item to use the loop piece on must be a list” or something like this.

To reproduce this crash, use the code piece to generate a large array (150-300 items) - e.g. an array of rows from a database, with a dew columns as key-value pairs), and try to loop through the with the loop piece. Ideally, with some other modules and a couple of delays, etc.

This needs fixing because your code piece is the only reliable way to query a paginated API and combine output.

Hi @Evgeny

This is quite interesting scenario, what’s happen is the input/output logs become large and when there is delay the current sand-boxed will be cleaned, so the engine try to save the state but since the logs are too large, It’s try to trim it down to 2 MB which remove the large array as first step, as outside the context of delay It’s just for saving the logs in the user interface.

We have to think of tradeoff here but one thing we agree on is that user experience is so bad as it doesn’t tell you what’s happened.

To fix the issue I would recommend using subflows, extract the common part inside loop to subflow.

Or use the queue method 🔀 Queue Piece for Aggregation / Throttling / Rate Limiting

I agree that the UX is bad.

But also - will the queue behave differently?

Say I were to add 560 items into the queue and loop through it with a delay in between iterations.

Will the queue/loop not crash the same way?

I opened ticket to brain storm how we should handle this more gracefully