Workflows took hours to process

[Evgeny]

See attached. Workflows that usually take seconds now took hours to process.

Runs:

LUvyRuSyxqBijwuUZ8R5P
9wYdbadZve8RZ6Mfxy09Z

[abuaboud]

Hi,

There was an issue today with group of the workers where the disk get full with deployment files, causing flows to be delayed, the issue is fixed and we are implementing and permanent solution to prevent that.

Apologize for this inccident.

Thank you,
Mo.

[abuaboud]

Postmortem: Resolving the Internal Error Issue in Flow Runs

Hi everyone, In the past couple of hours, we encountered an issue where around 4,000 flow runs experienced internal errors. This happened because some servers reached their disk capacity. The reason for the disk full issue was our frequent deployments, and the images/deployment files were not being deleted from the server. We’ve cleaned up the servers, and all impacted flows have already been retried. They should now have a normal outcome. Additionally, we’re implementing an automated way to …