[Ongoing] [Mitigated] Cloud Incident with Run / Flow

We are currently experiencing issues and investigating the issues and will keep you updated.

Edit: we have mitigated the impact by reroute the traffic to database temporary, the queue is moving slower than usual to avoid overwhelming the database.

1 Like

Update: The issue related to downstream vendor (Cloudflare r2) where we store the logs, we are trying to find a way to mitigate.

Thanks for the update.
I had to stop disable all my flows because they were sending emails to everybody (the flows were running without any trigger)

I also have several flows behaving crazy. Not all of them though

1 Like

I stopped the job consuming, we will reroute then turn it back on

Will the queued up flows run after the issue is fixed?

Yes, we will resume the runs once the issue is fixed, so what’s queued will not get lost

We are falling back to store logs in Database, but the queue will be slower to avoid database going down.

The issue is now resolved, we are resuming queue consumption

2 Likes

good job guys! thanks

I am still experiencing problems with mine.
It seems to be triggering webhooks that should not be triggering.
Are you under attack from an outside source?

1 Like

Seems that issue is not solved. We have several tasks running and in error. But they started without trigger event

2 Likes