[Ongoing] [Mitigated] Cloud Incident with Run / Flow

Announcements
1

We are currently experiencing issues and investigating the issues and will keep you updated.

Edit: we have mitigated the impact by reroute the traffic to database temporary, the queue is moving slower than usual to avoid overwhelming the database.

1 Like
2

Update: The issue related to downstream vendor (Cloudflare r2) where we store the logs, we are trying to find a way to mitigate.

3

Thanks for the update.
I had to stop disable all my flows because they were sending emails to everybody (the flows were running without any trigger)

4

I also have several flows behaving crazy. Not all of them though

1 Like
5

I stopped the job consuming, we will reroute then turn it back on

6

Will the queued up flows run after the issue is fixed?

7

Yes, we will resume the runs once the issue is fixed, so what’s queued will not get lost

8

We are falling back to store logs in Database, but the queue will be slower to avoid database going down.

9

The issue is now resolved, we are resuming queue consumption

2 Likes
10

good job guys! thanks

11

I am still experiencing problems with mine.
It seems to be triggering webhooks that should not be triggering.
Are you under attack from an outside source?

1 Like
12

Seems that issue is not solved. We have several tasks running and in error. But they started without trigger event

2 Likes