Hi Everyone,
We’ve been investigating an inconsistent issue that’s been happening over the past two weeks, affecting services like AITable and Mailgun with SMTP.
Timeline:
- New worker servers were added.
- Connection issues were reported a few days later.
- debugging per service didn’t help to identify the general issue.
Root Cause:
We discovered that all affected services were hosted on Google Cloud. While our servers are located in Germany (a non-sanctioned country), Google incorrectly identified one of the worker IPs as being from a sanctioned country, leading to connection blocks due to issue on their IP to Geography service on Google Side
Postmortem:
To mitigate the issue, we’ve:
- Removed the problematic IPs.
- Shifted external service connections to worker machines only to limit set of IPs.
- Acquired a static set of healthy IPs and assigned them to the machines, to prevent it happenning in the future.
Sorry for the inconvenience, and thank you for your patience.