Recent Connection Issues with various pieces (AITable, Mailgun, etc.) on cloud

Hi Everyone,

We’ve been investigating an inconsistent issue that’s been happening over the past two weeks, affecting services like AITable and Mailgun with SMTP.

Timeline: :hourglass_flowing_sand:

  • New worker servers were added.
  • Connection issues were reported a few days later.
  • debugging per service didn’t help to identify the general issue.

Root Cause:

We discovered that all affected services were hosted on Google Cloud. While our servers are located in :de: Germany (a non-sanctioned country), Google incorrectly identified one of the worker IPs as being from a sanctioned country, leading to connection blocks due to issue on their IP to Geography service on Google Side

Postmortem:

To mitigate the issue, we’ve:

  • Removed the problematic IPs.
  • Shifted external service connections to worker machines only to limit set of IPs.
  • Acquired a static set of healthy IPs and assigned them to the machines, to prevent it happenning in the future.

Sorry for the inconvenience, and thank you for your patience. :pray:

1 Like

Thank you so much for taking care of this :pray::green_heart:

1 Like

thank you so much you guys rock!