Capacity issues in US East region due to Google Cloud infrastructure issues (they are unable to provision new instances)

Incident Report for LiftIgniter

Resolved

Google Cloud has confirmed resolution on their end, and all alerts have resolved on our end. We're marking the incident as resolved. If you're a customer who is still seeing infrastructure issues, or if you have any questions about the incident, please reach out to us at liftignitersupport@thearenagroup.net.
Posted Jun 04, 2023 - 15:25 UTC

Monitoring

Capacity has been reprovisioned in the us-east1 region, and we resumed routing traffic to US East. While our metrics seem stable as of now, Google Cloud has not yet confirmed resolution of the issue on their end, so we will continue to monitor the situation for the time being.
Posted Jun 04, 2023 - 15:11 UTC

Identified

Due to a Google Cloud issue creating new VM instances in the us-east1 region (https://status.cloud.google.com/incidents/DyCcy7iGtWqLYEyJBWrC#7Nro3snGH5YXCyUfQuKS), we were having difficulty serving traffic that was being directed to that region. We were able to minimize the damage by directing as much traffic as possible to the us-west1 region that is still operational. We are monitoring for updates from Google Cloud as well as assessing the infrastructure impact.
Posted Jun 04, 2023 - 14:25 UTC