Issues with email service (and another component that is not end-user-facing) likely due to Google Cloud infrastructure issues

Incident Report for LiftIgniter

Resolved

Google Cloud confirms having fully resolved the incident https://status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre on their end, and all our systems appear to be stable.
Posted May 17, 2024 - 02:20 UTC

Monitoring

The email service is functioning normally -- so on-open email recommendations should resume working as expected. We are still seeing some symptoms of issues with Google Cloud's backend services, and are waiting on Google Cloud to address those issues and confirm resolution of the underlying issue at https://status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre. We will continue to monitor the situation closely until they have done so.
Posted May 17, 2024 - 00:05 UTC

Identified

Google Cloud has posted details of the incident on their end: https://status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre. We are waiting on them to resolve the issue.
Posted May 16, 2024 - 23:51 UTC

Investigating

We are investigating issues with our email rendering service due to which on-open email recommendations are expected to not render properly. This is likely due to issues with the Google Cloud infrastructure. We're still awaiting confirmation from Google Cloud that there's an issue on their end. In the meantime, we are looking into the best ways to mitigate and address the issue.

Another infrastructure component of ours is also affected, but this component is not end-user-facing and will not have an immediate impact on the end user experience.
Posted May 16, 2024 - 23:25 UTC