We have deployed a code update that we expect will make the system more robust and the likelihood of this kind of downtime much lower. Even prior to this update, we haven't had this issue for the past 4 weeks. We're marking the incident as resolved.
Posted Oct 18, 2019 - 00:44 UTC
Monitoring
We have just recovered from degraded performance for our inventory API servers used for our inventory API operations on the api.petametrics.com domain.
We will share more details later, but the problems we noticed were:
- Elevated rates of 5XX errors for insertion operations starting around September 19, 2019 1:20 PM Pacific Time (20:20 UTC) - Servers unresponsive to pings intermittently between on September 19, 2019 between 1:30 PM Pacific Time and 1:45 PM Pacific Time (20:30 to 20:45 UTC).
We have reinstated server capacity and are currently reviewing the situation.