The metrics have been stable after disabling edge locations, and the problem should not reappear as long as we keep edge locations disabled. We are marking the incident as resolved.
Posted Jul 31, 2018 - 03:38 UTC
Monitoring
Through our automated alerting, we were notified of CDN connectivity issues for our primary CDN from South and Southeast Asia, and a corresponding drop in pageviews seen by our backends. The problem appears to have started around 2018-07-31 02:20 UTC, though we have not been able to pinpoint the precise time due to obfuscation of metrics by DNS and content caching.
We identified the likely problem as being a misbehaving edge location. To address this, we have temporarily disabled edge locations at around 2018-07-31 02:50 UTC. We reached out to our primary CDN provider for further investigation.
After disabling edge locations, our CDN alerts and pageview volume drop alerts have resolved, so that the immediate problem is mitigated.
Affected countries include India, Malaysia, Taiwan, and Republic of Korea. Other countries in the South and Southeast Asia (and nearby) were likely also affected. Japan (which had experienced issues in a previous incident, see http://status.liftigniter.com/incidents/5xy6fjyc1rm4) was not affected.
The connectivity issues were not experienced by all end users in these regions; in particular, our automated CDN failover health check did not detect a problem with the primary CDN. Due to client-side caching, many users in these regions would have seen no issues.