API outage
Incident Report for Elevio

An update was made to the API early on the weekend (US time) to gather more usage statistics.

Things went well initially, but the infrastructure started to buckle under heavy load around 9 AM US EST.

As soon as we were notified of the increased error rates by our monitoring systems, we immediately rolled back the change. Unfortunately, during this rollback process, the system subsequently suffered from thundering herd due to a huge amount of failed requests being retried concurrently, even with autoscaling on.

We decided to double the allowable capacity temporarily to handle the increased load. The system subsequently stabilized a few minutes after that.

Posted Apr 10, 2018 - 00:22 AEST

This incident has been resolved.
Posted Apr 10, 2018 - 00:09 AEST
We're currently experiencing a server outage with our main API service, we are aware of the issue and working to resolve it. We will have this restored ASAP.
Posted Apr 09, 2018 - 23:49 AEST