Major AWS outages affecting our main API and event processing

Incident Report for Elevio

Resolved

All systems are now operating normally.

Posted Nov 26, 2020 - 20:37 AEDT

Monitoring

Our event processing systems are gradually recovering. However, full recovery may take up to a few hours.

Posted Nov 26, 2020 - 15:03 AEDT

Investigating

Our main API powering the Assistant has now recovered, however interaction events (article views, searches, module clicks, etc) aren't currently being processed. We continue to investigate the latter.

Posted Nov 26, 2020 - 11:32 AEDT

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Nov 26, 2020 - 11:16 AEDT

Update

Service is slowly being restored. However, full recovery may take up to a few hours.

Posted Nov 26, 2020 - 10:40 AEDT

Update

The main issues are stemming from major AWS outages, which is bringing down a large portion of the internet at the moment.

More information on the current status of AWS can be seen here: https://status.aws.amazon.com/
More details from other major applications being down: https://www.theverge.com/2020/11/25/21719396/amazon-web-services-aws-outage-down-internet

Posted Nov 26, 2020 - 07:44 AEDT

Investigating

We are currently investigating downtime with a number of AWS services, we will report back when we have further information

Posted Nov 26, 2020 - 01:30 AEDT