Outage Postmortem - June 11th, 2019

Follow

Overview

On June 11th, 2019, clients may have experienced website outages, site slowness and erroneous HTTP 404 errors, starting at 10:20am. The issue was resolved at 11:09am (all times EDT).

The root cause was a bug in our PHP compiler which caused erroneous HTTP 404 errors, (commonly referred to as "Page Not Found" errors) on URLs which had associated pages and should normally work. Our platform monitoring detected the problem and began the process of creating a new server in order to remove the suspected defective server and resolve the issue, however due to the PHP bug, the new server encountered the same problem, causing the process to repeat and eventually cause high server load due to data duplication and the removal of servers, which eventually began causing performance issues and downtime on websites.

The issue was resolved by improving the monitoring system to check a different location, thus halting the process of server regeneration, and purging our cache in order to resolve the PHP bug causing the HTTP 404 errors.

Follow Up

We suspect this issue to have been caused by an old version of PHP which we were using on our platform, and have since updated to a newer version. We will continue to monitor this to ensure that the root cause has been resolved, and to improve our detection of this issue in order to resolve the issue more quickly in future.

We sincerely apologise for the downtime caused by this - we understand how important it is that your website is available and performs as expected 100% of the time, and we will continue to aim for this. You can always monitor our platform status by visiting our Status page, and if you do notice any problems with your website, you can use our live chat service ("Chat with us!" at the bottom right of this page) to alert our Support team immediately and we will investigate the problem.

Have more questions? Submit a request

Comments