On May 15th, 2019, clients may have experienced website outages, site slowness and missing images and video, starting at 3am and reoccurring intermittently throughout the day, notably at 3:54am, and around 7:30am until 8:10am. The issue was resolved at 11:34am (all times EDT).
The root cause was two-fold - the downtime between 3am until around 4am was due to a migration to a new server structure. Our infrastructure team attempted to implement this with minimal disruption to our operations and site availability, however the server load was underestimated, and more testing could have been done to ensure we were aware of the impact this may have if any problems were encountered during the transition.
The downtime from around 7:30am until 8:10am was caused by a spike in database connections due to a PHP caching issue, as well as a related memory issue, which affected the performance of sites on our Phoenix platform.
As part of the immediate and long-term fix for these issues, we have added additional servers onto our network to help with future transitions and network structure changes, and are investigating various options for ensuring that these issues do not occur again. We'll also implement a better system of planning to ensure that any changes which could impact site availability or cause significant performance problems are planned for outside of business hours, preferably during off-peak hours at the weekend, where there is less overall traffic to sites on our platform.
We sincerely apologise for the downtime caused by this - we understand how important it is that your website is available and performs as expected 100% of the time, and we will continue to aim for this. You can always monitor our platform status by visiting our Status page, and if you do notice any problems with your website, you can use our live chat service ("Chat with us!" at the bottom right of this page) to alert our Support team immediately and we will investigate the problem.