Performance Degradation due to High Server Load for 10 Minutes
Incident Report for FreeAgent CRM
Postmortem

An unusually large bulk operation from a customer spiked CPU processing, causing performance issues for other customers for around 10 minutes. The vast majority of bulk operations are isolated from causing performance issues and run as jobs in a serverless architecture. This particular bulk operation is not insulated, but is also scheduled to be deprecated within the next 30 days. The corrective action was to add additional CPU capacity, effectively doubling the computing power available to the production service, until the operation is in fact deprecated.

Posted Aug 26, 2020 - 14:58 CDT

Resolved
An unusual bulk write operation from one of our customers spiked CPU load on the production server cluster, causing performance degradation from 11:40am Central to 11:50am Central.
Posted Aug 26, 2020 - 13:30 CDT