On Monday, 31 July 2023, 08:25, Cloud Mail experienced a slow response on SYNAQ Webmail platform, with incoming and outgoing mail. The following concerns were raised.
The resultant impact of the event was that a subset of users who were using Webmail experienced a slow response on incoming and outgoing mail.
The root cause of this event was due to degraded memory performance across the Cloud Mail Proxy stack. This resulted in slow connections on email when trying to login via Webmail.
To resolve this incident, our engineers removed one proxy at a time from the cluster, restarted the relevant proxy services on it and added it back to the cluster. This immediately resolved the degraded state and restored the platform to normal operations.
Upon investigation of the root cause of the incident the team at SYNAQ will implement the below remedial actions:
· Updating the existing monitoring for memory swapping to perform remediation actions at a much earlier time.