On October 20th, our cloud service provider experienced a high error rate and latency in the N. Virginia Region, which temporarily affected some of our internal tools and caused service disruption for O11 and ODC customers.
The issue was first reported at 08:37 UTC+1 for ODC, affecting access to ODC Studio and running applications in ODC Portal.
O11 had the first report at 08:54 UTC+1. Cloud customers weren't able to access their Service Centers, IdP Authentication started to present some failures, CloudWatch started to fail, and several other symptoms were felt, since it affected several networking resources, including 503 error messages.
Customers also experienced delays or interruptions using our services, including Forge, MABs, Licensing Registration, and accessing our Portals and telephony services.
Our Support and Operations teams closely monitored the situation throughout the day, providing continuous updates to affected customers and coordinating internally to restore full functionality as quickly as possible.
By 14:06 UTC+1, it was reported that a full recovery of services had been achieved, and customers began confirming that their services were operating normally. However, later in the afternoon, around 16h32 UTC+1, we started again receiving reports of connectivity issues, affecting O11 customers. During the recovery process, due to latency errors, several customers experienced network connectivity issues. Our teams continued providing updates and ensured all systems remained stable.
AWS confirmed that all its services returned to normal operations by 23:01 UTC+1. This disruption affected 142 AWS services, causing a direct impact on OutSystems services. They determined that the incident was caused by DNS resolution issues affecting DynamoDB. Following this, EC2 instance launches and Network Load Balancer health checks were also impacted, which in turn affected connectivity for Lambda, DynamoDB, and CloudWatch.
We apologize for any inconvenience caused and thank you for your patience.