Brief degradation on all services

Incident Report for Atlassian Bitbucket

Resolved

A piece of hardware in Bitbucket's internal network infrastructure failed at roughly 17:38 UTC on Saturday 18th February 2017. Automated failover systems kicked in as intended, and performed as intended; the load balancers also began queueing connection requests (as intended) while the failover process completed. Unfortunately, all this intentional behavior led to an unintentional time of service degradation between (roughly) 17:38 and 17:45; load balancer backlogs caused many requests to wait far longer than anticipated, and some requests may have timed out as a result.

We apologize for any inconvenience, and we're re-examining our failover strategy so that the backlog remains small when this happens next.

Posted Feb 18, 2017 - 12:00 UTC