Today between 12:25 PM and 1:15 PM GMT we suffered a major outage. At its peak just over half of all traffic sent to our servers did not receive any kind of response.

This was due to a very large attack on our infrastructure that didn't trigger our anti-DDoS protection due to the way the attack originated from a very large number of source addresses and created traffic similar to our legitimate customers. In addition to this one of our server nodes was offline before the attack began due to an unrelated fault which removed 25% of our North American cluster capacity.

The attack came to an end when we were able to mitigate the attack manually by engaging certain controls at our CDN partner and that immediately brought service back into normal operation.

With attacks against the service becoming more frequent we will be spending even more time looking at our mitigation strategy, today we were slow to react to this attack because our automatic system didn't engage and when trying to deal with it manually we found it difficult to pinpoint which addresses were launching the attack amongst our normal traffic.

Although we saw our traffic was several times greater than normal we couldn't identify quickly which addresses were part of the attack and which was legitimate customer traffic due to the attack purposefully mimicking legitimate requests to our service.

If we have anything more to share about this attack we will update this post. Until then, we are very sorry this occurred and we will strive to do better.