Service Disruption This Morning

Image description

Today between 1:30 AM and 1:45 AM GMT we suffered degraded performance resulting in many requests to the API going unanswered. This was due to a large DDoS attack against our infrastructure which caused all of our servers to intermittently fault. The traffic we received was the same as 6 hours worth of queries but sent over just a few minutes.

Our Anti-DDoS CDN (Content Delivery Network) was able to mitigate the attack starting around 12 minutes into the attack and by the 15th minute normal service was fully restored while the attack was ongoing for another 15 minutes until 2:00 AM GMT or so.

We apologise for any disruption caused by the attack, we usually have our Anti-DDoS detection at our provider set to be quite conservative due to our users often coming under large scale attacks themselves that could trigger our protection if they were to make a large number of queries to us in a short time span. This is why it took a few minutes before the protection activated and restored our service availability to normal.

Next week we'll be going through the activation parameters of our DDoS mitigation and see if we can tune it further for more immediate activation when an attack begins without triggering under normal circumstances.

Thanks for reading and have a wonderful weekend.