Possible partial packet loss due to DDoS attacks

28.03.2023
At the moment, packet loss events might occur in parts of our network due to a special form of DDoS attacks. How we are dealing with the situation.

In computing, a denial-of-service attack (DoS attack) is a cyber-attack in which the perpetrator seeks to make a machine or network resource unavailable to its intended users by temporarily or indefinitely disrupting services of a host connected to a network.

https://wikipedia.org/wiki/Denial_of_Service

DDoS - actually not a problem

Providing professional protection against DDoS attacks is an essential part of our product portfolio. This is no coincidence: As some of you may know, the fact that we ourselves were massively affected by DDoS with our freehosting project at the time gave rise at some point to the desire to be able to offer affordable DDoS protection for practically every end customer. Thus, starting in 2015, when DDoS protection was still generally considered unaffordable, we offered the first products with included DDoS protection. Over the following years, we worked our way further into the topic and for some years now we have been working largely autonomously, i.e. with solutions that we essentially administer ourselves, so that we can make any necessary adjustments for our customers at any time without third parties.

Now, as a DDoS protection provider, it is not uncommon to become a target yourself on a regular basis when an attacker fails to get their target offline. Every week, for example, our own websites are the target of such attacks; usually without success. So-called carpet-bombing attacks are unfortunately also completely normal. Here, instead of a single server, an entire network area (typically 256 IPs) is attacked simultaneously with an immense bandwidth, in order to make detection and filtering, which normally takes effect per IP address, more difficult, since the attack bandwidth allotted to a single IP address is not significant. But we also mitigate such attacks within minutes and usually even fully automatically. This was also the case last Sunday (26.03.2023), when suddenly hundreds of DDoS alerts came in within a few minutes, targeting exactly one of our IP networks. The attack was mitigated within a few minutes without significant restrictions and customers remained reachable.

So what?

However, since the late evening hours of yesterday Monday (27.03.2023), a wave of DDoS attacks of a different quality has been reaching us. A large number of different network areas are being attacked simultaneously. Due to more than sufficient connectivity of our carriers, the attacks did not cause a complete outage at any time, but certainly caused packet loss - mainly in the directly affected (attacked) networks. Due to the structure of the attacks, it is more difficult to respond to them this time.

Who has been (or is) affected?

We are known to be a friend of transparent communication, but here we have to balance understandable interest on the customer side with the protection of our network (and thus all customers). Therefore, we have decided not to communicate too many details that could play into the attacker's cards. However, at this stage we can say that around 20% of our IP networks are attacked much more regularly than others and that our Avoro label is much more affected compared to PHP-Friends as more gaming services are operated there, which unfortunately are simply the target of DDoS attacks more often by their very nature. However, since some network areas are also used by both labels, it is not possible to make a general statement. Since the attacks are primarily carried out via UDP, our current (additional) filter measures also primarily relate to UDP, so TCP applications have not been affected so far. IPv6 traffic is also completely unaffected by the attacks.

From our point of view, it is important to understand that, firstly, only a small proportion of all customers are affected and, secondly, no complete network outages have been recorded. Those who are affected will probably also notice this, and those who do not notice anything are probably not affected.

Next steps, time schedule

With the start of yesterday's carpet bombing attacks, we immediately adapted our filtering measures to the respective attack patterns and continue to do so, around the clock, whenever necessary - especially with the help of our main carrier. In this way, we were able to keep the damage to a minimum across the board and, above all, limit it to a few seconds in most cases, even for the affected customers.

In order to be able to carry out a more optimal (granular) filtering of the attacks, we ordered additional network equipment by express delivery last night, which will reach our data center tomorrow (Wednesday, March 29, 2023) and will then be installed and set up. We expect the situation to be resolved by Thursday, 30.03.2023 at the earliest.

Until then, the status quo remains to make all technically feasible adjustments with the existing equipment. We have refined our filters configured overnight during the day to such an extent that the collateral damage in the event of further attacks is even lower, i.e., it affects far fewer customers and is usually in the range of seconds for them. We understand that resulting packet loss is nevertheless very annoying for the affected customers.

It is in the nature of things that we do not know whether further attacks will be driven today as well as in the coming days, but we are (and remain) well prepared for the situation.

Thank you

We would like to thank all affected customers for their understanding and assure you that we will continue to work around the clock on a stable network. We will update the blog post with new information as soon as possible and are happy to answer your questions on all channels - support ticket, e-mail, social media - at any time.

Update 29.03.2023

The delivery of network equipment from the Netherlands, which was actually expected today, is currently "stuck" at customs and will therefore not arrive until tomorrow. However, we were still able to achieve significant improvements in our filtering measures today, so that the impact of the ongoing attacks was already low at the beginning of the day, has become increasingly smaller over the last few hours, and is currently zero to the best of our knowledge. Since the first attack, we have initiated live countermeasures in each case - as we did yesterday - and reacted to each adjustment of the attack patterns within a few minutes. We are therefore cautiously giving the all-clear at this point - the attacks should no longer have any impact on our customers.

In this article:

en_GB