With denial-of-service (DoS) attacks, attackers try to make a service unavailable. These attacks can target IT infrastructures or simply an individual website.
What measures do we take at 1&1 to protect our users and others?
I had a great, really informative conversation about DDoS protection with our system architect for IT operations, Anders Henke, this week. He and his team design security systems that protect your applications and websites from DDoS attacks and other types of attacks.
What are DDoS attacks?
Anders Henke: In DoS attacks (denial-of-service attacks), an attacker working at a single computer tries to make a service unavailable. The attacker can have any number of motives ranging from “I do not like the opinion in your blog post and will keep people from being able to read your blog” to blackmail – “Pay me x or I will take your website off the Internet”.
A simple example is an attack that is about twenty years old: Depending on the server software, an additional program that uses some RAM is started for each network connection. Now, if an individual computer opens many more than the usual number of network connections, this can occupy several gigabytes of RAM on the server – RAM that is then unavailable for other applications. This can have a negative effect on other applications.
DDoS attacks (distributed denial-of-service attacks) are the next phase in the “arms race”: Instead of a single attacking computer, multiple computers attack the service at the same time. These computers have often been hacked or infected with viruses, trojans or worms.
The computers can connect to form worldwide botnets. The attacker then gives all (or many) of these computers the central command to attack a certain service in a certain way.The attacks would then come from many different computers.
The result is that limiting the number of connections per IP address no longer works.
Large botnets can easily include up to half a million infected computers, making it more difficult to defend yourself.
Attackers do not always have to use hacked or infected computers.
Some Internet protocols are set up so that you make a little request and get a larger response. If someone fakes the IP address of this request, the response goes to the faked IP address.
For example, the attacker makes small requests from a number of DNS or NTP servers using the victim server’s IP address, and many DNS and NTP servers then send considerably larger “responses” to the victim server being attacked. The number of these responses or the bandwidth used can quickly overload the individual server’s Internet connection.
How do we protect our users against DDoS attacks?
Anders Henke: We protect in two directions. On the one hand, we protect ourselves; on the other, as a part of the Internet, we are also obligated to protect the rest of the Internet from ourselves.
The first basic methods we use are spoofing filters and bogon filtering. Our routers have many network connections, and forward IP packets received from a network connection, for example, toward the relevant server network or target provider using a different connection.
The router knows which networks can be reached through each connection – whether our own data center, a certain server network or the Internet is connected to a network connection.
Spoofing filters/network ingress and egress filtering
We have a basic automatic filter in the router for ensuring that it does not accept any IP packets from the network connection to the Internet claiming to be from our own data center. Similarly, a basic filter ensures that IP packets from our own server network or data center can only use sender addresses that come from our own network, not the rest of the Internet.
For example, if a user tries to send packets with a fake sender address with a root server, the fake sender address would at least have to be a 1&1 IP address in order for these packets to get to the Internet (or other 1&1 server networks). In many cases, these incidents are not attacks, but rather faulty configurations and a few fake packets.
If the amount of traffic, meaning the amount of data transferred, is threatening enough, then we can find out relatively quickly within our own network which server the data packets were sent from, and we can take action.
This type of filter really should have been the standard for over 15 years now, but it is a function that still has to be activated.
Some technicians at certain providers are not familiar with this type of function, which is why these functions are not in widespread use all over the Internet. As a result, when DDoS attacks occur, people can often only make a rough estimate of which lines the amount of traffic is coming in on, but not always restrict it to the sender addresses or a provider.
There are certain IP ranges that are not routed on the Internet or ones that have not been allocated to an IP range for assignment in IPv6. As long as these addresses are not expected on the Internet, you can filter out IP packets that claim to come from these addresses. They can only be fake packets known as bogon packets. This filter list has to be maintained manually; changes are made every couple years at best, or occasionally a few times a year.
We have been using this type of filter for a long time to contribute to a clean Internet.
DDoS protection in detail
Anders Henke: Effective protection always works on multiple levels. The first basic level is our Internet connections. We are represented worldwide at many major traffic exchange points to systematically accept traffic from other providers “locally” and transfer it to the 1&1 data centers using our WAN connections.
This local peering ensures that we have more precise spoofing filters, while offering us an extremely large bandwidth. This way, we can immediately identify cases in which, for example, fake traffic is claiming to come from Google IPs but does not come from Google peering.
In other words, we can accept much more traffic before our data centers than we can then transfer to the data centers as a total amount. That sounds excessive and ridiculous at first, but it makes sense: The sooner we can filter out a DDoS attack, the less traffic we have to forward.
Many DDoS attacks are not distributed globally, but rather local.
If, for example, a malware e-mail in South America or China has generated a large botnet that attacks one of our customers, we expect a relatively large amount of traffic from these regions. At the same time, we use basic functions to filter much of the incoming traffic right away.
We know, for example, which DNS and NTP servers our own web hosting servers communicate with. This means that we know that the web hosting servers are not expecting any DNS or NTP responses from other DNS/NTP servers – and can use basic firewall policies to intercept this type of traffic right in the network.
This has allowed us to protect our web hosting servers from the DNS/NTP amplification attacks people have been worrying about lately.
DDoS protection systems developed in-house for protection and flexibility
Many applications benefit from distributed operation on multiple servers. However, typical web hosting applications are not prepared for this type of cluster operation, and are ultimately limited to one server as a result.
For this reason, we have been operating DDoS protection systems we developed in-house for a long time. We activate them upstream of our web hosting servers as needed.
We know which services we run on our web hosting servers because we operate data centers with particularly high-performance servers for these very services. The incoming traffic is distributed to these severs and analyzed.
Every incoming request is systematically inspected for 23 different properties.
These properties include:
- Origin: Traffic for this website usually comes from the following regions.
- Performance deviations: In the last 2 minutes, the web server has been significantly faster or slower than is usual for similar requests.
- Extrapolation as to whether the concrete request is more “normal” or “bad” for the server
These properties result in a score (value/rating). Depending on the score, a request is either let through directly, reduced to an acceptable limit for a normal web browser, restricted or rejected directly.
We prioritize requests based on the specific situation
Good requests are prioritized, requests with a medium score, meaning requests that may not be good requests, are reduced or significantly restricted. Even without a major DDoS, this filters out a basic level of brute-force attackers, password scanners or other malware. This reduces the workload for our web hosting servers and allows the server to respond to important requests more quickly.
We filter based on the specific situation: If a server is doing well, we do not need the filter as much and we are more likely to let requests with an average score pass.
This way, as long as there is no attack, we make sure that we do not wrongly reject good requests.
Our system does not work like a conventional virus scanner with signatures that have to be updated regularly. Our system learns for itself and continuously adapts.
This is important for changing forms of DDoS, and in the evaluation, we can see how attackers start a wave of attacks and we successfully block them. This evaluation and continuous learning allow us to respond effectively to subsequent, adapted attacks.
Protection from attacks at the application level
For an external system, it can be very difficult to precisely identify attacks at the application level. These attacks have a low bandwidth, but can have very severe consequences.
That is why we protect our web hosting servers, for example, with additional measures such as the mod_security rules, which filter out certain forms of bad, dangerous traffic.
We continuously monitor our systems and activate our DDoS protection systems individually as needed. Other measures such as network ingress and egress filtering and bogon filtering are permanently activated.
All of these measures add up to multilevel protection, from the network and our dedicated protection systems to the actual web server.
This type of sophisticated DDoS protection is particularly important in (shared) web hosting: After all, many customers share a single web server. If one of these customers is attacked, our measures protect not only this customer, but all the customers on the web server.