Anti-DDoS technologies: why OVH must keep investing heavily to ensure the best protection for customers
A race against time. That’s how we should see the battle OVH has been fighting for years to protect customers from increasingly frequent, increasingly intense DDoS attacks. To resist the attacks of tomorrow, including those bigger than we can even imagine today, we have to invest heavily in our DDoS mitigation technologies. Read on to find out why.
An average day, from the perspective of a VAC system (the OVH anti-DDoS protection system). Legitimate incoming traffic in green. In red, illegitimate traffic (DDoS attacks) filtered by our protection system.
DDoS attacks - an unavoidable phenomenon
2013 saw a surge of DDoS attacks, some peaking at unprecedented levels. It made us realise that OVH had been under-investing in anti-DDoS protection technologies.
For the record, we later discovered that some of these attacks – particularly those targeting customers in the online gambling sector – originated from an unscrupulous competitor. As soon as the attack finished, they would contact the victims to offer their services...of course, making much of their DDoS protection.
Never mind. We’d already decided that the courts couldn’t be the only way to combat the attacks. For sure, legal authorities and their specialist cyber-crime units are attempting to fight back against DDoS attacks. But let’s be realistic. Investigations are long and complex, both technically and due to the international networks that give anyone who can pay, the ability to carry out wide-ranging attacks.
The real battles have to be fought at a technical level. DDoS attacks are inherent to our business as a hosting provider. It’s therefore our responsibility to implement adequate protective measures to mitigate the impact of attacks as much as possible, without raising false hopes that they will deter attacks.
In 2017 we’ve seen an average of 2000 attacks per day, of which 20 can be considered powerful (dozens of Gbps).
This figure has never dipped. You’d be surprised to see the list of countries hosting the largest numbers of bots – the zombie machines behind the biggest DDoS attacks. A recent article pointed out that while God works in mysterious ways, the Vatican’s computers, servers, phones and other connected objects are much easier to read. Why? The city has the world’s highest density of bots per internet user. Perhaps this explains the hacking of Vatican tourists’ phones...proof that there are no miracles in cybersecurity.
Sharing the cost of anti-DDoS protection to benefit everyone
In 2013, we asked our customers to contribute (with an increase of €2 per month per service) in order to deploy our anti-DDoS VAC system as quickly as possible and make up for lost time.
We set ourselves apart from our competitors by not offering DDoS protection as an optional paid service. Instead, we offered everyone the service by including it in all solutions as standard, and spreading out the cost.
Remember, a DDoS attack always results in collateral damage when no protection is in place. Depending on the intensity of the attack, if it is not mitigated, all neighbours of the targeted server rack might temporarily lose their service.
That is why we felt that offering protection to everyone was the best option. Many DDoS attacks have criminal or pointless motives, but they can also be a censorship tool, as demonstrated by American security expert Biran Krebs. Protection against DDoS attacks therefore not only helps to maintain network quality, but also allows us to better protect freedom of expression, which has always been highly important to us.
New-generation anti-DDoS & international deployment
Deploying the VAC in 2013 has given OVH customers a level of protection that was unrivalled at the time and still ahead of the game today. This was demonstrated in September 2016, when OVH fought off a record attack of 1 Tbps.
The VAC system is still performing well, working away in the background to protect OVH customers. Mostly, they don’t realise they’ve been the target of an attack until we email to let them know.
Only, as you know, in the meantime OVH has embarked on an ambitious international development plan, setting up new datacentres in Europe, Asia-Pacific and North America. The aim is to have a presence wherever our customers wish to expand their markets.
We therefore had to think hard about how to scale up the VAC technology. It was initially based on three modules in Roubaix, Strasbourg and Montreal, with a fourth added in Gravelines in 2016.
Aware that the proprietary technology deployed in 2013 was reaching its limits in terms of capacity and scalability, over the last two years we have developed our own anti-DDoS solution. This is based on a range of technological features: FPGA filtering (re-programmable computer chips with a more powerful processing capacity compared to CPU), x86 servers with 6WIND software acceleration and the open-source DPDK library, plus the latest generation of Mellanox 100GbE network cards. This now represents over 100,000 lines of code (our mitigation approach is constantly developing) in our new-generation, made-in-OVH VAC.
To scale up our anti-DDoS protection, we started by replacing the first four VAC 40G units with the new-generation VAC 100G units, each with a capacity of 600 Gbps. Then we started deploying these “vacuum cleaners for illegitimate traffic” in datacentres in new locations.
The aim of boosting the number of VAC units is to deal with attacks as closely as possible to their source. This prevents “transportation” on the backbone and thus avoids mobilising bandwidth unnecessarily, which risks saturating certain connections.
The new-generation VAC has been deployed at the same rate as new datacentres. So far in 2017, OVH has deployed five additional VACs: in Singapore, Sydney, Warsaw, Limburg (Frankfurt) and London. This brings the number of VACs to nine, with a total capacity of more than 4 Tbps. Deployments are planned for the East Coast of the USA (Vint Hill datacentre, early October), Spain and Italy.
In parallel with the international deployment of new-generation anti-DDoS systems, we are pressing on with more R&D. The race against time we mentioned in the introduction has no finishing line. As the internet’s size and capacity expands, the intensity of the attacks will continue to grow. And their sophistication will grow too. On the one hand, we’re studying the mechanisms behind the attacks in order to continuously improve our mitigation tactics. On the other, of course we know the attackers are trying to understand how our protective mechanisms work in order to get around them more easily. We have to stay ahead of the game.
Our users themselves want us to perfect our anti-DDoS protection systems. VAC does protect all OVH customers by default, but it’s only activated when an attack is detected. It then starts to filter illegitimate traffic to keep the targeted server up and running. Some customers, for example in the finance sector, have opted to keep the VAC permanently activated. The slowdown in service during the three seconds or less that it takes, on average, to detect and react to an attack, is not acceptable to them.
We know this level of reactivity will become standard for an increasing number of users. In the field of the Internet of Things (IoT), for example, detection should lead to an immediate reaction.
Indeed, IoT is quickly going to pose another challenge: how to correctly distinguish a DDoS attack from a massive influx of data from connected sensors, which are multiplying exponentially in industrial sectors in particular.
Today, the VAC system is designed to protect our customers from external attacks, i.e. from outside our network. For attacks originating within the OVH network, we can detect and cut them off at the root in less than 30 seconds by isolating the implicated services from the rest of the network. It works efficiently but we can and would like to offer a higher level of protection. We would position an additional anti-DDoS protection lower down in the network, as close as possible to the server, that allows us to be even more reactive in protecting servers from internal attacks.
It’s actually what we already offer on our GAME servers, whose specific requirements in terms of anti-DDoS protection pose a formidable technical challenge. This has driven our innovation in the same way that advances in the aerospace industry often benefited everyone else a few years later. To attain this additional level of protection and offer it to everyone, we’re going to use our vRouter technology. This is a virtual router based on x86 servers, which we are gradually installing in our datacentres. This allows us to deploy high-performance filtering throughout our datacentres.
The proof of concept is currently underway in Roubaix, and the technology is already working in 20,000 servers. We’re also experimenting with techniques that allow us to offer full protection against certain types of attacks by anticipating them and activating protection measures before the first packets reach our network.
In essence, we don’t just want to provide effective anti-DDoS protection, we want to offer the best possible protection. To prove how serious we are, we are considering including the risk of undergoing a DDoS attack in our SLA. In other words, we would contractually guarantee the availability of our services, even in the case of a DDoS attack.
Increase in interconnection capacities: the hidden cost of anti-DDoS protections
Along with the significant cost of the protection system that mitigates the attacks, i.e. intercepts and filters out illegitimate traffic without affecting legitimate traffic, there is a hidden cost. This relates to the need to increase our interconnection capacities (peering) with ISPs all over the world. The aim is to avoid saturating links between the sources of the DDoS attack (which, as the name suggests, are distributed) and our various VACs. These peerings increasingly come with a charge.
OVH needs a lot of excess capacity to be able to absorb the peaks of high-intensity attacks without saturating parts of the network. Just knowing how to deal with an attack is not enough – you must also have the ability to deal with it. Of our 12 Tbps of network capacity, we actually only use 3.5 Tbps on average.
It’s clear that the investment required to combat DDoS attacks is, at least partially, unrelated to OVH’s growth rate. Increasingly powerful attacks force us to accelerate spending at a quicker rate than the OVH server park is growing.
IT infrastructure security is now, more than ever, a critical concern for businesses. Cyber threats, from ransomware to DDoS attacks, have been regularly making headlines in the last few months. There is, without a doubt, a growing awareness of the sheer volume of these attacks. Certain types of threats are difficult to eradicate, which is incidentally why insurers consider cyber attacks to be a high-potential market. However, today, the issue of DDoS attacks is pretty well covered, as long as you choose a cloud provider that prioritises and invests in this issue. This is why OVH has decided to request a contribution from customers, via a €1 to €10 ex. VAT/month increase in the price of VPS, Public Cloud instances and dedicated servers (except Game). This increase will be visible on the OVH websites by the end of October and it will apply to new orders of dedicated servers and VPS. The same will be rolled out for Public Cloud at the start of December. Existing customers whose services are implicated in the price increase, will receive personal email notification of the revised pricing due to take effect on the 1st of December. If these customers have chosen a 3, 6, or 12 month subscription, the increase will impact their services on the next renewal date.