The DDoS Attack That Never Happened
9:09 AM: The First Wave
The attack started at 9:09 AM. 50,000 TCP SYN requests per second started hitting the public website of a major financial institution. A few minutes later the rate increased to 100,000, and finally to 300,000 SYN requests per second.
44 minutes in, a second wave crashed in. A flood of SYN sessions started opening against the same server, first at a rate of 100,000 sessions per second, and quickly increasing until within 30 seconds, there were 4 Gbps sustained on the wire, with more than 2.7 million flows active.
The monitoring systems reported the unusual traffic, the security team was alerted and within a few minutes, assembled on a conference bridge. They contacted their DDoS mitigation service to notify that they were under attack and needed to have them take over the traffic.
In the first 4 minutes the systems held up, but of the typical 350 user requests per second, only 100 were served, with rapidly declining performance. After that all services were completely inaccessible for end users. It took a full 20 minutes until the mitigation service started redirecting the traffic and “scrubbing” it to get rid of attacker sessions, and then services came back online.
11:20 AM: Two More Attacks
But then at 11:20 AM two additional attacks kicked in, simultaneously sending a flood of UDP packets and another of SYN-ACK packets, which the servers tried to respond to and quickly ran out of CPU resources. The security team knew these attacks were going on, and the mitigation service tried to route the attack traffic.
However, at this point the special router tunnels used to route traffic to the DDoS mitigation service failed. Both good user traffic and attack traffic were allowed to hit the servers again with no limitation. Within 2 minutes all public-facing services went offline. It took 17 minutes for the routing tunnels to be recovered, and the attacking traffic went away again.
12:46 AM: Game Over
At 12:46 the attack ended, and the security team went to lunch.
The result: hours of downtime prevented and millions of dollars saved over the next two years. That is right, this attack actually had good consequences.
Because this DDoS attack never really happened. It was a realistic test run by the security team itself to test its people, process and the external vendor they were relying on.
Now they have gathered valuable information that will allow them to prepare for a real attack and prevent devastating damage. The costs of a DDoS attack include direct damages such as loss of sales or advertising revenue, the possibility of network infiltration and data theft by hackers who use DDoS as a “smoke screen”, and indirect damage such as negative publicity.
Here is what they learned from this realistic DDoS simulation:
- Their monitoring systems work well and detected several different types of attack
- Their DDoS mitigation service takes far too long to respond to a real attack
- The mitigation service’s routing mechanism is not robust to multiple DDoS attack profiles
- Even with the current defenses a large-scale attack can result in substantial downtime
The main takeaway from this particular exercise – look into other service options for DDoS mitigation and test again to ensure fast response time and robust routing. This will prevent huge damages in the event of a real attack.
Please note that the attack we described above was a simple attack occurring at the network layer, using large volumes of traffic aimed at a server. There are more complex DDoS attacks that occur at the application layer – hackers can identify “expensive” user operations and simulate real users performing such operations, creating a bottleneck that can bring a server down. These attacks are much harder to detect and defend against.
Why you must simulate DDoS at your organization
DDoS attacks are prevalent and are a big risk to any business with an online presence. Every organization should do at least a basic simulation of a DDoS attack. This can help you discover:
- How many DDoS packets are dropped / taken care of by your DDoS mitigation solution. This is the basic number promised by any DDoS solution, and you should validate it with a simulation.
- How does your DDoS mitigation solution work in a real attack scenario, and how do they deal with different types and magnitudes of attack?
- What level of service will you be able to provide your users while under a DDoS attack of different magnitudes? What is the lowest threshold of latency you can afford?
- How many concurrent users will you be able to serve? What is the lowest threshold below which you do not want to go?
- How do your security team, external vendors and security solutions you have purchased work together to deflect an attack and prevent damage?
- Is your security team geared to prevent other attacks, typically a network penetration attempt, that might be happening in parallel to the DDoS attack? Or is everyone 100% focused on the most visible attack?
All of these are critical questions which you cannot answer without simulating an attack in your organization.
And once you answer them, you’ll be able to:
- Choose the most suitable and cost-effective DDoS mitigation infrastructure and partners, knowing you have battle-tested it and proven its value for money.
- React quickly to an attack and minimize the damage.
- Communicate to stakeholders what to expect in case of an attack – it will not be “business as usual”, but which minimum thresholds of website performance they can expect.
We will also explain the limits of a simple simulation, and show a how DDoS can be tested at larger scale and with more realistic conditions, using professional equipment.
Learn how to simulate an attack in your own IT lab
Does it sound like simulating your own DDoS attack is too complex? It doesn’t have to be.
The story of the test attack above is based on a real test we carried out with a client, using special equipment we have built to simulate large-scale network attacks. Meet Ixia, a leader in security testing and network visibility. We work with 77 of the Fortune 100 and 47 of the world’s top 50 telecom carriers, and have helped thousands of organizations simulate and test for DDoS attacks.
We put together a quick guide that will help you construct a simple DDoS simulation in your own lab, using open source software and without requiring special skills. This will give you the immediate benefit of experiencing a “real” DDoS attack and doing an initial test of your people and processes.