Suddenly our nagios IRC bot politely informs us that the HTTP service is down for one of the servers. Fortunately SSH still works like a charm. A quick inspection reveals no peculiarities, except for an amazing number of Apache clients doing nothing. A quick glance in the logs learns us that Apache has reached its MaxClients setting. An Apache restart doesn't help, it reaches its MaxClients setting again within the minute.

# tail /var/log/apache2/error.log

[Fri Dec 05 14:31:30 2008] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Fri Dec 05 14:50:54 2008] [notice] caught SIGTERM, shutting down
[Fri Dec 05 14:50:56 2008] [notice] Apache/2.2.x (Debian) configured -- resuming normal operations
[Fri Dec 05 14:51:35 2008] [error] server reached MaxClients setting, consider raising the MaxClients setting

Could be a (D)DoS attack. However, the server doesn't show typical signs of a DoS attack like high load or network traffic. Next stop is netstat which shows us a huge number of tcp connections in SYN_RECV state:

# netstat -n -p TCP

tcp        0      0 10.1.1.1:80       91.64.4.146:64979       SYN_RECV   -
tcp        0      0 10.1.1.1:80       84.24.103.112:4005      SYN_RECV   -
tcp        0      0 10.1.1.1:80       79.223.69.239:61510     SYN_RECV   -
tcp        0      0 10.1.1.1:80       67.86.135.44:43312      SYN_RECV   -
tcp        0      0 10.1.1.1:80       86.88.67.226:50600      SYN_RECV   -
tcp        0      0 10.1.1.1:80       173.20.137.110:3813     SYN_RECV   -
tcp        0      0 10.1.1.1:80       84.58.10.121:4878       SYN_RECV   -
tcp        0      0 10.1.1.1:80       91.37.40.151:2408       SYN_RECV   -
tcp        0      0 10.1.1.1:80       173.20.137.110:3441     SYN_RECV   -
...

Lots of half-open connections are a good indication of a SYN Flood attack. So we need to do two things:

1. Harden the TCP/IP stack
2. Block attackers

Harden the TCP/IP stack

If you're not familiar to SYN Floods or other TCP-based resource starvation attacks, read this SecurityFocus article. It also covers hardening your TCP/IP stack against SYN Flooding and SYN Spoofing. Most important is enabling SYN Cookies.

Block attackers

Now we've hardened the TCP/IP stack, spoofing has been eliminated. The attacking source address is really the attacker, so it makes sense to block it. However, these days it's not very hard to create, buy or rent a large botnet consisting of thousands of hacked computers ready to be engaged in massive DDoS attacks. So manually block IP's won't work.

We'll have to write a script to automatically block attacking IP's. The script consists of three parts:

1. Find out which IP's to block
2. Block IP
3. Unblock IP after 24 hours

The first part is the hardest part. In our case, we could just take all IP's having too many connections in SYN_RECV state. In your case you might need to use another condition, e.g. count requests to a specific URI.

$netstat = shell_exec('/bin/netstat -apn | grep SYN_RECV');
$lines = preg_split('/[\r\n]+/', $netstat);

$ips = array();
foreach ($lines as $line) {
    $fields = preg_split('/\s+/', $line);
    if(preg_match('/[\d\.]+/is', $fields[4], $match)) {
       $key = $match[0];
       $ips[$key] = empty($curr[$key]) ? 1 : 
      ($ips[$key] + 1);
    }
}

Now we have all IP's and number of concurrent connections in SYN_RECV state. Next step is block IP's with more than 50 connections. This number is quite random, any sane treshold will do.

foreach ($ips as $key => $val) {
    if ($val > 50) {
       exec("sbin/iptables -A INPUT -s " . 
      escapeshellarg($key) . " -j DROP");
    }
}

For the last step you need to create your own administration of blocked IP's and timestamps. Check out the full script [Download] for this and some other features like a whitelist and simple logging.

Reinventing the wheel?

You might wonder why we try to reinvent the wheel. Though avoiding to reinvent the proverbial wheel is a good practice and key principle in software development, it's useful to rethink the wheel concept from time to time. Or to quote Jeff Atwood: Don't Reinvent The Wheel, Unless You Plan on Learning More About Wheels.

Most high profile websites will probably have a dedicated intrusion detection or prevention system (IDS/IPS) in place. IT security companies can implement a commercial solution, e.g. Cisco, IBM or Juniper systems. You could also try to build one yourself. Snort is a free and open source IDS/IPS. It's the de facto standard and used by many commercial solutions as well.

However, for most websites or web applications an IDS/IPS is overkill. Start with good server configuration, hardened OS and server software and a good router and firewall. And add an IDS or custom IP blocking script if necessary.

Finally

In this case we controlled the attack within 5 minutes after we launched the block-ips script. The websites came back online and the attack stopped after three days.

Further reading:

• Wikipedia: Denial-of-service attack
• CERT Advisory: TCP SYN Flooding and IP Spoofing Attacks
• SecurityFocus article: Hardening the TCP/IP stack to SYN attacks
• D.J. Bernstein: SYN cookies