Skip links
Main content

Protection against SYN Floods

Saturday 14 March 2009 15:03

Just after lunch our office is a calm and peaceful place. The only sound comes from a handful of developers discussing possible solutions for a challenging problem in a project...

By Bert Slagter

Suddenly our nagios IRC bot politely informs us that the HTTP service is down for one of the servers. Fortunately SSH still works like a charm. A quick inspection reveals no peculiarities, except for an amazing number of Apache clients doing nothing. A quick glance in the logs learns us that Apache has reached its MaxClients setting. An Apache restart doesn't help, it reaches its MaxClients setting again within the minute.

# tail /var/log/apache2/error.log

[Fri Dec 05 14:31:30 2008] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Fri Dec 05 14:50:54 2008] [notice] caught SIGTERM, shutting down
[Fri Dec 05 14:50:56 2008] [notice] Apache/2.2.x (Debian) configured -- resuming normal operations
[Fri Dec 05 14:51:35 2008] [error] server reached MaxClients setting, consider raising the MaxClients setting

Could be a (D)DoS attack. However, the server doesn't show typical signs of a DoS attack like high load or network traffic. Next stop is netstat which shows us a huge number of tcp connections in SYN_RECV state:

# netstat -n -p TCP

tcp 0 0 10.1.1.1:80 91.64.4.146:64979 SYN_RECV -
tcp 0 0 10.1.1.1:80 84.24.103.112:4005 SYN_RECV -
tcp 0 0 10.1.1.1:80 79.223.69.239:61510 SYN_RECV -
tcp 0 0 10.1.1.1:80 67.86.135.44:43312 SYN_RECV -
tcp 0 0 10.1.1.1:80 86.88.67.226:50600 SYN_RECV -
tcp 0 0 10.1.1.1:80 173.20.137.110:3813 SYN_RECV -
tcp 0 0 10.1.1.1:80 84.58.10.121:4878 SYN_RECV -
tcp 0 0 10.1.1.1:80 91.37.40.151:2408 SYN_RECV -
tcp 0 0 10.1.1.1:80 173.20.137.110:3441 SYN_RECV -
...

Lots of half-open connections are a good indication of a SYN Flood attack. So we need to do two things:

  1. Harden the TCP/IP stack
  2. Block attackers

Harden the TCP/IP stack

If you're not familiar to SYN Floods or other TCP-based resource starvation attacks, read this SecurityFocus article. It also covers hardening your TCP/IP stack against SYN Flooding and SYN Spoofing. Most important is enabling SYN Cookies.

Block attackers

Now we've hardened the TCP/IP stack, spoofing has been eliminated. The attacking source address is really the attacker, so it makes sense to block it. However, these days it's not very hard to create, buy or rent a large botnet consisting of thousands of hacked computers ready to be engaged in massive DDoS attacks. So manually block IP's won't work.

We'll have to write a script to automatically block attacking IP's. The script consists of three parts:

  1. Find out which IP's to block
  2. Block IP
  3. Unblock IP after 24 hours

The first part is the hardest part. In our case, we could just take all IP's having too many connections in SYN_RECV state. In your case you might need to use another condition, e.g. count requests to a specific URI.

$netstat = shell_exec('/bin/netstat -apn | grep SYN_RECV');
$lines = preg_split('/[\r\n]+/', $netstat);

$ips = array();
foreach ($lines as $line) {
   $fields = preg_split('/\s+/', $line);
   if(preg_match('/[\d\.]+/is', $fields[4], $match)) {
      $key = $match[0];
      $ips[$key] = empty($curr[$key]) ? 1 :
     ($ips[$key] + 1);
   }
}

Now we have all IP's and number of concurrent connections in SYN_RECV state. Next step is block IP's with more than 50 connections. This number is quite random, any sane treshold will do.

foreach ($ips as $key => $val) {
   if ($val > 50) {
      exec("sbin/iptables -A INPUT -s " .
     escapeshellarg($key) . " -j DROP");
   }
}

For the last step you need to create your own administration of blocked IP's and timestamps. Check out the full script [Download] for this and some other features like a whitelist and simple logging.

Reinventing the wheel?

You might wonder why we try to reinvent the wheel. Though avoiding to reinvent the proverbial wheel is a good practice and key principle in software development, it's useful to rethink the wheel concept from time to time. Or to quote Jeff Atwood: Don't Reinvent The Wheel, Unless You Plan on Learning More About Wheels.

Most high profile websites will probably have a dedicated intrusion detection or prevention system (IDS/IPS) in place. IT security companies can implement a commercial solution, e.g. Cisco, IBM or Juniper systems. You could also try to build one yourself. Snort is a free and open source IDS/IPS. It's the de facto standard and used by many commercial solutions as well.

However, for most websites or web applications an IDS/IPS is overkill. Start with good server configuration, hardened OS and server software and a good router and firewall. And add an IDS or custom IP blocking script if necessary.

Finally

In this case we controlled the attack within 5 minutes after we launched the block-ips script. The websites came back online and the attack stopped after three days.

Further reading:

« Back

Reactions on "Protection against SYN Floods"

s.venkatesan
Placed on: 10-25-2009 08:37
Hello,

Idea of blocking the TCP flooding is nice. The same concept I proposed for IPv6 TCP flooding attack with the legitimate IP address.

But the problem with me for the solution is how to fix the threshold value because the syn message may be blocked in the middle of the transportation and also as per the survey, the routers on the network may drop some packet due to overload that packet may be the syn/ack packet.

How you set the threshold value 50.

regards
s.venkatesan
Edited
Procurios Admin has edited this message on: 11-04-2009 12:26
xenki
Placed on: 01-08-2011 17:39
great and useful info thanks very much i'll be back again
no luck
Placed on: 05-27-2011 07:09
Really great script. Thanks for saving my day.

Log in to comment on news articles.