Many students who have either taken training classes on the Cisco PIX or ASA security appliances or read associated published material are already acquainted with the phrase “TCP SYN cookie”. This post will serve to explain some of the historical background, as well as the numerous hardware implementations.
The TCP SYN Flood denial-of-service attack is now more than fifteen years old. Kevin Mitnick used it as part of his now-famous coordinated attack against his future prosecutor, Tsutomu Shimomura. A little more than five years later, SYN floods played a part in the Distributed Denial of Service (DDoS) attacks that brought down Yahoo, eBay, CNN, and others. Now SYN flood scripts and executables are freely available for download (more details here and here).
An early Cisco IOS® Firewall implementation utilized a feature known as TCP Intercept which could be operated in either monitor or intercept modes. These would either report excessive TCP SYN activity via syslog, or prevent it entirely. The drawback of this approach was its use of a memory-resident TCP state table, which, during peak periods of flooding, could result in depleted router resources. So while attackers might be thwarted in their efforts to bring down servers, they might bring down the router instead!
In the mid-1990’s a variety of proposals were put forth in an attempt to provide a more effective TCP SYN flood defence mechanism. Among these is the TCP SYN Cookie mechanism offered by Daniel J. Bernstein and briefly diagrammed here:
TCB (Transmission Control Block) referenced above is a data structure which holds the connection state information and can be several hundred bytes in size depending on the implementation in the operating system. What the SYN Cookie mechanism does is to encode information that would normally be kept in the memory resident TCB in the Initial Sequence Number (or cookie as in the diagram above) returned in the SYN-ACK.
The acknowledgement (ACK) from the Initiator (or client) to this sequence number can be decremented by one to confirm the state information for this client. Note that this encoding scheme allows the Listener (or server) to purge any connection state table as indicated by the destruction of the TCB above.
TCP SYN Cookies are not only available on the ASA and PIX firewalls (first introduced in PIX OS 6.2), but also on the 12000 series routers supporting the Cisco IOS XR software as well as the Application Control Engine (ACE) 4700 series.