Some Observations and Opinions on Security Troubleshooting with syslog vs debug

Over the years in both the classroom and the customer site I had the “opportunity” to troubleshoot a Cisco security deployment. I put that word in quotes because, let’s face it, troubleshooting is done to solve problems which can be excellent learning opportunities.  Two tools which frequently are chosen for this task, which are native to most Cisco devices, are: a) debug and b) syslog. This post offers my personal recommendations as when to choose one versus the other.

Let’s start with the use of debugging. The debug command is supported on both the ASA security appliance and the Cisco IOS® router. An important and noteworthy implementation difference between the two platforms is that logging must be enabled for debug output to be seen on the router, but not on the ASA. A sample debug output for an IPSec Internet Key Exhange (IKE) Phase I exchange:

Jan 19 21:37:58 [IKEv1]: IP = 200.200.20.2, Error processing payload: Payload ID: 1
Jan 19 21:37:58 [IKEv1 DEBUG]: IP = 200.200.20.2, IKE MM Responder FSM error history (struct &0xc8f8bcc8)  <state>, <event>:  MM_DONE, EV_ERROR–>MM_START, EV_RCV_MSG–>MM_START, EV_START_MM–>MM_START, EV_START_MM–>MM_START, EV_START_MM–>MM_START, EV_START_MM–>MM_START, EV_START_MM–>MM_START, EV_START_MM
Jan 19 21:37:58 [IKEv1]: IP = 200.200.20.2, Removing peer from peer table failed, no match!
Jan 19 21:37:58 [IKEv1]: IP = 200.200.20.2, Error: Unable to remove PeerTblEntry

Most observers would agree that such output is quite cryptic, and, in most cases, requires a fairly thorough knowledge of the protocol to make sense of the messages. I remember needing a few minutes after observing the cryptic message along the lines of  “…packet is malformed and failed sanity check” for me to correctly conclude that this was caused by mismatched VPN preshared keys!

By contrast, using syslog often provides a “big picture” view instead of getting “lost in the weeds”.  Below is a screenshot of a very valuable tool now bundled with the Adaptive Security Device Manager known as the Real Time Syslog Viewer:

Not only does the use of the different colored fonts on the white background present a more appealing and “easier on the eye” format, but the colors were chosen such that the “cooler” colors (blue, purple) represent more trivial messages while the “hotter” colors (yellow and red (not shown)) represent the more serious events. Also, the three tabs at the bottom should not be overlooked as these provide additional event explanations, recommendations, and details for the highlighted row.

In conclusion, I don’t wish to appear too negative toward the use of appropriate “debug” commands.  They are well-suited to understanding the operation of a protocol, especially when coupled with a network sniffer application. Secondly, as I told numerous students, verbose output (versus no output) is usually best — an indication of success. However, for quick problem identification, the often unambiguous output of a real time log viewer can’t be beat.

In this article

Join the Conversation