This post is the first part of a multi-part series (I haven’t decided how many parts there will be) on effective Cisco security deployment troubleshooting. While later posts will focus more on either a particular device or technology, I’ll cover some general guidelines and concepts here. Outlined below are fundamental principles based more upon personal experience than on published material.
The types of problems encountered in network topologies where security devices are present vary:
- No connection
- Packet denied
- Authentication/Authorization failure
- False alarms (also known as False Positives)
This is only a partial list at best. However, a generalized common approach can be taken. The primary goal, first and foremost, should be isolate the problem. This is usually in two phases:
- Limiting analysis to a single device (firewall, router, IPS, switch, server, or workstation)
- Narrowing the focus to an application or configuration component or setting
Once this is done, specific tools can be brought to bear on the situation.
At this point you’re probably thinking, “Tell me something I don’t already know!”
Fair enough, let’s examine some general troubleshooting principles I found to be useful.
Principle #1: Focus Your Efforts on the Receiver
In diagnosing connection and VPN problems, you frequently need to probe why the attempt is being denied once there is evidence that the packets arrive successfully. Too often SSL or IPSec VPN client logs don’t provide enough information on why connections fail. Consequently, the receiver frequently provides the detail needed through selective debugging and logging.
Principle #2: Know the Protocol
For applications that use alternate TCP/UDP port pairs as well as VPN session establishment, knowledge of both the sequence and the timing of the protocol is a necessary and fundamental component of determining the fault. With this knowledge comes the helpful skill of being able to “pick apart” a decoded capture using a sniffer application which can help determine exactly what is or isn’t being communicated between the endpoints.
Principle #3: Consider the Use of Timestamps
With the increasing speed of both networks and individual computer processing power, the completion of connections and sessions can sometimes happen within a one second interval. If Network Time Protocol (NTP) is deployed not only on security devices but also on the session endpoints, the use of timestamps for syslog, debug, and sniffer traces becomes especially useful in understanding the proper sequencing of the operation being examined.
Principle #4: Use Multiple Tools
From the preceding principles you can conclude that I absolutely recommend the use of multiple diagnostic sources. A reporting application at the receiving side (e.g. Failed Attempts in Cisco Secure ACS) can effectively supplement the often obscure and abbreviated debug/syslog output.
When it comes to troubleshooting, I encourage you to “think outside the box” and consider taking an unconventional approach under special circumstances. A sniffer trace from a VPN virtual adapter, for example, has been helpful to me.
I’ll put these principles into practice in later posts by providing more detailed examples of typical security scenarios.