You are hereDebugging IPsec VPN Tunnels
Debugging IPsec VPN Tunnels
Today I had to debug an IPsec VPN tunnel between OpenSwan and Cisco PIX.
Being the third person to work on this 'problem', you could call this last-possible-escalation. Time to give a little hint to know what to look at and where the problem can be. Let's hope these tips&tricks will help others...
Usually IPsec VPN problems can be resumed to five points (order is important):
- Incorrect Phase 1 settings
- Incorrect Phase 2 settings
- Wrong routing
- NATting when the VPN traffic should be NATted
- Incompatible IPsec stacks
Phase 1 & Phase 2
The first two points seem very simple. But unfortunately it's usually the first place where the problem lies. They key is to clearly choose the settings and give them to 'the other side'. Communication is the key of success here.
Once you and 'the other side' have configured the tunnel, check the logs. If you see messages saying that 'Phase 1 is completed' stop looking at the settings of Phase 1. They are correct ! Check Phase 2 now.
If your tunnel doesn't complete Phase 1, then check the settings again, and again. Ask the other party to dictate what they see. The annoying part is that when you have different brands of VPN devices, the interface to configure these is usually different. So it's more difficult to compare the settings.
A one-million-euro rule: "If you can't see the configuration with your own eyes, don't trust what they say. See to believe !"
Ok, now your IPsec implementation tells you the tunnel is up & running. But "it still doesn't work".
Wrong routing
A simple traceroute or tracert should tell you what is wrong. Nothing very difficult.
NATting
Traffic that enters the VPN tunnel shouldn't be NATted. Usually the firewall NATs all traffic from inside to outside. As your VPN tunnel also has traffic from inside to outside it can become NATted. Check these settings to prevent your tunnel-traffic to be natted. If natted it won't match the encryption domain and will not enter the tunnel.
If the tunnel works for traffic in one direction, but not in the other direction the solution is 'no NAT'.
With Cisco devices the debug icmp trace is a great help. (don't forget the terminal monitor to see the debugging). Now start a ping from a host in your network and see the output in your CLI. Full info about the ICMP packet and what NAT has been applied will appear on the screen.
Incompatible IPsec stacks
This happens, not often but it happens. It's the most difficult thing to detect as you must be 100% sure that your settings are correct. Don't take this easy solution as granted too fast.
To solve this try to upgrade your firmware or use another device with a newer/older firmware and the same configuration.
Try with two devices of the same brand. As long as it works the customer will probably be happy.
A last thing
A last controversial thing I had to do in my career was to take a difficult decision. I had to make a VPN tunnel work between a Fortigate and a Symantec thing (sorry, but I really can't call this thing a firewall out of respect of real firewalls). First the Symantec firewall didn't show certain settings, so I had to reverse-engineer the default configuration. Also another downside was that the device just wouldn't bring up the tunnel all by itself when seeing 'interesting traffic'. It had to be started from the Fortigate. Once that stupid VPN tunnel was working (both directions) it just couldn't stay stable. The Fortigate already had 6 stable VPN connections to other sites, all running FortiOS (based on Linux). But this connection to that crap just crashed regularly without reason.
After many hours of troubleshooting I took my courage with both hands, called the customer and told him: "I could continue debugging this crap, but I can't promise any result. Instead I ask you to throw that Symantec thing away and replace it with a Fortigate of less than €500. In the end I will have a working VPN tunnel in maximum 4 hours, making this a total cost of €900. Compare this to the unknown cost of further troubleshooting without certainty of results.
The customer thanked me for my honesty, a day later they ordered the device and once it was delivered I had a working, stable VPN tunnel in a few hours.
Lesson learned.
IPSec is a strong and reliable piece of technology, but it's too much a bag of large standards with too many parameters to choose from.
Just every proprietary firewall-box provider implements his own set of cutomiseable parameters related to other hardcoded defaults.
Yes, Symantec sucks hard. More than e.g. Watchguard.
If you want to be sure to be able to reliably connect different ipsec-stacks, just stick to open source implementations.
All the other problems you mentioned are about human mistakes. Watch and learn :)
Communication is often the first problem. IPSec has a lot of options. Depending on the vendor, some may not even be configurable. People usually just know the few options they can set in their management UI, but know nothing about all the hardcoded values, which are important if you want to set up a VPN between devices of different brands.
The main difference is usually how the devices deal with multiple subnets at either end of the tunnel. Say you have 192.168.1.0/24 and 192.168.2.0/24 on one end and 192.168.100.0/24 at the other end. Some vendors can simply create 1 VPN and route all the traffic through. Others will set up two VPNs, one between 192.168.1.0/24 and 192.168.100.0/24 and one between 192.168.2.0/24 and 192.168.100.0/24.
Few devices are flexible enough to allow both methods.
I have a template which I always use to communicate/negotiate the VPN configuration settings with the remote side. If they know their stuff, it usually takes only a few minutes to configure the VPN.
But don't try setting up VPNs with home users devices such as Dlink unless you have to much spare time. Decent firewalls cost only a couple hundred euros, no matter what brand you choose. That is a lot less than you would spend paying someone by the hour to debug the problem.