Today I had to debug an IPsec VPN tunnel between OpenSwan and Cisco PIX.
Being the third person to work on this 'problem', you could call this last-possible-escalation. Time to give a little hint to know what to look at and where the problem can be. Let's hope these tips&tricks will help others...

Usually IPsec VPN problems can be resumed to five points (order is important):

1. Incorrect Phase 1 settings
2. Incorrect Phase 2 settings
3. Wrong routing
4. NATting when the VPN traffic should be NATted
5. Incompatible IPsec stacks

Phase 1 & Phase 2

The first two points seem very simple. But unfortunately it's usually the first place where the problem lies. They key is to clearly choose the settings and give them to 'the other side'. Communication is the key of success here.
Once you and 'the other side' have configured the tunnel, check the logs. If you see messages saying that 'Phase 1 is completed' stop looking at the settings of Phase 1. They are correct ! Check Phase 2 now.
If your tunnel doesn't complete Phase 1, then check the settings again, and again. Ask the other party to dictate what they see. The annoying part is that when you have different brands of VPN devices, the interface to configure these is usually different. So it's more difficult to compare the settings.

A one-million-euro rule: "If you can't see the configuration with your own eyes, don't trust what they say. See to believe !"

Ok, now your IPsec implementation tells you the tunnel is up & running. But "it still doesn't work".

Wrong routing

A simple traceroute or tracert should tell you what is wrong. Nothing very difficult.

NATting

Traffic that enters the VPN tunnel shouldn't be NATted. Usually the firewall NATs all traffic from inside to outside. As your VPN tunnel also has traffic from inside to outside it can become NATted. Check these settings to prevent your tunnel-traffic to be natted. If natted it won't match the encryption domain and will not enter the tunnel.

If the tunnel works for traffic in one direction, but not in the other direction the solution is 'no NAT'.

With Cisco devices the debug icmp trace is a great help. (don't forget the terminal monitor to see the debugging). Now start a ping from a host in your network and see the output in your CLI. Full info about the ICMP packet and what NAT has been applied will appear on the screen.

Incompatible IPsec stacks

This happens, not often but it happens. It's the most difficult thing to detect as you must be 100% sure that your settings are correct. Don't take this easy solution as granted too fast.

To solve this try to upgrade your firmware or use another device with a newer/older firmware and the same configuration.
Try with two devices of the same brand. As long as it works the customer will probably be happy.

A last thing

A last controversial thing I had to do in my career was to take a difficult decision. I had to make a VPN tunnel work between a Fortigate and a Symantec thing (sorry, but I really can't call this thing a firewall out of respect of real firewalls). First the Symantec firewall didn't show certain settings, so I had to reverse-engineer the default configuration. Also another downside was that the device just wouldn't bring up the tunnel all by itself when seeing 'interesting traffic'. It had to be started from the Fortigate. Once that stupid VPN tunnel was working (both directions) it just couldn't stay stable. The Fortigate already had 6 stable VPN connections to other sites, all running FortiOS (based on Linux). But this connection to that crap just crashed regularly without reason.

After many hours of troubleshooting I took my courage with both hands, called the customer and told him: "I could continue debugging this crap, but I can't promise any result. Instead I ask you to throw that Symantec thing away and replace it with a Fortigate of less than €500. In the end I will have a working VPN tunnel in maximum 4 hours, making this a total cost of €900. Compare this to the unknown cost of further troubleshooting without certainty of results.

The customer thanked me for my honesty, a day later they ordered the device and once it was delivered I had a working, stable VPN tunnel in a few hours.

Lesson learned.