Understanding IP Sec VPN Operation:

By | July 10, 2020


IPSEC [IP Security] is used to secure IP [IPV4 and IPV6] traffic over an untrusted network [such as the Internet] by enabling a system to select required security protocols [AH or ESP], algorithms and cryptographic keys.

IP Sec Protocol used in a site-to-site VPN which interconnects two sites without requiring the computers at those sites to have any specialized VPN software installed [ like a remote access VPN ].

Why IP Sec

Internet Protocol [IP] is not secure and was designed in the early stages of internet where security was not an issue. Below are the possible security issues we may face when we send IP traffic over the internet

  • There is No data integrity or confidentiality IP provides.
  • Source Spoofing attack
  • Reply packet attack

IP Sec Frame work diagram

IP Sec Provide four major services

  1. Confidentiality: Means don’t disclose the data to unauthorized user. IP sec accomplish this by using encryption algorithms such as DES, 3DES, AES.
  2. Integrity: Prevent the information from being modified by unauthorized user. IP sec accomplish this by using hashing algorithms such as MD5, SHA, HMAC.
  3. Authentication : Make sure the data exchanged with the right party.IP Sec accomplish this using pre-shared key, RSA signature key
  4. Rejection of replayed packets: Goal is here to avoid hackers injecting or making changes in packets that travel from a source to a destination. IP sec accomplish this by using sequence number and window size [default value – 64]. This check is done before integrity check. Packet will be dropped in below cases

Each packet sent by sender will be assigned sequence number [e.g. packet 1- sequence 1] and receiving end will note this number so. So in this case if attacker send a packet and

  • If the sequence number falls within the window and was previously received, the packet is dropped, and the replay counter is incremented.
  • If the sequence number is less than the lowest sequence in the window, the packet is dropped, and the replay counter is incremented

Below logs message will be logged if check fail and packet drop

%CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed connection id=#, sequence


Use below command to check the drops

Show crypto IPsec sa peer < IP address > detail | in pkts replay failed


IPSEC Mode of operation

IPsec can be run in either tunnel mode or transport mode.

  • Tunnel mode:

With tunnel mode, an entire IP packet is encapsulated with an AH or ESP header and an additional outer IP header. The IP addresses of the outer IP header are the tunnel endpoints [ in our example CE 1 : and CE 2 IP address ] , and the IP addresses of the encapsulated IP header are the ultimate source and destination addresses [ Bob PC and Lima PC IP address ].

Widely used between VPN gateways [CE-1 and CE-2] and hence it is used in Site to Site VPN Tunnel.

  • Transport Mode :

With Transport mode, IPsec encrypts only the IP payload and does not encrypt original IP Header [Bob PC and Lima PC IP address].It uses AH or ESP header for Protection.

Widely used between two end system [Bob and Lima PC if they are using VPN client software] and hence being used in remote access VPN.

Encapsulation security payloads used by Tunnel and Transport modes:

  • Encapsulation Security Payload (ESP)

Work at TCP layer of the OSI model [port number 50].It provide the service to IP and upper layer protocol such as TCP. ESP does not encrypt the original IP header [new header in tunnel mode, original header in transport mode], and do integrity check for it and encrypts only the IP data by placing a header in between the original IP header and data. It provides below services by using ESP header, trailer and authentication.

  1. Data confidentiality
  2. Data integrity
  3. Data origin authentication.
  4. Replay attacks.

  • Authentication Header (AH)

Work at TCP layer of the OSI model [port number 51]. It doesn’t provide the confidentiality [no encryption] so it is not popular but it provide the data authentication and reply attack services. The AH provide protection to outer header [new header in tunnel mode, original header in transport mode] that ESP cannot provide.

This is why if NAT Address being used for communication the ESP doesn’t care but AH does and those packet will be dropped during integrity check process.

How IP Sec works

We have finished the basic overview of IPsec. Let’s understand the IPsec operation by going through both theory and practical at a time. We can call this section as IPsec theory in action

Above is our topology. By using IPsec tunnel we want to secure the communication between client Bob [work at Site A] and client Lima [Work at HQ] over internet. Let’s assume for now that both the router [CE1, CE2] are configured for IPsec operation [I will explain the commands as I explain the theory].

The traffic between the Bob and Lima for both routers is interesting traffic [traffic need to encrypted] and that can be defined using access list / Prefix list and then mapped with crypto map.

CE-1 #

CE-2 #

Let’s say client Bob open his computer command prompt and pings the Lima computer to check whether he can reach her computer over the internet. This first packet [ICMP Echo request] of the Ping is sent non-encrypted packet with original source and destination IP address as shown below [Private IP address: RFC 1918].

First ICMP Packet without Encryption and with original source [CE-1] and destination ip address [CE-2]

When this packet hit the router CE- 1, it looks source and destination IP address which matches ACL configured in crypto map and it than know this is interested traffic and needs to be encrypted so it check whether IPsec SA – Phase 2 [more on this in below section] is active or not [if IP sec SA phase 2 has been already established than it directly encrypt and encapsulate that packet and forward it to other side]. Here in our case it will not has this is the first packet of the communication so Router CE-1 now first need to Build ISAKAMP Security Association [SA] with CE-2 using Internet Key Exchange Protocol which run over UDP port 500 . This is the Phase 1.

<- Interesting Traffic Received->

We See below message on CE-1 when it receive that ICMP echo request

Please note I have run below debug command in order to see the packets.

Phase 1 – ISAKAMP SA

Phase 2 – IPSEC SA

CE-1 first needs to build ISAKAMP SA to Secure IPSEC SA which again secure user data

Please note – An SA is agreement between two devices to understand on how to do crypto. SA are built when two VPN devices successfully negotiate the security policy. SA are built in two Phases.

1. Phase 1- ISAMKAMP SA

2. Phase 2 – IPsec SA

What is Phase 1 – ISAKAMP SA?

This is like a management communication channel between two VPN routers. This is built to secure the management communication. It can be configured in two modes.

  1. Aggressive mode : Use three-way packet exchange to establish tunnel, Fast but not secure
  2. Main mode: Use six-way packet exchange to establish tunnel, slow but secure.

In our case as we are using two cisco router it by default use main mode. In 6-way packet exchange each CE-1 and CE-2 will send three packet to each other to build the tunnel.

CE-1 started beginning mode exchange and sent First Packet to CE-2

As we see below the first packet contain proposal message which has all the parameters which we set using below command. This should match on both side.

1. Encryption algorithm as AES and

2. Authentication has Pre share Key [PSK].The key used here is “ccie “

CE-2 receive that packet. It process it and see the parameters[Encryption key, Authentication Type, DH group , Hashing algorithm, Key life time. ] in it match with its policy set. If it matches it send its own proposal.

CE-1 receives that packet and process it and if everything match it accept the proposal.

The first two packet [Type 1 to Type 2 message] once matched. The CE-1 and CE-2 now exchanges Type 3 and Type 4 messages which is used for Diffie-Hellman (DH) exchange and contain DH public value and a random number.

CE-1 Logs

IKE_I_MM3 > in below logs indicate Message 3 is sent.

IKE_I_MM4 > in below logs indicate Message 4 is received.

Once the DH Values has been exchanged CE-1 and CE-2 exchanges Packet 5 [Message type 5] and Packet 6 [Message type 6] to authenticate among themselves. These message will be encrypted using the agreed upon encryption methods established in message types 1 and 2.

CE- 1 Logs

Once they authenticate with each other the IKE_Phase 1 is completed and you will see below message.

Now if we did some changes in policy set and we want to reflect it right away [ usually ISAKMAP phase 1 messages exchanged once in life time ]than we need to clear the session using below command.

As soon as we did that CE-1 will now send the informational message to CE-2 with regards to the changes and these message which will be encrypted. With this example we can say that Phase 1 is for secure communication and it is management channel.

If pre shared key misconfigured between the peers we will see below messages.

Once IKE Phase 1 is complete, IKE Phase 2 is initiated.

CE-2 initiate IKE Phase 2 with the protection of IKE phase 1. Phase 2 occur in quick mode and it is 3 – way packet exchange. It can occur in

  • Without key exchange – No PFS enabled.
  • With Key exchange – PFS enabled, DH run once more to generate the shared secret.

In our case it will without key exchange as we dint enable PFS due to this phase 2 might use same key which is being used in phase 1.

CE-2 send first packet [Message type 1 – encrypted] to CE-1 in Quick Mode. This is proposal message and contain payloads[ Hash,SA] and the attribute set using Transform set command shown below. This should be match on both side.

Phase 2 configuration

CE-2 receive the First Packet and process the payload. If it match with its own configuration set in transform set it accept and send its own proposal message [Packet 2 – Message type 2]

CE-1 receives packet-2 and process payload. If it matches with its own configuration set in transform set than the message will be accepted. Here it matches shown below so the CE-1 send Packet -3 as acknowledgement message. Once this is done the Phase -2 will be successfully completed.

Third packet.

Once the phase 2 and phase 1 is up than all the traffic between Bob and Lima PC will be encrypted and encapsulated in ESP header as shown below.


With the above packet analysis we can conclude that IP Sec Protocol can be used in a site-to-site VPN to interconnects two sites without requiring the computers at those sites to have any specialized VPN software installed [ like a remote access VPN ]. I hope this article help

One thought on “Understanding IP Sec VPN Operation:

Leave a Reply

Your email address will not be published. Required fields are marked *