NIST SP 800-77 Rev. 1 日本語訳

NIST Special Publication 800-77

Revision 1

Elaine Barker Quynh Dang Sheila Frankel Karen Scarfone Paul Wouters

This publication is available free of charge from: https://doi.org/10.6028/NIST.SP.800-77r1

NIST Special Publication 800-77

Revision 1

Elaine Barker Quynh Dang Sheila Frankel*

Computer Security Division Information Technology Laboratory

Karen Scarfone Scarfone Cybersecurity Clifton, VA

Paul Wouters Red Hat Toronto, ON, Canada

*Former employee; all work for this publication was done while at NIST

This publication is available free of charge from: https://doi.org/10.6028/NIST.SP.800-77r1

June 2020

U.S. Department of Commerce Wilbur L. Ross, Jr., Secretary

National Institute of Standards and Technology

Walter Copan, NIST Director and Under Secretary of Commerce for Standards and Technology

Authority

This publication has been developed by NIST in accordance with its statutory responsibilities under the Federal Information Security Modernization Act (FISMA) of 2014, 44 U.S.C. § 3551 et seq., Public Law (P.L.) 113-283. NIST is responsible for developing information security standards and guidelines, including minimum requirements for federal information systems, but such standards and guidelines shall not apply to national security systems without the express approval of appropriate federal officials exercising policy authority over such systems. This guideline is consistent with the requirements of the Office of Management and Budget (OMB) Circular A-130.

Nothing in this publication should be taken to contradict the standards and guidelines made mandatory and binding on federal agencies by the Secretary of Commerce under statutory authority. Nor should these guidelines be interpreted as altering or superseding the existing authorities of the Secretary of Commerce, Director of the OMB, or any other federal official. This publication may be used by nongovernmental organizations on a voluntary basis and is not subject to copyright in the United States. Attribution would, however, be appreciated by NIST.

National Institute of Standards and Technology Special Publication 800-77 Revision 1 Natl. Inst. Stand. Technol. Spec. Publ. 800-77 Rev. 1, 166 pages (June 2020) CODEN: NSPUE2

This publication is available free of charge from: https://doi.org/10.6028/NIST.SP.800-77r1

Certain commercial entities, equipment, or materials may be identified in this document in order to describe an experimental procedure or concept adequately. Such identification is not intended to imply recommendation or endorsement by NIST, nor is it intended to imply that the entities, materials, or equipment are necessarily the best available for the purpose.

原本PDFの抜粋画像

There may be references in this publication to other publications currently under development by NIST in accordance with its assigned statutory responsibilities. The information in this publication, including concepts and methodologies, may be used by federal agencies even before the completion of such companion publications. Thus, until each publication is completed, current requirements, guidelines, and procedures, where they exist, remain operative. For planning and transition purposes, federal agencies may wish to closely follow the development of these new publications by NIST.

Organizations are encouraged to review all draft publications during public comment periods and provide feedback to NIST. Many NIST cybersecurity publications, other than the ones noted above, are available at https://csrc.nist.gov/publications.

Comments on this publication may be submitted to: National Institute of Standards and Technology Attn: Computer Security Division, Information Technology Laboratory 100 Bureau Drive (Mail Stop 8930) Gaithersburg, MD 20899-8930 Email: revision_of_SP800-77@nist.gov

All comments are subject to release under the Freedom of Information Act (FOIA).

The Information Technology Laboratory (ITL) at the National Institute of Standards and Technology (NIST) promotes the U.S. economy and public welfare by providing technical leadership for the Nation’s measurement and standards infrastructure. ITL develops tests, test methods, reference data, proof of concept implementations, and technical analyses to advance the development and productive use of information technology. ITL’s responsibilities include the development of management, administrative, technical, and physical standards and guidelines for the cost-effective security and privacy of other than national security-related information in federal information systems. The Special Publication 800-series reports on ITL’s research, guidelines, and outreach efforts in information system security, and its collaborative activities with industry, government, and academic organizations.

Internet Protocol Security (IPsec) is a widely used network layer security control for protecting communications. IPsec is a framework of open standards for ensuring private communications over Internet Protocol (IP) networks. IPsec configuration is usually performed using the Internet Key Exchange (IKE) protocol. This publication provides practical guidance to organizations on implementing security services based on IPsec so that they can mitigate the risks associated with transmitting sensitive information across networks. The document focuses on how IPsec provides network layer security services and how organizations can implement IPsec and IKE to provide security under different circumstances. It also describes alternatives to IPsec and discusses under what circumstances each alternative may be appropriate.

communications security; Internet Key Exchange (IKE); Internet Protocol (IP); Internet Protocol Security (IPsec); network layer security; networking; virtual private network (VPN).

The terms “shall” and “shall not” indicate requirements to be followed strictly in order to conform to the publication and from which no deviation is permitted.

The terms “should” and “should not” indicate that among several possibilities, one is recommended as particularly suitable, without mentioning or excluding others, or that a certain course of action is preferred but not necessarily required, or that (in the negative form) a certain possibility or course of action is discouraged but not prohibited.

The terms “may” and “need not” indicate a course of action permissible within the limits of the publication.

The terms “can” and “cannot” indicate a possibility and capability, whether material, physical, or causal.

The authors—Elaine Barker, Quynh Dang, and Sheila Frankel of NIST; Karen Scarfone of Scarfone Cybersecurity; and Paul Wouters—wish to thank everyone who has contributed to this revision of Special Publication (SP) 800-77, particularly Andrew Regenscheid, David Waltermire, and Lily Chen of NIST; Deb Cooley and James Banoczi of the National Security Agency (NSA); and Graham Bartlett of Cisco.

The authors acknowledge the following individuals (with their original affiliations) and organizations that assisted in the development of the original SP 800-77:

Audience

This document has been created for network architects, network administrators, security staff, technical support staff, and computer security program managers who are responsible for the technical aspects of preparing, operating, and securing networked infrastructures. The material in this document is technically oriented, and it is assumed that readers have at least a basic understanding of networking and network security.

Trademark Information

All names are registered trademarks or trademarks of their respective companies.

Patent Disclosure Notice

NOTICE: The Information Technology Laboratory (ITL) has requested that holders of patent claims whose use may be required for compliance with the guidance or requirements of this publication disclose such patent claims to ITL. However, holders of patents are not obligated to respond to ITL calls for patents and ITL has not undertaken a patent search in order to identify which, if any, patents may apply to this publication.

As of the date of publication and following call(s) for the identification of patent claims whose use may be required for compliance with the guidance or requirements of this publication, no such patent claims have been identified to ITL.

No representation is made or implied by ITL that licenses are not required to avoid patent infringement in the use of this publication.

Internet Protocol Security (IPsec) is a suite of open standards for ensuring private communications over public networks. It is the most common network layer security control, typically used to encrypt Internet Protocol (IP) traffic between hosts in a network and to create a virtual private network (VPN). A VPN is a virtual network built on top of existing physical networks that provides a secure communications mechanism for data and control information transmitted between computers or networks. IPsec is also used as a component that provides the security for many other internet protocols. The User Datagram Protocol (UDP) usage guidelines [1] specify IPsec as one of the methods to secure UDP.

The Internet Key Exchange (IKE) protocol is most commonly used to establish IPsec-based VPNs. The terms IKE and IPsec are often used interchangeably, although that is not correct. In practice, the terms “IPsec VPN,” “IKEv2 VPN,” “Cisco IPsec,” “IPsec XAUTH¹,” and “L2TP2/IPsec” all refer to IPsec-based VPN connections. Some examples of technologies and protocols that use IKE and/or IPsec are:

VPNs protect communications carried over public networks, such as the internet, as well as private networks, such as fiber networks or Multiprotocol Label Switching (MPLS) networks. A VPN can provide several types of data protection, including confidentiality, integrity, data origin authentication, replay protection, and access control. The primary VPN architectures are as follows:

The guide provides an overview of the types of security controls that can offer protection for network communications that are widely used throughout the world. IP communications are composed of four layers that work together: application, transport, network, and data link. Security controls exist for network communications at each of the four layers. As data is prepared for transport, it is passed from the highest to the lowest layer, with each layer adding more information. Because of this, a security control at a higher layer cannot provide full protection for lower layers because the lower layers add information to the communications after the higher layer security controls have been applied. The primary disadvantage of lower layer security controls is that they are less flexible and granular than higher layer controls. Accordingly, network layer controls have become widely used for securing communications because they provide a more balanced solution.

IPsec is a network layer security protocol with two main components:

Optionally, IPsec can use the IP Payload Compression Protocol (IPComp) to compress packet payloads before encrypting them, but this has not been widely used.

Only implementations of NIST-approved cryptographic algorithms specified in Federal Information Processing Standards (FIPS) or NIST Special Publications (SPs) and contained in FIPS-validated cryptographic modules shall be used in IPsec VPN deployments for compliance with this guidance. The FIPS 140 specifications [13][14] define how cryptographic modules will be validated. One requirement of FIPS 140 is that the module be capable of operating in a mode where all algorithms are NIST approved. NIST-approved algorithms are specified in a FIPS (e.g., FIPS 180, Secure Hash Standard) or in a NIST SP (e.g., SP 800-56A, Recommendation for Pair-Wise Key Establishment Schemes Using Discrete Logarithm Cryptography). Some implementations can run in both FIPS mode and non-FIPS mode, so it is important to set and verify the mode of operation of the IKE and IPsec modules.

The Cryptographic Module Validation Program (CMVP) is a joint effort between NIST and the Communications Security Establishment (CSE) of the Government of Canada for the validation of cryptographic modules against FIPS 140 [13][14]. The Cryptographic Algorithm Validation Program (CAVP) provides validation testing of FIPS-approved and NIST-recommended cryptographic algorithms and their individual components. Cryptographic algorithm validation is a prerequisite of cryptographic module validation.

Cryptographic recommendations in this document are based on the time of publication of this document and may be superseded by other publications in the future. The References section contains a list of relevant FIPS, SPs, and Internet Engineering Task Force (IETF) standards related to IKE and IPsec.

Approved algorithms and their options for IKE and IPsec as of this writing are listed in Table 1.

Table 1: Approved Algorithms and Options

NIST’s current cryptographic timeline requirements are specified in SP 800-131A [18]. Federal agencies that want to provide IPsec VPN services in compliance with NIST’s recommendation or requirements must ensure that their systems will be upgraded to NIST-recommended or approved algorithms and key lengths before the current algorithms and their key lengths become disallowed. Federal agencies should require their IPsec VPN vendors to have the necessary upgrades available early enough for testing and deployment in the field.

The strongest possible cryptographic algorithms and key lengths that are NIST-approved should be used for authentication, encryption, and integrity protection unless they are incompatible with interoperability, performance, or export constraints.

In addition to providing specific recommendations related to configuring cryptography for IPsec, this guide presents a phased approach to IPsec planning and implementation that can help in achieving successful IPsec deployments. The five phases of the approach are as follows:

As part of implementing IPsec, organizations should also implement additional technical, operational, and management controls that support and complement IPsec implementations. Examples include establishing control over all entry and exit points for the protected networks, ensuring the security of all IPsec endpoints, and incorporating IPsec considerations into organizational policies.

This publication seeks to assist organizations in mitigating the risks associated with the transmission of sensitive information across networks by providing practical guidance for implementing security services based on Internet Protocol Security (IPsec). This document presents information that is independent of particular hardware platforms, operating systems, and applications other than providing real-world examples to illustrate particular concepts. Specifically, the document includes a discussion of the need for network layer security services, then focuses on how IPsec provides them and how organizations can implement IPsec. The document uses a case-based approach to show how IPsec can be used to provide security for different scenarios. It also describes alternatives to IPsec and discusses the circumstances under which each alternative may be appropriate.

The remainder of this document is organized into the following sections and appendices:

This section provides a general introduction to network layer security—protecting network communications at the layer that is responsible for routing packets across networks. It first introduces the Internet Protocol (IP) model and its layers, then discusses the need to use security controls at each layer to protect communications. It includes a brief introduction to IPsec, primarily focused on the types of protection IPsec can provide for communications. This section also briefly introduces VPN services, architectures, features, and common uses and explains what types of protection a VPN can provide.⁵

IP networking (sometimes called TCP/IP, although it encompasses more than just TCP, the Transmission Control Protocol) is the standard used throughout the world to provide network communications. IP communications are roughly composed of four layers that work together. When a user wants to transfer data across networks, the data is passed from the highest layer through intermediate layers to the lowest layer, with each layer adding additional information.⁶ The lowest layer sends the accumulated data through the physical network; the data is then passed up through the layers to its destination. Essentially, the data produced by a layer is encapsulated in a larger container by the layer below it. The four IP layers, from highest to lowest, are shown in Figure 1.

Figure 1: IP Model

Security controls exist for network communications at each layer of the IP model. As previously explained, data is passed from the highest to the lowest layer, with each layer adding more information. Therefore, a security control at a higher layer cannot provide full protection for lower layers because the lower layers perform functions of which the higher layers are not aware. The following items discuss the security controls that are available at each layer:

Designing a cryptographically sound application protocol is very difficult, and implementing it properly is even more challenging, so creating new application layer security controls is likely to create vulnerabilities. Also, some applications, particularly commercial off-the-shelf (COTS) software, may not be capable of providing such protection.

While application layer controls can protect application data, they cannot protect communication metadata, such as source and destination IP addresses, because this information exists at a lower layer. Whenever possible, application layer controls for protecting network communications should be standards-based solutions that have been in use for some time. One example is Secure/Multipurpose Internet Mail Extensions (S/MIME) [19], which is commonly used to encrypt email messages. Another example is the Secure Shell (SSH) [20] protocol, which encrypts remote login sessions.

The use of TLS or DTLS typically requires each application to support TLS or DTLS; however, unlike application layer controls (which typically involve extensive customization of the application), transport layer controls such as TLS and DTLS are less intrusive because they simply protect network communications and do not need to understand the application’s functions or characteristics. Although using TLS or DTLS may require modifying some applications, these protocols are well-tested and are a relatively low-risk option compared to adding protection at the application layer instead.

Alternatively, an application could use a TLS proxy instead of building native support for TLS or DTLS. The transport layer can only provide transport security, not data origin security. For example, a TLS-based connection between two email servers protects the transport from eavesdroppers but does not protect the message content transmitted within that TLS connection from manipulation by one of the two email servers. TLS and DTLS are sometimes deployed as a generic VPN solution protecting all IP traffic instead of only protecting a netflow. Such VPNs, commonly called Secure Sockets Layer (SSL) VPNs, work on the network layer but use an application at the transport layer.

Another advantage of network layer controls is that since IP information (e.g., IP addresses) is added at this layer, the controls can protect both the data within the packets and the IP information for each packet. However, network layer controls provide less control and flexibility for protecting specific applications than transport and application layer controls.

Compared to controls at the other layers, data link layer controls are relatively simple, which makes them easier to implement. They also support other network layer protocols besides IP. Because data link layer controls are specific to a particular physical link or local WiFi signal, they are poorly suited to protecting connections to remote endpoints, such as establishing a VPN over the internet.

An internet-based connection is typically composed of several physical links chained together; protecting such a connection with data link layer controls would involve many parties and different protocols for each part of the physical chain. It is easier to consider the internet as a whole to be untrustworthy and use controls at the network, transport, or application layer. Data link layer protocols have been used for many years primarily to provide additional protection for specific physical links that should not be trusted.

Because network layer security controls can provide protection for many applications at once without modifying them, these controls have been used frequently for securing communications, particularly over shared networks such as the internet. Network layer security controls provide a single solution for protecting all data from all applications, as well as protecting IP address, protocol, and port information. However, in many cases, controls at another layer are better suited to providing protection than network layer controls. For example, if only one or two applications need protection, a network layer control may be excessive. An application is often not aware of the (lack of) protection offered by the network or data link layer. Controls at each layer offer advantages and features that controls at other layers do not. Information on data link, transport, and application layer alternatives to network layer controls is provided in Section 8.

IPsec has emerged as the most commonly used network layer security control for protecting communications. IPsec is a framework of open standards for ensuring private communications over IP networks. The Internet Key Exchange (IKE) protocol is used to securely negotiate IPsec parameters and encryption keys. IKE is described in Section 3.

The IPsec Working Group at the Internet Engineering Task Force (IETF) is responsible for maintaining and publishing the standards for IKE and IPsec. Documents produced by IETF Working Groups are defined in two types: Request for Comments (RFCs), which are completed specifications, and Internet-Drafts, which are working documents that may become RFCs. IKEv2 is specified in [23]. The Encapsulating Security Payload (ESP), the core IPsec security protocol, is specified in [24]. Algorithm implementation and usage guidelines are specified in [25] for IKEv2 and [26] for IPsec. Various extensions to IKEv2 have their own RFC specifications. The IKE and IPsec protocols originated at the IETF almost three decades ago. Some of their history, such as the difference between IPsec-v2 and IPsec-v3, has been documented in the IPsec roadmap document [27].

Depending on how IPsec is implemented and configured, it can provide any combination of the following types of protection:

Policy-based IPsec is more secure than routing-based IPsec, as the security of the policy works independently from the security of the remote endpoint. Policy-based IPsec is not vulnerable to accidental or malicious routing table changes, and it prevents leaking packets to the local network, since local packets do not use the routing table. IPsec-based access control works independently from other access control mechanisms, such as firewall services or other mandatory access control mechanisms.

Normally, new keys are generated based on the generated shared secret of the original key exchange using a key derivation function (KDF). To guarantee that new key material has no relationship to the old key exchange, fresh session keys can, optionally, be generated by performing a new Diffie-Hellman (DH) key exchange instead of reusing the old key exchange’s generated shared secret to generate new session keys. This method of using a fresh key exchange provides perfect forward secrecy (PFS). When resources allow, PFS should be used.

The most common use of IPsec implementations is providing VPN services. A Virtual Private Network (VPN) is a virtual network built on top of existing physical networks that can provide a secure communications mechanism for data and IP information transmitted between networks or between different nodes on the same network. Because a VPN can be used over existing networks, such as the internet, it can facilitate the secure transfer of sensitive data across public networks. This is often less expensive than alternatives such as dedicated private telecommunication links between organizations or branch offices. Since dedicated private lines are often multi-tenant solutions themselves (such as those partitioned via Multiprotocol Label Switching [MPLS] [28] and run by third-party telecommunication companies), even they are now usually protected by an IPsec VPN. Remote access VPNs provide flexible solutions, such as securing communications between remote workers and the organization’s servers. A VPN can be established within a single network to protect particularly sensitive communications from other parties on the same network or even deploy a mesh of IPsec connections between all nodes in a single network so that no unencrypted data ever appears on the network. Section 2.4 discusses these different deployment models.

Below are further discussions of the cryptographic security services provided by IPsec for VPNs.

VPNs use symmetric cryptography to encrypt and decrypt their command and data channels. Symmetric cryptography is generally more efficient and requires less processing power than asymmetric cryptography, which is why symmetric encryption is typically used to encrypt the bulk of the data being sent over a VPN. National Institute of Standards and Technology (NIST)-approved algorithms that implement symmetric encryption include the Advanced Encryption Standard (AES) and the Triple Data Encryption Standard (3DES).⁷ One of the NIST-approved modes for symmetric encryption algorithms is AES-Galois Counter Mode (AES-GCM); see Table 1 for additional NIST-approved symmetric encryption modes.

Integrity is provided by a message authentication algorithm. The algorithm takes input data and a secret integrity key and produces a MAC. The data and MAC are sent across the network. The receiver calculates the MAC on the received data using the same secret integrity key (which has been previously established between the sender and receiver). If there is any change in the message and/or its MAC, a verification of the MAC will fail, and the message must be discarded. Common algorithms that implement integrity protection are:

180 [30] (i.e., Secure Hash Algorithm [SHA]: SHA-1 or the SHA-2 family of hash functions, also known as HMAC-SHA-2).⁸ The SHA-3 family might be added in the future.

VPNs typically use the DH key exchange algorithm to create a confidential communication channel to calculate a shared key between the two endpoints that an eavesdropper cannot obtain or compute. DH key exchanges can be based on finite field cryptography (“classic” or “modular” DH) or on elliptic curve cryptography (ECDH). After performing the DH key exchange and calculating the shared key, the endpoints still need to authenticate each other to ensure that the confidential communication channel is set up with the expected party and not somebody else.

A digital signature algorithm is used for peer authentication. It uses two separate keys: a public key and a private key. The private key is used to digitally sign data to be sent to a receiving party along with the signature, and the public key is used by the receiver to verify the digital signature on the received data. These keys are often referred to as public/private key pairs. When an individual’s private key is used to digitally sign data, only that same individual’s corresponding public key can be used to verify the digital signature. Common algorithms that are used to generate and verify digital signatures include RSA, the Digital Signature Algorithm (DSA)¹⁰, and the Elliptic Curve Digital Signature Algorithm (ECDSA). NIST-approved digital signature algorithms are specified in FIPS 186 [34][35].

VPNs usually use asymmetric cryptography for identity authentication. Such a public/private key pair can be used by itself or contained within an X.509 certificate. A VPN entity is authenticated by proving it has possession of the private key that corresponds to a known public key as well as the secret key computed by the parties during the DH key exchange. This binds the private communication channel (i.e., the VPN) to the expected identities. The public key can verify this proof without having a copy of the private key. Thus, as long as both parties each have the other’s public key and their own private key, they can establish an authenticated private channel through which they can communicate.

A less secure method of identity authentication is using a pre-shared key (PSK). Parties authenticate each other’s identity based on the fact that no one else has possession of this shared key, which must be established out-of-band.¹¹ A VPN entity’s identity is authenticated by proving that it has possession of the PSK as well as the secret key computed by the parties during the DH key exchange. This binds the private communication channel to the expected identities. The main disadvantage of VPNs using PSKs for authentication is that all parties that know the PSK can impersonate every other party in the group. PSKs are also vulnerable to online and offline dictionary attacks. That means that PSKs must be highly random (providing at least 112 bits of security strength) and must not be based on simple words or phrases. Otherwise, an attacker observing the key exchange can attempt to use an offline brute-force attack to find the PSK by calculating the authentication payload based on dictionary words and comparing the generated authentication payloads to the observed authentication payload. Unfortunately, experience has shown that administrators often use weak PSKs that are vulnerable to dictionary attacks.

VPNs do not remove all risk from networking, particularly for communications that occur over public networks. One potential problem is the strength of the implementation. For example, flaws in an encryption algorithm or the software implementing the algorithm could allow attackers to decrypt intercepted traffic, and random number generators that do not produce sufficiently random values as keys could provide additional attack possibilities. Another issue is encryption key disclosure; an attacker who discovers a symmetric key could decrypt previously recorded or current traffic. An attacker obtaining the private key of a public/private key pair (or PSK) used for identity authentication could potentially pose as a legitimate user.

Another area of risk involves availability. A common model for information assurance is based on the concepts of confidentiality, integrity, and availability. Although VPNs are designed to support confidentiality and integrity, they generally do not improve availability (i.e., the ability for authorized users to access systems as needed). In fact, many VPN implementations actually tend to decrease availability somewhat because they add more components, complexity, and services to the existing network infrastructure.

Risks are highly dependent upon the chosen VPN architecture and the details of the implementation. Section 2.4 describes the primary VPN architectures.

There are four primary architectures for IPsec-based VPNs:

IPsec-based VPNs are often used to provide secure network communications between two networks. This is typically done by deploying a VPN gateway onto each network and establishing a VPN connection between the two gateways. Traffic between the two networks that needs to be secured passes within the established VPN connection between the two VPN gateways. The VPN gateway may be a dedicated device that only performs VPN functions, or it may be part of another network device, such as a firewall or router. Figure 2 shows an example of an IPsec network architecture that uses the gateway-to-gateway model to provide a protected connection between the two networks.

Figure 2: Gateway-to-Gateway VPN Architecture Example

This model is relatively simple to understand. To facilitate VPN connections, one of the VPN gateways issues a request to the other gateway to establish an IPsec connection. The two VPN gateways exchange information with each other and create an IPsec connection. Routing on each network is configured so that as hosts on one network need to communicate with hosts on the other network, their network traffic is automatically routed through the IPsec connection, protecting it appropriately. A single IPsec connection establishing a cryptographically protected tunnel between the gateways can support all communications between the two networks, or multiple IPsec connections can each protect different types or classes of traffic. The gateways connect to each other using IPv4 or IPv6 protocols. When using tunnel mode, the IP address family of the outer ESP packets transmitted between the gateways does not need to be the same as the IP address family of the encrypted IP packets. For example, an IPsec connection between the hosts on IPv6 addresses 2001:db8:1:2::45 and 2001:db8:1:2::23 could be used to transport IPv4 traffic from 192.0.2.0/24 to 198.51.100.0/24. These types of IPsec connections are often called 6in4 or 4in6, respectively, to denote the inner and outer IP families.

Figure 2 illustrates a gateway-to-gateway VPN that does not provide full protection for data throughout its transit. In fact, the gateway-to-gateway architecture only protects data between the two gateways, as denoted by the solid line. The dashed lines indicate that communications between VPN clients and their local gateway and between the remote gateway and destination hosts (e.g., servers) are not protected by the gateway-to-gateway architecture. The other VPN models provide protection for more of the transit path. The gateway-to-gateway architecture is most often used when connecting two secured networks, such as linking a branch office to headquarters over the internet. The gateway-to-gateway architecture is the easiest to implement in terms of user and host management. Gateway-to-gateway VPNs are typically transparent to users; the use of a gateway-to-gateway VPN connection is not noticeable to them. Also, the users’ systems and the target hosts (e.g., servers) do not need to have any VPN client software installed, nor should they require any reconfiguration, to be able to use the VPN.

If the gateway-to-gateway VPN connects two different organizations, it is possible that some special DNS configuration is required if machines in one network need to be able to reach machines in the other network by DNS name. If machines are found by their IP address, no special DNS handling is required.

An increasingly common VPN architecture is the remote access architecture. The organization deploys a VPN gateway onto its network; each remote access user then establishes a VPN connection between their remote device (host) and the VPN gateway. As with the gateway-to-gateway architecture, the VPN gateway may be a dedicated device or part of another network device. Figure 3 shows an example of an IPsec remote access architecture that provides a protected connection for the remote user.

Figure 3: Remote Access VPN Architecture Example

In this model, IPsec connections are created as needed for each individual mobile device that has been configured to act as an IPsec client with the organization’s IPsec gateway. When a remote user wishes to use computing resources through the VPN, the user’s host initiates communications with the VPN gateway. The user is typically asked by the VPN gateway to authenticate their identity before the connection can be established. The VPN gateway can perform the authentication itself or consult a dedicated authentication server. The client (the remote device in Figure 3) and gateway exchange information, and the IPsec connection is established. The user can now use the organization’s computing resources, and the network traffic between the user’s host (the remote device in Figure 3) and the VPN gateway will be protected by the IPsec connection.

Some organizations do not want to receive all of the internet traffic generated by a remote host. For example, if a host is browsing the internet, that traffic could be prohibited from going through the VPN connection. Only traffic for the organization itself will be sent over the VPN connection; a separate internet connection would be required. This is called a split-tunnel VPN. Other organizations do not trust the remote hosts to directly communicate with the internet over a separate internet connection while also being connected via a VPN connection to the organizational computer resources since that internet connection could be used to attack or infiltrate the VPN connection. If an organization normally has a strict firewall preventing unauthorized access of the organization’s computer resources by the hosts in the local network, it would not want a remote host to bypass this security when it is connecting from a remote location. In that case, a remote host will send all its traffic via the VPN connection to the VPN gateway; this allows IPsec protection to be applied to all of the traffic. Traffic received and decrypted by the VPN gateway that is not meant for the local organization can be sent to the organization’s firewall for inspection and then sent onwards through the organization’s internet connection. Similarly, response traffic from the internet flows back via the organization’s firewall to the VPN gateway and is then sent via the VPN connection to the remote host.

As shown in Figure 3, the remote access VPN does not provide full protection for data throughout transit. The dashed lines indicate that communications between the gateway and the destination hosts (e.g., servers, laptops, desktops) on the right side of the figure are not protected. The remote access VPN architecture is most often used when connecting hosts on unsecured networks to resources on secured networks, such as linking traveling employees around the world to headquarters over the internet. The remote access VPN is somewhat complex to implement and maintain in terms of user and host management: the VPN gateway (or a designated device) must manage the credentials of all of the remote devices (hosts) and their authorized users, and all of these might frequently change. Remote access VPNs are typically not transparent to users because they must authenticate before using the VPN. Also, the user’s remote device needs to have a VPN connection configured. Some devices do not allow more than one VPN connection to be active at a time.

Remote access users can find themselves on networks that, intentionally or not, cause VPN connections to fail. Some unintentional failures can be worked around by always having the latest software and IPsec VPN features supported.¹² Standard IKE runs over the UDP protocol, and ESP can also use UDP. Some networks block all UDP packets, causing IKE and ESP-over-UDP traffic to be dropped. As a method of last resort, IPsec communication can be tunneled over TCP, which is a more universally accepted protocol. For added insurance, TLS can be used in conjunction with TCP to work around network failures with native IPsec packets.

Modern devices often have more than one network interface, and the user can switch between network interfaces automatically. For instance, when a mobile device loses a WiFi connection, it can automatically fall back to a mobile network (Long-Term Evolution [LTE]/5th Generation [5G]) provider. IPsec provides mobility support to ensure that the VPN connection continues working without interruption when switching between such networks.

The host-to-host VPN architecture is used for a variety of reasons. For security reasons, some hosts may only accept connections protected by a VPN. This makes it more secure against unauthenticated access attempts. For example, if the web server software on the host is vulnerable to a specific attack, it is only exposed to those who also have VPN credentials to contact the host. Another common issue is the presence of attackers performing port scans or dictionary attacks against the login method (e.g., SSH). With a VPN, these ports are not accessible to attackers.

In this case, the organization configures the server to provide VPN services and the system administrators’ machines (or some users’ machine) to act as VPN clients. The system administrators use the VPN client when needed to establish protected connections to the remote server. Figure 4 shows an example of an IPsec network architecture that uses the host-to-host architecture to provide a protected connection to a server for an administrator (or a user) that is remote to that server. The purpose of a host-to-host VPN connection is to protect the traffic from one end of the connection to the other end.

Figure 4: Host-to-Host VPN Architecture Example

In this model, IPsec connections are created as needed for each individual remote VPN administrator or user. The remote hosts have been configured to act as IPsec clients with another remote host that is a server. When a user or administrator wishes to use or modify resources on the server, that user or administrator’s host initiates IPsec communications with the server. The server acts as an IPsec server that requests the person authenticate before the connection can be established. The host and the server exchange information, and if the authentication is successful, the IPsec connection is established. The person can now access the server, and the network traffic between the host and the server will be protected by the IPsec connection.

As shown in Figure 4, the host-to-host VPN (indicated by the solid line) provides protection for data throughout its transit. This can be a problem because network-based firewalls, intrusion detection systems, and other network devices cannot be deployed to inspect the traffic in transit, which effectively circumvents certain layers of security.¹³ The host-to-host VPN is most often used when a small number of trusted users or administrators need to access a system from remote locations using insecure protocols (e.g., a legacy system) as long as the system can be updated to provide VPN services.

Host-to-host VPNs can be resource-intensive to implement and maintain in terms of configuration management. Host-to-host VPNs are not transparent to users because they must authenticate the user before using the VPN. Also, all end user systems and servers that will participate in VPNs need to have VPN software installed and/or configured. However, the host-to-host architecture can be deployed in a more automated way that requires no end user interaction to establish a VPN.

A special case of host-to-host VPNs is a large-scale host-to-host IPsec deployment. This is typically used when one wants to encrypt all connections within a network, cloud, or datacenter. Whenever one host in such a network wishes to communicate with another host in the network, it first establishes an IPsec connection. This is also called mesh encryption. Usually, these IPsec connections are packet triggered. An application in one host sends a packet to another host remote to it. The kernel of the host on which the application runs receives the packet from the application and determines that it does not have an IPsec connection to that remote host, so it triggers the setup of an IPsec connection. Once the IPsec connection is established, the packet is encrypted and sent to the remote host. This way, no unencrypted packet is ever sent over the network. Hosts authenticate each other using X.509 certificates or Domain Name System Security Extensions (DNSSEC)¹⁴. These types of authentication are based on a shared trust anchor: an X.509 certificate authority (CA) or a DNSSEC zone key. This allows hosts to be added to a network without the need to reconfigure all other hosts to learn about the newly deployed host.

One advantage of this type of IPsec architecture is that every host is responsible for its own protection; no large expensive IPsec gateways are required, which also means there is no single point of failure added to the network architecture. Hosts in a network can be configured to insist on IPsec or to attempt IPsec but allow cleartext communication if that fails. This architecture can be combined with the gateway-to-gateway architecture, where hosts within one network can initiate IPsec to hosts in another network, extending the network mesh encryption to both networks. The two networks are connected by a gateway-to-gateway architecture so that the internet can still be used to connect these two networks at the cost of packets being encrypted twice—once by the host-to-host IPsec deployment and once by the gateway-to-gateway IPsec deployment.

Software Defined Networking (SDN) is an architecture of dynamic cloud networking. An SDN network (sometimes called a Software Defined Wide Area Network, or SDWAN) is a network with a security controller and various nodes (hosts). The security controller enforces the security policies of its nodes. All of the nodes are configured by the security controller, usually via the Network Configuration Protocol (NETCONF) [36]. For communication between two nodes within a network or within two different networks, each node consults its local security controller. If the nodes have enough resources to set up IPsec, the security controller(s) can relay the authentication and connection parameters to its node(s), and the two nodes can then negotiate the IPsec VPN connection.

Figure 5: SDWAN Architecture Example

This is shown in Figure 5 for communication between hosts (i.e., nodes) A1 and B1 (at the top of the figure). Host A1 contacts its security controller SC1. SC1 and SC2 (host B1’s security controller) negotiate the IKE and IPsec parameters and convey them to their respective hosts (A1 or B1, as appropriate). Host A1 can now initiate an IKE session with B1, establishing an IPsec connection between A1 and B1. The IPsec secret key material is only known by the A1 and B1 hosts and not by the security controller. The hosts could optionally transfer these secret keys to their security controller to facilitate monitoring via decryption by the security controller or another dedicated monitoring device that takes its configuration from the security controller.

When establishing a connection between two hosts in two different networks, if the hosts do not have enough resources to negotiate an IPsec connection, the security controller in each network can negotiate an IPsec connection with the security controller in the other network on behalf of its host and give the keying material and security policies for the IPsec connection to that host. The two hosts receive the exact same IPsec policies and the same encryption keys from their security controllers to install in their IPsec subsystems (key exchange is performed by the two corresponding security controllers). This latter method is called an IKEless IPsec connection. It is not the preferred method since, in this case, the security controllers are aware of all of the secret keys used by their hosts, and the security controllers (or whoever manages to get control of one of them) can decrypt all of the host-to-host IPsec-protected traffic or masquerade as one of the hosts under its control. An IKEless connection can also be established by a single security controller between two of its own managed nodes.

A third method for configuring hosts by a security controller is for the hosts to give their key-exchange public keys to the security controller. When two hosts establish an IPsec connection, the security controller distributes each host’s key-exchange public key and a nonce to the other host. Each of the hosts uses the public key and nonce from the other host along with its own private key to generate a secret shared key, which is then used for an IPsec connection. The security controller does not know the private keys or the shared key of the IPsec devices. Therefore, the security controllers cannot decrypt any host-to-host communication and cannot masquerade as one of the hosts.¹⁵ The same method can be used for two nodes behind different security controllers. In that case, the security controllers distribute the nodes’ public keys to each other and relay these to the nodes they manage.

The hardest part of deploying an IPsec implementation is the authentication mechanisms, which depend on the prior deployment of a CA or other identity verifier mechanism. If a network only needs to protect itself from passive attackers (i.e., attackers that can eavesdrop but not send their own malicious packets), then anonymous IPsec can be used. Therefore, anonymous IPsec connections are typically host-to-host connections and not gateway-based connections¹⁶ because an IPsec gateway typically requires authentication of the connecting gateway or host and authenticates itself to that entity. A variant of this is server-only authenticated IPsec. This works similarly to regular HTTPS connections where a client connects to the server and the server has to authenticate itself to the client, but the client remains anonymous. Any client authentication then happens at the application layer and not at the network layer.

The advantage of anonymous IPsec is that it can be deployed quickly. Once in place and protecting against passive attackers, the configuration can be slowly migrated to an authenticated IPsec capability that also protects against active attacks.

Due to its security risk, anonymous IPsec VPNs are discouraged by NIST.

Section 2 describes the IP model and its layers—application, transport, network, and data link— and explains how security controls at each layer provide different types of protection for IP communications. IPsec, a network layer security control, can provide several types of protection for data, depending on its configuration. The section describes VPNs and highlights the VPN architectures. IPsec is a framework of open standards for ensuring private communications over IP networks and is the standard used for network layer security control. It can provide several types of protection, including maintaining confidentiality and integrity, preventing packet replay attacks and traffic analysis, and enforcing access restrictions.

When two hosts want to set up an IPsec connection with each other, the parameters of the IPsec connection need to be negotiated (e.g., the source and destination IP addresses that are allowed, the encryption algorithms to use, and the cryptographic key material to use for the encryption and decryption of packets). The hosts also need to authenticate each other. All of this is done using the Internet Key Exchange (IKE) protocol. The version of the IKE protocol described in this section is IKE version 2 (IKEv2) and is specified in RFC 7296 [23].¹⁷ The differences between IKEv1 and IKEv2 are described at the end of this section.

Typically, IKE runs as a privileged process, while IPsec usually runs as part of the operating system kernel. The IKE process is responsible for configuring the kernel for IPsec. The kernel is responsible for the actual packet encryption and decryption operations. The IKE process can insert a policy into the kernel that will instruct the kernel to warn the IKE process when an unencrypted packet matching certain source and destination IP addresses and/or other criteria is about to be transmitted. If the peers can mutually authenticate each other and agree on other policy details, then the IKE process can negotiate an IPsec tunnel that covers this packet. This is used for creating IPsec tunnels on demand.

The IKE protocol can be considered the command channel. The IPsec protocol is the data channel; it encrypts and decrypts the IP packets and verifies that the source and destination IP address conform to the negotiated policies. The IKE protocol command channel itself also needs to be encrypted to ensure the privacy of the parameters of the IPsec connection. In other words, first the IKE encrypted connection is established, and then one or more IPsec connections are established through the protected IKE command channel.¹⁸ The IKE protocol is used to establish an IKE Security Association (IKE SA) [23].¹⁹ An IPsec connection is called an IPsec SA or Child SA.²⁰ Both IKEv2 SAs and IPsec SAs are identified by Security Parameters Index (SPI) numbers; for IKEv1, other fields are used as the SA identifier until the IPsec SPIs are established.

The IKE protocol consists of UDP messages on port 500 and 4500. As shown in Figure 6, each IKE packet consists of a fixed IKE header (the first five lines of the figure) followed by the variable-length IKE data.

Figure 6: The IKEv2 Packet Format

The initiator of an IKE exchange generates a four-byte Initiator SPI. The responder generates the four-byte Responder SPI. In the first IKE packet sent by the Initiator, the Responder SPI is 0x00000000. The SPI numbers are used to uniquely identify an established IKE SA. Each endpoint selects the corresponding IKE encryption/decryption key for an encrypted IKE message from its endpoint to the other endpoint based on the SPI numbers.

An IKE session consists of IKE packet exchanges. Each exchange consists of a single request packet and a single reply packet. If there is any packet loss, it is the initiator’s responsibility to retransmit its request.²¹ Each exchange packet has a message ID, which starts at zero and is incremented for each message exchange. The message ID allows for detecting retransmitted packets and handling out-of-order IKE packets. There is a distinct message ID for messages started at each IKE peer (endpoint).

The IKEv2 protocol uses two exchanges to establish an IKE SA and an associated IPsec SA. The IKE SA is then used to send and receive further configuration and management commands. The first exchange is called IKE_SA_INIT, and the second exchange is called IKE_AUTH. Together, these two exchanges are referred to as the initial exchanges. Once these two exchanges are completed, both the initiator and the responder have established the IKE SA and one IPsec SA. Once the IKE SA is established, other additional exchange types are used to establish additional IPsec SAs, rekey the existing IKE SA or the IPsec SAs, make configuration changes, perform a liveness detection of peers, and terminate IKE or IPsec SAs.

The following sections describe the IKE exchanges in detail and explain how they work together to establish IPsec connections.

The exchange type for additional IPsec SA messages is called CREATE_CHILD_SA. Another common exchange type is the INFORMATIONAL exchange, which is used for notification messages such as IPsec SA deletions, rekeying, liveness (dead peer detection), and mobility updates. Each exchange can relay additional information about supported features or algorithms using Notify payloads.

The IKE_SA_INIT exchange sends the cryptographic IKE proposals for setting up the encrypted IKE SA. Each proposal consists of a list of components needed to establish an IKE SA. These components are called transforms. For IKEv2, four types of transforms are required: encryption (AEAD algorithms or encryption algorithms), integrity (none for AEAD²², or a MAC otherwise), (Elliptic Curve) Diffie-Hellman—(EC)DH, and pseudorandom function (PRF). The IKE_SA_INIT exchange also includes data that will be used to generate a shared secret that is used to derive symmetric keys to protect later traffic between the two peers, such as the sender’s (EC)DH public value (carried in the Key Exchange [KE] payload), a random nonce (in the nonce payload), and both IPsec SPIs (in the IKE Header). The initiator can propose multiple alternative transform combinations, and the responder selects the preferred proposal and returns a single proposal with those transforms, its own KE and nonce payloads, and a responder SPI.

The initiator needs to determine the cryptographic policy that can be accepted by the responder. The initiator sends a list of transforms that represents its policy.²³ For the initiator’s (EC)DH Key Exchange algorithm, it will include the corresponding KE payload (e.g., an EC public key) of its preferred Key Exchange algorithm. If the responder does not allow this (EC)DH algorithm in any of its transforms, the responder will reply with an INVALID_KE notification that indicates the responder’s preferred (EC)DH algorithm based on the list that the initiator sent. The initiator can use this response to create a new IKE_SA_INIT packet with a proper KE payload that is acceptable to both initiator and responder policies.

Since an (EC)DH computation is central processing unit (CPU) intensive, a malicious entity could send many spoofed IKE_SA_INIT messages, causing the responder to perform multiple (EC)DH key exchange calculations to try and complete the (EC)DH calculations and resulting in a denial of service attack. When a responder deems it is under attack, it may respond to an IKE_SA_INIT message with a special COOKIE payload instead of the regular payloads. The initiator that receives a response containing a COOKIE must resend its IKE_SA_INIT packet but now include that COOKIE payload. This proves to the responder that the initiator can receive packets—that is, the original packet received by the responder was not simply a spoofed packet by an attacker. The responder typically encodes some information in the COOKIE—such as the initiator’s IP address, a timestamp, and a random value—so that the COOKIE cannot be reused by an attacker redistributing it to its attacking nodes that are spoofing packets. Now the responder is assured that the initiator is an actual participant in the IKE exchange and not simply sending malicious packets using a forged (spoofed) IP address. While this could still be an attacker, at least now the responder can defend itself against this abuse by, for example, rate-limiting the number of IKE negotiation attempts allowed by that IP address.

The IKE_SA_INIT exchange is also used to detect the presence of network address translation (NAT) devices. If a NAT device is detected, the IKE negotiation will move to port 4500, and the IPsec connection will be configured to use UDP or TCP encapsulation to avoid problems with the NAT device rewriting the IP address of the IPsec packets. Often, NAT routers also drop all IP protocols except UDP and TCP, so by encapsulating the IPsec (ESP) packets into UDP or TCP, the packets will not be dropped by the NAT router. The endpoint behind the NAT device will also send one-byte KEEPALIVE packets, typically at 20-second intervals, to ensure that the NAT device will keep open the port mapping that is used by the endpoint behind NAT. This is especially important with deployments of Carrier Grade NAT (CGN) that are typically deployed on mobile data networks (LTE/5G). The KEEPALIVE packets serve no purpose beyond passing the NAT device and are discarded by any endpoint IPsec stack that receives them.

After the IKE_SA_INIT exchange has completed, both endpoints have performed the (EC)DH key exchange and have generated the secret value called the SKEYSEED. All encryption and authentication keys will be derived from this value using the negotiated PRF transform.²⁴ From here on, all further packets are encrypted. However, both the initiator and the responder still need to authenticate each other’s identity.

The peers still need to verify each other’s identities and prove that the initial unencrypted IKE SA messages were not modified in transit. The IKE_AUTH exchange contains the payloads needed for the responder to authenticate the initiator and its previous IKE_SA_INIT exchange. The IKE_AUTH exchange also contains payloads to negotiate the first IPsec SA, such as the proposals and transforms to negotiate the cryptographic parameters, the source/destination packet policies for the IPsec SA in the form of traffic selectors for the initiator (TSi) and responder (TSr), and other options such as the mode of the IPsec SA and Configuration Payload requests for obtaining an IP address and a DNS nameserver IP address.

Since authentication can involve X.509 certificates and intermediary CA certificates, this packet can end up being larger than the network MTU. To work around networks that do not handle IP fragmentation properly, the IKE protocol itself supports fragmentation to prevent fragmentation at the network layer. Typically, only the IKE_AUTH packets trigger IKE fragmentation.

Typical authentication methods are X.509 certificates, raw (unsigned) public keys (e.g., RSA or ECDSA public keys, usually formatted in the SubjectPublicKeyInfo [SPKI] format), or PSKs. IKE supports the Extensible Authentication Protocol (EAP). If EAP authentication is required, more than one IKE_AUTH exchange might be required to complete the authentication. The authentication method can be different between the two endpoints, although they often use the same method. One example of using different authentication methods by each party is a remote access VPN where the server is authenticated using its X.509 certificate, but clients are authenticated via EAP-TLS.²⁵

Once the IKE_SA_INIT and IKE_AUTH exchanges have successfully completed, the two hosts have set up an IKE SA and an IPsec SA. Any further communication will be sent using the encrypted and authenticated IKE SA.

The IKE_AUTH exchange negotiates the IPsec SA network parameters, such as the source and destination IP address, the address family, the source and destination ports, and the protocol, using traffic selectors. A traffic selector consists of:

Additional traffic selector components are possible, too, such as Network Label or Security Context.

Traffic selectors are negotiated in sets of two. A set of two traffic selectors denotes the policy for the source and destination traffic—one for the inbound traffic and one for the outbound traffic of the IPsec SA. The IKE_AUTH request contains at least the TSi and TSr. The TSi describes the sending and receiving address of the initiator, and the TSr describes the sending and receiving address of the responder.

IKEv2 allows the concept of narrowing, where the responder picks a subset of the TSi/TSr that the initiator requested. This facilitates setting up a number of smaller-range IPsec SAs instead of one large network-to-network IPsec SA. This can enhance parallel processing. It is also used for the initiator obtaining an IP address from the responder where the initiator requests every address on the internet (by requesting 0.0.0.0/0) and is narrowed down by the responder to one IP address (e.g., 192.0.2.1/32).

An additional traffic selector pair can be included that contains the actual source, destination, and protocol values from the packet that triggered the IKE session at the initiator. This assists the responder in narrowing traffic selectors to a range that includes the traffic that the initiator wants to send to the responder.

Optionally, during IKE_AUTH, the hosts can also exchange Configuration Payloads (CPs). The initiator can request a number of configuration options, and the responder can respond with appropriate values. The main CPs are:

All these CPs enable the remote access VPN client to find and use resources on the remote network. By obtaining an IP address on that remote network, other hosts on that network can potentially reach the remote VPN clients as if they were present locally. CPs are not used and are ignored on gateway-to-gateway and host-to-host IPsec deployments.

CPs are the successor to the IKEv1 non-standard Extended Authentication (XAUTH) and Mode Configuration (ModeCFG) payloads.

The CREATE_CHILD_SA exchange is used for three separate tasks:

Creating an additional IPsec SA uses similar IPsec payloads as those used to create the initial IPsec SA in the IKE_AUTH exchange. Either endpoint can initiate a CREATE_CHILD_SA exchange. Lifetimes for IKE and IPsec SAs are not negotiated. Each peer is responsible for rekeying the relevant SAs before the lifetime of their local policy is exceeded.

Rekeying is the process of creating fresh cryptographic keys for an IKE SA or IPsec SA. IKE and IPsec keys are ephemeral and only stored in volatile memory for the duration of the lifetime of the SA. Once an SA is rekeyed, the old cryptographic keys are wiped from memory. In the event that one of the IPsec hosts is compromised, only the current session keys are still in memory, and previously recorded encrypted SA traffic cannot be decrypted. IKE SA and IPsec SA session keys typically have a lifetime of one to eight hours. A rekey request can be for one of the IPsec SAs or for the IKE SA. A new IPsec SA is negotiated and installed as a new IPsec SA without yet removing the old IPsec SA. The outbound IPsec SA is used immediately. Once traffic is received on the new inbound IPsec SA, the old IPsec SA is completely deleted. This ensures that rekeying does not lead to any traffic flow interruptions or leaking of unencrypted packets.

Once an IKE SA rekey is complete, the associated IPsec SAs of the old IKE SA are transferred to the new IKE SA. The old IKE SA is then deleted.

The purpose of the IKE INFORMATIONAL exchange is to provide the endpoints with a way to send each other status and error messages. Some commonly used informational messages are:

Either endpoint can initiate an informational exchange. The other endpoint is obliged to return an answer to prevent the initiator (of the informational exchange) from retransmitting. A delete message denotes that the SPI of the IPsec SAs or the IKE SA is to be deleted. Deleting the IKE SA will also cause all of its IPsec SAs to be terminated and deleted.

An endpoint that has not received any IPsec traffic in a while might want to verify that the remote endpoint is still alive. To do so, it can send an informational exchange message (i.e., a probe message) containing zero payloads.²⁷ An endpoint receiving such an informational message must respond with an empty informational message. If these probes are not answered for a configured time period, the IKE SA and IPsec SA are terminated.

A mobile device that is switching its connection (e.g., from LTE/5G to WiFi) needs to send an informational message with a notification to its remote endpoint. The remote endpoint uses both the content of the informational message, as well as the IP addresses observed from the IKE packet itself, as an indication of which IP address to use as the updated IP address for the mobile endpoint. Successful decryption of the packet (with properly incremented Message ID to prevent replays) verifies the new IP address to use. This process is called Mobile IKE (MOBIKE) and is specified in [38].

Different deployments require different authentication methods. Usually, hosts authenticate each other using the same authentication method. But sometimes a client host authenticates a server host differently from the method used by the server to authenticate the client.

This method, also called machine certificate authentication, is most often used for deploying IPsec within an organization when it involves a large number of devices. The organization can set up a new internal X.509 certificate deployment or reuse an existing X.509 certificate-based solution. Setting up a new host does not require any changes to the already deployed hosts. Certificate revocation lists (CRLs) and the Online Certificate Status Protocol (OCSP) can be used to revoke a particular certificate. Remote access VPN clients are often authenticated using X.509 certificates. Cloud (mesh) encryption also often uses certificate-based authentication.

A host that requires the other end to authenticate itself using certificates can send a CERTREQ payload (during IKE_SA_INIT or IKE_AUTH). Both parties then exchange their certificates in CERT payloads during the IKE_AUTH exchange. Intermediate CAs can also be sent as part of the CERT payload.²⁸

Since certificate-based authentication requires certificates generated by CAs that may not be trusted by the organizations verifying the certificates, this method is not always a usable solution to connect two different organizations since one (or both) of the organizations would need to trust an external CA party not under their own control. For U.S. Government organizations, the Common Policy CA can be used as a mutually trusted CA.

EAP is a framework for adding arbitrary authentication methods in a standardized way to any protocol. It uses a model of a client, a server, and a backend authentication, authorization, and accounting (AAA) server. The client initiates an EAP authentication to the server. The server forwards these messages to and from the AAA server. The AAA server will let the server and client know that the client and server have successfully authenticated each other. AAA protocols with EAP support include Remote Authentication Dial-In User Service (RADIUS) [39] and Diameter [40].

The most common EAP method used with IKEv2 is EAP-TLS, although EAP-Microsoft Challenge Handshake Authentication Protocol version 2 (EAP-MSCHAPv2) is used as well. EAP-TLS uses certificates issued to users instead of certificates issued to hosts. Some devices, such as mobile phones, often do not make such a distinction. However, laptops generally have non-privileged users that cannot modify the operating system’s machine certificate store. These users cannot install a machine certificate but can install a certificate for themselves for use with EAP-TLS.

Usually, clients use EAP to authenticate themselves to the server, but the server is authenticated by the clients using regular certificate-based authentication.

Authentication using the raw public key of the other entity in a communication (there are no certificates which bind the public key with the other entity’s identity) is mostly used for IoT devices or when authentication of the public keys is done via publication in DNSSEC.²⁹ IoT devices often do not have the memory, storage, or CPU capacity to perform X.509 certificate validation. These devices often have a hard-coded public key in their firmware that authenticates its IPsec peers.

When public keys are stored in DNS, and the DNS is secured against tampering or spoofing using DNSSEC, there is no more need to use X.509 certificates. Certificates provide trust via the entity that signs the certificate, but in this case, the public key stored in the DNS itself is already signed by DNSSEC. The trust anchor is not a CA but rather a DNSSEC trust key responsible for that part of the DNS hierarchy. Instead of certificates stating the validity period of the public key, raw public keys in DNS are valid as long as these public keys are still published in the DNS. DNSSEC prevents the replay of old DNS data by adding signature lifetimes to DNS records. This type of deployment is most commonly used within a single administrative network, similar to machine-based certificate authentication.

PSK-based authentication is often deployed because it is the easiest to configure. Each end of the communication has the identity of the other end and their PSK. It does not require generating public keys or certificates or running an EAP infrastructure. It is most commonly used for gateway-to-gateway deployments as it does not involve adding a third-party trust anchor to the VPN gateway device.

Some deployments use a PSK for authentication that is shared between the VPN server and all of the remote access VPN clients. Once the PSK has been obtained by an attacker, it can be used to impersonate any client or the remote access VPN server itself. Even if the clients are using one-time passwords (OTPs), a man-in-the-middle attacker can obtain an OTP and log in as the remote user to the real remote access VPN. Therefore, group PSKs are strongly discouraged.

PSKs are often derived from dictionary words and are less than 32 characters long. Such insecure deployments are vulnerable to offline dictionary attacks.³⁰ PSKs must have a high entropy value. A good PSK is pseudorandomly created and has at least 128 bits of entropy.

NULL authentication is a special kind of authentication. It really means that no authentication is required. There are two common use cases for this.

The first use case is to deploy IPsec to a large number of nodes (i.e., hosts) where the goal is to only protect against passive attacks. It does not protect against attackers that can perform a man-in-the-middle attack. An advantage is that no authentication system (e.g., certificates, EAP, or DNSSEC) needs to be deployed. For small-scale deployments, this method should never be used, and strong PSKs should be used instead. Sometimes a NULL authentication deployment is gradually upgraded to an authenticated deployment.

The second use case only uses NULL authentication for the initiator client. The responder (i.e., the server) still authenticates itself to the initiator using another authentication method, such as a machine certificate. This creates a situation that is similar to HTTPS-based websites: the client remains anonymous, but the server is authenticated. This is the method used for internet-based opportunistic IPsec, where two IPsec hosts attempt to establish an IPsec connection without a preexisting configuration or knowledge of each other. This usually involves authentication based on DNSSEC or a widely acknowledged CA such as Let’s Encrypt.³¹ The advantage of this type of deployment is that only the servers need to have an identity for authentication. The clients (usually laptops and phones) do not need to have any kind of identity and can remain anonymous—at least at the network layer. Similar to HTTPS, the application layer might require the client to authenticate before it is allowed to access a particular resource.

NIST does not approve the use of NULL authentication-based IPsec. Any deployment of NULL authenticated IPsec shall be categorized as being identical to plaintext unprotected network traffic.

During the IKE_SA_INIT exchange, both endpoints exchange information about what they believe their IP address is.³² The other end of the communication will confirm whether the information matches the source address of the packet it received. If the endpoints detect that a NAT is present, they will move further IKE communication from port 500 to port 4500. Changing the UDP port was originally done to prevent bad interactions with NAT devices that tried to support “IPsec passthrough.” This feature caused more harm than good, and by moving to a new port, the IPsec passthrough modifications performed by NAT devices were avoided.

Currently, no NAT devices perform IPsec passthrough. Once an IPsec SA has been negotiated, the hosts will also enable UDP or TCP encapsulation of ESP packets to facilitate traversing the NAT over a single port. This avoids two problems. The first problem is that NAT devices commonly only support UDP and TCP, meaning that IPsec (ESP) packets would not be dropped by some NAT devices. The second problem is that the NAT device needs to keep a port mapping between the internal device’s ports that are used and how these ports are mapped onto the NAT device’s public-facing ports. It is easiest if a device behind the NAT device only needs one port mapping for IKE and IPsec (ESP) traffic. The host behind NAT will also send one-byte keepalive packets to ensure that the NAT device does not expire its NAT port mapping if the VPN does not produce any traffic for some time. Otherwise, if the remote IPsec host starts sending traffic towards the NAT device, the NAT device would no longer remember which internal device to forward that traffic to, and the IPsec connection would no longer function.

Some cloud providers issue an ephemeral or semi-static public IP address to some virtual machines inside their cloud. The virtual machines are deployed with only an internal [41] IP address. The cloud infrastructure uses NAT to translate the public IP address to the virtual machine’s private IP address. This NAT will also trigger the NAT traversal mechanism of IKE. This poses another problem. If the IPsec tunnel is configured with the public IP address as the tunnel endpoint, the virtual machine cannot create packets with its public IP address as the source address, since this public IP address is not configured on the machine itself. Packets received after decryption are dropped because the operating system is not looking for packets with the public IP address. A common workaround is for such virtual machines to configure the public IP address on one of their network interfaces.

IKE packets can be larger than the common ethernet MTU of 1500 bytes. If these packets are sent over the network, they will most likely be fragmented. Too often, those fragments will be dropped by a firewall, and the host will fail to receive the fragments for reassembly. This problem is avoided by using IKE fragmentation, which fragments the packets at the application layer instead of the network layer.

IKEv2 fragmentation is specified in RFC 7383 [42]. The main difference with the IKEv1 vendor-specific implementations of fragmentation is that IKEv2 fragments are encrypted. This makes it harder for an attacker to interfere. Note that while the fragments are encrypted, the fragments are not (yet) authenticated because the IKE exchange has not yet completed. Once all fragments have been received, the original IKE packet can be reconstructed and processed as if it was received in one packet.

IKEv2 fragmentation is supported for every exchange type except IKE_SA_INIT. Typically, only the IKE_AUTH exchange requires fragmentation, since that exchange includes the X.509 certificates. These certificates can be larger than 1500 bytes, especially when using RSA public keys that use a key size of 2048 bits or more.

Devices such as mobile phones and laptops often have multiple network interfaces, which allows those devices to switch to cheaper and/or faster networks when available. Phones may use the local WiFi network at the office or at home and mobile networks (5G/LTE) at other locations. Switching also occurs when an existing network connection suddenly degrades. Switching networks changes the source IP address used by the device. VPN traffic is still sent to the old, no longer used IP address until the device establishes a new IPsec connection.

MOBIKE [38] addresses this issue. It assumes that an internal IP address is assigned by the VPN on the device using CPs. This internal IP address will remain with this device regardless of the outer IP address used by the device. Once a device switches between its network interfaces, it will send an INFORMATIONAL exchange packet with an UPDATE_SA_ADDRESS notification. This packet will be sent using the new IP address. The VPN server will be able to recognize the IPsec SA based on the SPI numbers, despite the fact that it is suddenly coming from a different IP address. Once decrypted and authenticated, the VPN server will notice the UPDATE_SA_ADDRESS payload and change the endpoint IP address (and port if encapsulation is used due to NAT). It will reply with a confirmation message. At this point, all IPsec SA traffic is sent and received using the client’s new IP address. Since the VPN client’s applications are only using the internal IP address assigned by the VPN server for communication to the remote access network, and this IP address does not change when the device itself changes its network interface and outer IP address, all existing connections remain intact. The applications are not even aware that the network interfaces have switched.

A device that wakes up from battery saving mode will generally send a MOBIKE update whether or not its IP address changed. This ensures that any NAT state updates that have happened since the device went to sleep are reported back to the VPN server. For example, the NAT device might have terminated the unused NAT port mapping between the device and the VPN server. The MOBIKE packet will create a new fresh NAT port mapping entry, and the VPN server will immediately be able to update the client’s IP address and port number and activate the updated VPN connection.

MOBIKE allows for more complicated setups with multiple IP addresses. While MOBIKE can be used as a failover mechanism for the gateway-to-gateway architecture, care should be taken with such a deployment. If one of the endpoints is compromised, its state could be copied onto a (rogue) machine on the other side of the world, and a MOBIKE update message could be sent to redirect all traffic to the rogue location. The most secure option is to disable MOBIKE unless the IPsec configuration is for a remote access VPN client.

It is unclear when a quantum computer will become available. Sufficiently large quantum computers will be able to break the finite field (classic) DH and ECDH key exchanges within the timeframe in which it would be expected that IPsec traffic should remain confidential. That is, the key exchange could be broken in weeks or months, while the expectation of confidentiality would be in the timeframe of decades. Adversaries could store today’s encrypted communications for later decryption using quantum computers. This problem is not unique to IKE. Other encryption protocols, such as TLS, suffer from the same problem. It is expected that in the near future, quantum-resistant algorithms will be standardized and deployed for IKE, TLS, and other protocols. Until then, some deployments of IKE and IPsec might use post-quantum pre-shared keys (PPKs) to strengthen the current algorithms against potential future attacks using quantum computers.

With the exception of IKEv1 using a very strong PSKs, all IKEv1 and IKEv2 configurations are vulnerable to quantum computers. IKEv2 supports PPKs [43] as a countermeasure. For the purpose of defending against quantum computers, the PPK works similarly to the PSK in IKEv1 in that the PPK is mixed into the key derivation process in addition to the DH values. The PPK must be a cryptographically strong random key and is exchanged out of band. PPKs are identified by a static or ephemeral PPK Identity. This can be used to protect the identity of the connecting clients and facilitate the use of OTPs as the source of the PPK.

IKEv2 allows the gradual migration of a network from not using PPK to using PPK. First, some hosts are configured with PPK, and when two hosts both support PPK and have each other’s PPK ID with corresponding PPK, the hosts will use the PPK as an additional input to create the KEYMAT and SKEYSEED that are used as input to the PRFs that generate the keying material for the IKE and IPsec SAs. Once all hosts support PPK, their configurations can be updated to mandate PPK.

While this protects the IPsec SAs (since their key material derivation depends on the PPK), the initial IKE SA DH process is not protected by the PPK and can still be broken by a quantum computer. This will lead to a loss of privacy of the IKE identities and other information exchanged during the initial IKE Exchange, such as the traffic selectors used for the first IPsec SA. This can be prevented if the IKE implementation allows for setting up a childless IKE SA (without IPsec) and then immediately rekeying the IKE SA. This rekeyed IKE SA is protected by the PPK, and IPsec SAs can then be set up using this new IKE SA without exposing any information to adversaries with quantum computers.

PPKs shall have at least 128 bits of entropy.

The IKE Redirect [44] notify payload allows an IPsec server to send a redirection request to connecting or connected VPN clients. This can be used to reduce the load of overloaded IPsec servers or to take a server out of use (for instance, to update its operating system). Redirected clients must use the same credentials they were originally using before being redirected. A redirection message includes an IP address or DNS name of the forwarding VPN that the VPN client will need to initiate a connection with.

Redirected messages sent in IKE_AUTH are only processed after both ends have authenticated each other. This allows a server to only send specific clients to another server (e.g., all clients of a certain customer in a multi-tenant deployment or some individual power users generating a lot of traffic). The (overloaded) servers are still required to perform full IKE exchanges to all connecting clients and then redirect them to different server hosts.

Redirected messages sent in IKE_SA_INIT are not authenticated. Clients that accept such redirected messages should take the necessary precautions to prevent denial-of-service attacks. The advantage for the host performing the redirection is that it can redirect clients without performing a full IKE exchange.³³ The disadvantage is that redirections in IKE_SA_INIT cannot select the specific clients for redirection by their IDs since the client ID has not yet been transmitted to the server.

Redirected messages can be used to provide a redundant set of servers for the gateway-to-gateway deployment. A failing server can redirect clients to the other (backup) server. In such an architecture, it is recommended that redirected messages be limited for each endpoint based on preconfigured IP addresses.

The IKEv2 protocol builds on the lessons learned with IKEv1. IKEv2 is simpler, faster, and more secure. IKEv2 has some important new features, such as mobility support (MOBIKE), support for newer cryptographic algorithms, anti-distributed denial-of-service (DDoS) support, and server redirection support. It is recommended that existing IKEv1 installations be upgraded to IKEv2.

For those familiar with IKEv1, the main differences between IKEv1 and IKEv2 are:

While it is possible to hard-code the IPsec information using out-of-band communication (called manual keying), this shall not be used. The IKE protocol handles a number of other security properties, none of which are enforced when using manual keying. Encryption keys would never be refreshed when a fixed key is manually input and used, so any compromise would allow an attacker to decrypt all previously monitored traffic under the fixed key. Some values, such as nonces, counters, and initialization vectors (IVs), shall never be used more than once. Otherwise, the encryption may become vulnerable (weaken).

The only time that manual keying might be acceptable is if another trusted entity, such as a security controller in the SDWAN paradigm, assumes these responsibilities. Another example is the 3GPP protocol, which negotiates the IPsec parameters between a cell tower and handset using a non-IKE protocol.

Administrators sometimes mistakenly believe that manual keying is easier to set up than automated keying via IKE. However, manual keying is much harder to set up than IKE.

Manual keying is typically only used for software testing and IPsec benchmark tests.

This recommendation discourages the use of manual keying.

IPsec is a collection of protocols that assist in protecting communications over networks.³⁵ This section focuses on the primary component of IPsec—ESP, which protects the confidentiality and integrity of data packets. The section also briefly covers the other IPsec components: the IP Payload Compression Protocol (IPComp) and the Authentication Header (AH) protocol. All of the parameters and cryptographic keys needed by the IPsec protocols are negotiated using the IKE protocol as described in Section 3.

ESP is the core IPsec security protocol. It has largely been unchanged since its second version, which was published as RFC 2406 in 1998 [46]. The current version (IPsec-v3) was specified in RFC 4303 in 2005 [24]. It contains only a few updates to the IPsec-v2 specification in RFC 2406. Since all of the changes to ESP are either backwards compatible or are new features that would need to be negotiated via IKE before they are enabled for ESP, there are no compatibility issues between IPsec implementations receiving and sending ESP packets. Regardless, practically all current implementations support IPsec-v3. Features only available in IPsec-v3 are:

The use of padding and the capability of sending dummy messages increase TFC by making it harder for an eavesdropper who cannot decrypt the packets to deduce anything from the encrypted packet sizes or timings.

ESP provides encryption and integrity protection. The outer header is not fully protected, allowing for routers that forward ESP packets to still modify certain flags, such as the Quality of Service (QoS) and Time to Live (TTL) values.

ESP’s encryption functionality can be disabled through the selection of the Null ESP encryption algorithm or the AES-GMAC AEAD algorithm. AES-GMAC is a variant of the AES-GCM algorithm, which provides integrity protection without encryption. ESP can be used to provide either encryption and integrity protection or only integrity protection. AH deployments should be migrated to these ESP algorithms. ESP supports AEAD and classic (non-AEAD) encryption with integrity methods.

ESP has two modes: transport and tunnel. In tunnel mode (see Figure 7), a new packet is constructed containing the (original) IP packet being sent through the tunnel by: 1) placing an ESP header and trailer around the original IP header and its payload; 2) encrypting the original header, payload, and ESP trailer; 3) computing an integrity check value (ICV) over the ESP header and the encrypted data; 4) placing the ICV at the end of the packet being constructed; and 5) adding a new IP header to the beginning of the packet. The ICV computation does not include the new IP header.

The new IP header lists the endpoints of the ESP tunnel (such as two IPsec gateways) as the source and destination of the packet and contains the entire, now-encrypted, original packet as its payload. Because of this, tunnel mode can be used with all VPN architectures described in Section 2.4. As shown in Figure 7, tunnel mode can encrypt and protect the integrity of both the data and the original IP header for each packet. Encrypting the original IP header and its payload protects their confidentiality; encrypting the original IP header conceals the nature of the communications, such as the actual source or destination of the packet, protocol, and ports used that would indicate which application is likely being used. The ICV is used to detect any changes to the data over which the ICV is computed.

Figure 7: ESP Tunnel Mode Packet

ESP tunnel mode is used for gateway-to-gateway deployments, remote access VPNs, and various network virtualization deployments. It is also required when the IPsec connection needs to traverse a NAT, which rewrites the outer IP address.

ESP transport mode is often used for host-to-host deployments within data centers, local networks, and virtual machines where no NAT is deployed. In transport mode (see Figure 8), ESP uses the original IP header instead of creating a new one. The ESP payload and trailer are encrypted, and an ICV is computed over the ESP header and the encrypted data. Integrity protection is not provided for the IP header. The overhead of the transport mode is less than that of the tunnel mode because it does not have to create an entire new IP header.

Transport mode is incompatible with NAT. For example, in each TCP packet, the TCP checksum is calculated on both the TCP and IP fields, including the source and destination addresses in the IP header. If NAT is being used, one or both of the IP addresses are altered, so NAT needs to recalculate the TCP checksum. If ESP is encrypting packets, the TCP header is encrypted; NAT cannot recalculate the checksum, so NAT fails. This is not an issue in tunnel mode; because the entire TCP packet is hidden, NAT will not attempt to recalculate the TCP checksum of the inner encrypted packet, only of the outer IP address which is not part of the ESP encryption. However, tunnel mode and NAT have other potential compatibility issues.³⁶ Section 7.2.1 provides guidance on overcoming NAT-related issues.

Figure 8: ESP Transport Mode Packet

ESP uses symmetric cryptography to provide encryption for IPsec packets. Accordingly, both endpoints of an IPsec connection protected by ESP encryption must use the same key to encrypt and decrypt the packets. When an endpoint encrypts data, it divides the data into small blocks (for the AES algorithm, blocks of 128 bits each) and then performs multiple sets of cryptographic operations (known as rounds) using the data blocks and key. Encryption algorithms that work in this way are known as block cipher algorithms. When the other endpoint receives the encrypted data, it performs decryption using the same key and a similar process but with the steps reversed and the cryptographic operations altered.

After encryption has been performed, the first step for providing integrity protection is to create a MAC on a message using a MAC algorithm and a secret key shared by the two endpoints. The MAC is added to the packet, and the packet is sent to the recipient. The recipient can then regenerate the MAC using the shared key and confirm that the two MACs match, thus determining whether the data has been modified. IPsec mostly uses an HMAC algorithm [47] for integrity protection, which uses approved hash functions. Examples of HMAC are HMAC-SHA256. Another common non-HMAC integrity algorithm is AES-CMAC-96 [48]. Federal agencies are required to use NIST-approved algorithms and FIPS-validated cryptographic modules. HMAC with a hash function from the SHA-2 family and AES-CMAC-96 are NIST-approved. However, AES-eXtended Cipher Block Chaining (AES-XCBC-96), for example, is not an approved algorithm.

Encryption with separate integrity protection (as described in Section 4.1.2) requires two separate cryptographic processes over the data using two different secret keys. AEAD combines these two processes, which significantly increases performance. It also provides more constant-time processing when errors occur, resulting in a more robust error handling process that is less susceptible to timing attacks. The reverse process produces either the plaintext data or an error indication. For IKEv2 and ESP, AES-GCM is specified in [49] as an AEAD algorithm. Due to the way that IKEv1 handles the separation of encryption from data integrity protection in IKE packets, AEAD algorithms cannot be used in IKEv1. IKEv1 can, however, still negotiate AEAD algorithms for ESP.

The nonce used by an AEAD algorithm must be unique for every encryption operation with the same secret key but does not need to be unpredictable.³⁷ The nonce in IKE is built using an implicit part (the salt) and an explicit part (the IV). The implicit part is based on the keying material calculated from the DH key exchange and negotiated PRF, similar to how secret encryption keys are generated. This value is never transmitted and binds the encryption to the DH channel. The explicit part is transmitted and usually based on an increasing, and thus unique, counter. Reuse of the IV with the same secret key compromises the security of the data. Thus, these algorithms shall be used in conjunction with IKE and cannot be used with static or manual keys. An SA shall be terminated before the counter reaches its maximum possible value.

Examples of common algorithms used by ESP are AES-GCM [50] and AES-CBC [51] with HMAC-SHA-2. Most algorithms have limitations on the amount of data that can be safely encrypted with a single key and requirements for auxiliary parameters.

Triple DES has been deprecated since 2019 and will be disallowed after 2023. It is much slower than AES-GCM and AES-CBC. Triple DES is also cryptographically weaker and requires more frequent rekeying to avoid birthday attacks due to its smaller block size of 64 bits. The HMAC-MD5 and HMAC-SHA-1 integrity algorithms are also no longer NIST-approved.

For the latest cryptographic recommendations, see NIST SP 800-131A [18] and FIPS 140-3 [14].

ESP adds a header and a trailer around each packet’s payload. As shown in Figure 9, each ESP header is composed of two fields:

The next part of the packet is the payload. It is composed of the encrypted payload data and the IV, which is not encrypted. This is helpful in deterring traffic analysis. The IV is used during encryption. Its value is different in every packet, so if two packets have the same content, the inclusion of the IV will cause the encryption of the two packets to have different results. This makes ESP less susceptible to cryptanalysis.

To obfuscate the length and frequency of information sent over IPsec, the protocol allows for sending dummy data called traffic flow confidentiality (TFC) padding. TFC padding can be added to the unencrypted data before encryption, or it can be injected as a whole new packet with padding only being encrypted to a certain size between real encrypted data transmissions. An observer cannot tell if TFC is enabled and, more importantly, can no longer make any reasonable assumptions based on packet size or frequency. One common deployment of TFC is to pad all packets to the maximum MTU value, resulting in all sent ESP packets having the exact same length. This would increase the amount of encrypted data sent, so on links where transmission costs depend on the amount of data sent (e.g., LTE/5G), there is a cost associated with using TFC.

The third part of the packet is the ESP trailer, which contains at least two fields and may optionally include one more:

the payload is usually a transport layer protocol, often TCP (protocol number 6) or UDP (protocol number 17). Every ESP trailer contains a Next Header value.

Figure 9: ESP Packet Fields

Reviewing and analyzing actual ESP packets can provide a better understanding of how ESP works. Figure 10 shows the bytes that compose an actual ESP packet and their American Standard Code for Information Interchange (ASCII) representations. The ESP packet only contains four sections (ignoring the link layer): IP header, ESP header, encrypted data (payload and ESP trailer), and (optionally) authentication data information. It is not possible to determine if this packet was generated in transport mode or tunnel mode by examining the encrypted data. However, because the IP header is unencrypted, the IP protocol field in the header does reveal which IPsec protocol the payload uses (in this case, ESP). As shown in Figure 7 and Figure 8, the unencrypted fields in both modes (tunnel and transport) are the same.

Figure 10: ESP Packet Capture Using Wireshark, Showing Sequence Number 1

Although it is difficult to tell from Figure 10, the ESP header fields are not encrypted. Figure 11 shows a network traffic capture made with the tcpdump tool of encrypted traffic generated by the ping command, followed by an IKE session which is followed by another ping that is now protected by ESP. Each direction uses its own negotiated SPI value for its packets, which corresponds to an ESP connection being composed of two one-way connections, each with its own SPI. Both hosts initially set the sequence number to 1, and both incremented the number to 2 for their second packets. The tcpdump tool labels IKE packets as “isakmp”—a legacy name from the IKEv1 protocol.

13:45:34.118804 IP 203.0.113.1 > 198.51.100.1: ICMP echo request, id 27083, seq 2, length 64 13:45:34.118850 IP 198.51.100.1 > 203.0.113.1: ICMP echo reply, id 27083, seq 2, length 64 13:45:39.469941 IP 203.0.113.1.isakmp > 198.51.100.1.isakmp: isakmp: parent_sa ikev2_init[I] 13:45:39.472043 IP 198.51.100.1.isakmp > 203.0.113.1.isakmp: isakmp: parent_sa ikev2_init[R] 13:45:39.481690 IP 203.0.113.1.isakmp > 198.51.100.1.isakmp: isakmp: child_sa ikev2_auth[I] 13:45:39.525826 IP 198.51.100.1.isakmp > 203.0.113.1.isakmp: isakmp: child_sa ikev2_auth[R] 13:45:39.587728 IP 203.0.113.1 > 198.51.100.1: ESP(spi=0xc55ed62b,seq=0x1), length 120 13:45:39.587773 IP 198.51.100.1 > 203.0.113.1: ESP(spi=0xf6fc7c09,seq=0x1), length 120 13:45:40.646761 IP 203.0.113.1 > 198.51.100.1: ESP(spi=0xc55ed62b,seq=0x2), length 120 13:45:40.646800 IP 198.51.100.1 > 203.0.113.1: ESP(spi=0xf6fc7c09,seq=0x2), length 120

Figure 11: tcpdump Capture of ping, IKE, and ESP Packets

ESP packets cannot traverse a NAT device in all circumstances. If an IPsec connection uses transport mode, changing the IP address on the packets will invalidate the integrity checks imposed by IPsec. The NAT device cannot rewrite the ICV because it does not have access to the keying material needed to do so. For all intents and purposes, the NAT device is a malicious actor that IPsec protects against.

The ESP protocol has no ports. If multiple clients send ESP from behind the same NAT router, it would be difficult to track the ESP packets to the respective clients as they would all have the same destination IP—that of the NAT device. While SPI numbers are uniquely generated for each IPsec host, there is no guarantee that two hosts behind the same NAT will not ultimately pick the same SPI number for an IPsec SA. Furthermore, NAT routers often do not understand or translate anything other than the UDP and TCP protocols, causing ESP packets to be dropped by the NAT device.

To overcome the transport issues of the ESP protocol, ESP can be encapsulated in UDP (ESPinUDP). The NAT device can rewrite the IP address of the outer UDP packet and track multiple clients by the UDP port number. For historical reasons,³⁹ when IKE detects a NAT during the negotiation, it switches the IKE negotiation from UDP port 500 to UDP port 4500. It uses a regular UDP packet header followed by a four-byte header with all zeroes (Non-ESP Marker) following the UDP header. Then the IKE header follows.

ESPinUDP also uses port 4500 to ensure that the NAT device only has one NAT mapping for all traffic (ESP and IKE). Following the regular UDP packet header, the ESP header follows. The first four bytes of the ESP header is the SPI number, which cannot be 0. Thus, an implementation receiving a packet on port 4500 can determine whether the packet is an ESPinUDP packet or an

IKE packet, depending on whether or not it sees the SPI number of the non-ESP marker. Usually, the kernel receiving an ESPinUDP packet will simply strip the UDP header away without bothering with the UDP checksum (which not all NAT routers properly recalculate) and process the remaining ESP data as if it was received as an ESP packet without encapsulation. If the kernel detects an IKE packet, it will send this packet to the IKE process for processing by the IKE daemon.

Starting with IKEv2, even if no NAT was detected, endpoints need to support receiving ESP and ESPinUDP packets on all of their IPsec SAs. Each endpoint may decide when to use encapsulation and when not to. IKEv2 also allows for initiating a new IKE_SA_INIT on UDP port 4500, bypassing UDP port 500 completely.

Implementations supporting TCP encapsulation [53], where ESP packets are wrapped into a TCP stream, can also choose to use TCP. This provides a much-needed method to prevent IPsec from being easily filtered and blocked. A lack of TCP encapsulation was one of the reasons why SSL VPNs came into existence, as these could not be easily blocked by blocking the IPsec protocols (UDP port 500 and 4500 and protocol ESP). TCP encapsulation ports cannot be negotiated, as this would require that the negotiations start on the well-known port susceptible to blocking. Therefore, the TCP port has to be preconfigured manually or via the IPsec client provisioning system.

The ESP in TCP encapsulation uses an ASCII prefix tag of “IKETCP” so that an additional layer can be used, such as TLS. In that case, encrypted packets are encapsulated using a TCP connection that uses TLS. The packet processor can read the prefix and detect the start of an IKE/ESP stream, in which case it can send this traffic to the proper handler. Since restrictive networks often still (have to) allow access to HTTPS websites, using TLS on port 443 to protect (or really, hide) the TCP stream containing the encapsulated ESP packets will yield the best results. However, networks are often only misconfigured to drop all UDP traffic. Moving to ESP encapsulation on TCP port 4500 without TLS framing will usually be enough to be able to establish IPsec connections.

Implementations are encouraged to regularly try to go back to UDP encapsulation. TCP encapsulation means there are possibly two TCP layers involved in a packet: the TCP connection being encrypted and the TCP connection carrying the ESP packet. These two TCP layers will both independently determine retransmissions. Especially when there is packet loss, these two TCP streams will badly interfere with each other.

ESP can be deployed with IPComp. Before a packet is encrypted, the packet will be considered for compression. If the packet is very small already, such as an ICMP message, no compression is done, and the packet is encrypted as is; otherwise, the packet is compressed. However, various compression algorithms do not guarantee that an attempted compression does not end up being larger than the original. If this turns out to be the case, the original packet is encrypted without compression. If the compressed result is smaller, the compressed packet is encrypted. On the receiving end, the packet is decrypted, and if it was compressed, it will be decompressed.

However, applications that send large amounts of data usually already compress their data. At that point, attempting to compress already compressed data will not yield smaller packets, and a host will ultimately waste CPU cycles at the IPsec layer attempting futile compression. As such, IPsec level compression has not seen widespread use. This might change in the near future with the emergence of IoT devices and other battery-powered devices that use mobile data (LTE/5G). These devices save battery power by transmitting fewer bytes, even if that reduction requires more CPU power for compression. For non-IoT-based IPsec, NIST does not recommend using IPComp due to the risk of weakening security and the introduction of complexity to the encryption function. It is also not universally implemented since its status was always optional.

As with ESP, AH can be used in tunnel mode and transport mode. It offers only integrity algorithms and provides no confidentiality. The ESP protocol can use null encryption (ESP algorithm number 12) with an integrity algorithm, such as HMAC-SHA-2, to accomplish the same as AH. Alternatively, ESP can use an AEAD algorithm, such as AES-GMAC (ESP algorithm number 21), to offer integrity without confidentiality to replace AH.

NIST discourages the use of AH in this publication. The IETF has specified that AH is an optional IPsec protocol, which means it is not mandatory to implement and might not be available with all IPsec implementations. It is recommended that null encryption with the ESP protocol be used instead of the AH protocol when encryption is not desired.

Some implementations support the legacy IPsec-v2 ESP without authentication in combination with AH. This is usually referred to as AH+ESP. This combined mode (ESP for encryption and AH for integrity) is no longer recommended [25] since it provides no advantage over regular ESP with authentication. Regular ESP with authentication also reduces the effective MTU compared to AH+ESP, due to the additional overhead of an AH header plus an ESP header versus just an ESP header with authentication.

This section has described the IPsec protocols ESP, IPComp, and AH. The following summarizes the key points from the section:

This section describes the interactions between the IKE and IPsec subsystems. The interaction depends on the implementation. This section focuses on the standard protocols used to communicate between IKE and IPsec. Although some devices have their own proprietary method of communication, in general, the concepts explained in this section will apply to those proprietary implementations as well.

The IKE protocol is usually implemented as an application running on the operating system, whereas the IPsec protocol is generally implemented in the kernel of the operating system. Some devices implement the IPsec subsystem as a program running on the operating system kernel, but for the remainder of this section, it is assumed that IPsec is implemented in the operating system kernel.

The communication between IKE and IPsec is usually implemented using the PF_KEYv2 [54] or NETLINK [55] protocol. Linux uses NETLINK with the XFRM application programming interface (API), whereas BSD-based systems use PF_KEYv2.40

This section puts IKE and IPsec components together to illustrate how IPsec sessions are set up and executed. Each example includes the use of IKE to establish SAs.

Each IPsec SA has a state containing information such as the SPI numbers and the encryption keys and algorithms used as well as a policy containing the source and destination addresses and ports used for matching traffic that is covered by the IPsec SA for encryption/decryption. While each state must have a policy, not all policies need to have a state. For example, on-demand IPsec connections have a policy that allows the kernel to detect that an outgoing packet should trigger an IKE negotiation. Once the IKE SA has been established and an IPsec SA has been negotiated, the IKE daemon will install an IPsec state with corresponding policies. During the negotiation, the kernel can drop the packet, cache the packet for later transmission, or send the packet as is without encrypting it. Usually, UDP packets are dropped, since their unreliable nature requires that applications sending these packets need to know when to transmit their packets anyway. TCP packets are usually cached because TCP retransmissions are usually very slow, and it would make the on-demand tunnel very slow if the first TCP packet was always lost. Leaking packets in cleartext only occurs when the network considers the IPsec protection optional instead of mandatory.

Once an IPsec SA has been established between two hosts, all traffic that falls within the IPsec SA policy must be IPsec-protected. If, for some reason, unencrypted traffic is received, it is assumed to have been forged, and the traffic will be dropped.

The kernel maintains a state for each IPsec SA. An IPsec connection between two hosts consists of a pair of IPsec SAs—one for inbound and one for outbound traffic. These IPsec states are contained in the SAD. Figure 12 shows an example of an IPsec SA using an AEAD algorithm.

src 198.51.100.1 dst 203.0.113.1 proto esp spi 0xba293cd3(3123264723) reqid 1(0x01) mode tunnel replay-window 32 seq 0x00000000 flag af-unspec (0x00100000) aead rfc4106(gcm(aes)) 0x2ee20e32be3017c1878b9ae514081ba1d[…] 128 anti-replay context: seq 0x148a3, oseq 0x0, bitmap 0xffffffff lifetime config: limit: soft (INF)(bytes), hard (INF)(bytes) limit: soft (INF)(packets), hard (INF)(packets) expire add: soft 0(sec), hard 0(sec) expire use: soft 0(sec), hard 0(sec) lifetime current: 102600783(bytes), 84090(packets) add 2019-01-06 21:57:45 use 2019-01-06 21:57:50 stats: replay-window 0 replay 0 failed 0

src 203.0.113.1 dst 198.51.100.1 proto esp spi 0x6273ec0a(1651764234) reqid 1(0x01) mode tunnel replay-window 32 seq 0x00000000 flag af-unspec (0x00100000) aead rfc4106(gcm(aes)) 0x0afaf19501d6d94174bb3036b84d59d78e[…] 128 anti-replay context: seq 0x0, oseq 0x7829, bitmap 0x00000000 lifetime config: limit: soft (INF)(bytes), hard (INF)(bytes) limit: soft (INF)(packets), hard (INF)(packets) expire add: soft 0(sec), hard 0(sec) expire use: soft 0(sec), hard 0(sec) lifetime current: 2422796(bytes), 30761(packets) add 2019-01-06 21:57:45 use 2019-01-06 21:57:50 stats: replay-window 0 replay 0 failed 0

Figure 12: Example of an ESP IPsec SA (Inbound and Outbound) Using an AEAD Algorithm on Linux

If a non-AEAD algorithm is used, such as AES-CBC with HMAC-SHA-1, the SA will contain the encryption and integrity keys separately. Figure 13 illustrates this. Note that this example uses FreeBSD, which calls the AES algorithm by its original candidate name, Rijndael.

2001:db8:1:2::23 2001:db8:1:2::45 esp mode=tunnel spi=1675186937(0x63d952f9) reqid=1(0x00000001) E: rijndael-cbc 1dd058ed 63905223 147979df 1865bfb3 A: hmac-sha1 fde84c78 b2c90386 600927e3 1eb3dcf8 3163d053 seq=0x00000000 replay=0 flags=0x00000000 state=mature created: Feb 2 17:29:42 2019 current: Feb 2 17:37:19 2019 diff: 457(s) hard: 3600(s) soft: 2960(s) last: hard: 0(s) soft: 0(s) current: 0(bytes) hard: 0(bytes) soft: 0(bytes) allocated: 0 hard: 0 soft: 0 sadb_seq=1 pid=1404 refcnt=1

2001:db8:1:2::45 2001:db8:1:2::23 esp mode=tunnel spi=3301523791(0xc4c9414f) reqid=1(0x00000001) E: rijndael-cbc d32b7287 8e0ef003 3a2bac01 4b14d0c7 A: hmac-sha1 1a3b1fc7 091e76f5 860456f2 5342ceaa bc33a3d3 seq=0x00000000 replay=4 flags=0x00000000 state=mature created: Feb 2 17:29:42 2019 current: Feb 2 17:37:19 2019 diff: 457(s) hard: 3600(s) soft: 2611(s) last: hard: 0(s) soft: 0(s) current: 0(bytes) hard: 0(bytes) soft: 0(bytes) allocated: 0 hard: 0 soft: 0 sadb_seq=0 pid=1404 refcnt=1

Figure 13: Example of an ESP IPsec SA Using a Non-AEAD Algorithm on FreeBSD

The IPsec SA state information consists of:

The maximum counters and lifetimes have a soft and hard value. When the soft value is reached, the kernel will notify the IKE daemon so that it can take preventative action. When the hard value is reached, the IPsec SA is deleted by the kernel, and the IKE daemon is notified. Each time a packet is encrypted or decrypted, this state is updated appropriately.

The kernel maintains a list of IPsec policies in the SPD. The policy describes the nature of the traffic that matches a policy rule and links it to the state used to encrypt or decrypt the packet. Policies without states are used for on-demand IPsec connections. Figure 14 shows examples of two policies corresponding to the SAs in Figure 12.

src 192.168.13.6/32 dst 0.0.0.0/0

dir out priority 1040383 ptype main tmpl src 198.51.100.1 dst 203.0.113.1 proto esp reqid 1 mode tunnel

src 0.0.0.0/0 dst 192.168.13.6/32

dir in priority 1040383 ptype main tmpl src 203.0.113.1 dst 198.51.100.1 proto esp reqid 1 mode tunnel

Figure 14: Examples of Policies Corresponding to Figure 12 on Linux

The IPsec security policy information consists of:

Using the SPD and SAD, packets are processed for encryption and decryption, and all of the security policies are applied. If a policy violation is detected, the packet is dropped (e.g., when an encrypted packet is decrypted into a packet with a source address that is not allowed by the security policy of the SA).⁴³ A policy can also point to a non-IPsec SA target. Commonly implemented targets are PASS (never encrypt with IPsec), DROP, REJECT (DROP and send an ICMP message), and HOLD (cache the packet until an IPsec SA has been established).

Looking at the SAD and SPD entries of the previous figures, it can be seen that the host with IP address 198.51.100.1 is allowed to send ESP packets to the host with IP 203.0.113.1. The encrypted IP packet included can only have the source IP address 192.168.13.6 but can have any destination IP address. AES-GCM is used as the AEAD encryption algorithm. In other words, there is a VPN client running on 198.51.100.1 that started a VPN connection to the VPN server on 203.0.113.1 and received the internal IP address 192.168.13.6.

The IP address family of the IPsec host does not need to match the IP address family of the included encrypted IP packets. Figure 15 shows policies for two IPsec gateways using IPv6 addresses that are used to connect two IPv4 subnets with each other.

src 192.0.0.0/24 dst 192.0.2.0/24

dir out priority 1042407 ptype main tmpl src 2001:db8:1:2::45 dst 2001:db8:1:2::23 proto esp reqid 16389 mode tunnel

src 192.0.2.0/24 dst 192.0.0.0/24

dir in priority 1042407 ptype main tmpl src 2001:db8:1:2::23 dst 2001:db8:1:2::45 proto esp reqid 16389 mode tunnel

Figure 15: Example of IPsec Policies for a Gateway Architecture Connecting IPv4 Subnets using IPv6 on Linux

The output of the commands to inspect the current SAD and SPD differs by vendor. Figure 16 shows the SAD and SPD entries for an IPv6 in IPv4 IPsec connection in tunnel mode using the ipsecctl command on OpenBSD.

FLOWS: flow esp in from 2001:db8:0:1::/64 to 2001:db8:0:2::/64 peer 203.0.113.1 srcid FQDN/east dstid FQDN/west type use flow esp out from 2001:db8:0:2::/64 to 2001:db8:0:1::/64 peer 203.0.113.1 srcid FQDN/east dstid FQDN/west type require

SAD: esp tunnel from 198.51.100.1 to 203.0.113.1 spi 0x03f86d3a auth hmac-sha2-256 enc aes-256 esp tunnel from 203.0.113.1 to 198.51.100.1 spi 0x4df47d50 auth hmac-sha2-256 enc aes-256

Figure 16: Example of IPsec States and Policies Connecting IPv6 Subnets using IPv4 on OpenBSD (line breaks added)

Regardless of the implementation, the following types of messages are sent between the IKE and IPsec subsystems:

In this example, the goal is to establish an IPsec connection that provides encryption and integrity protection services between endpoints A and B. The IPsec architecture is gateway-to-gateway; endpoint A uses gateway A on network A, and endpoint B uses gateway B on network B. If an IKE SA is not already in place, a packet will trigger the establishment of an IKE SA. In IKEv2, this is accompanied by the establishment of an IPsec SA as well:

If endpoint B wishes to reply to the packet, steps 6 to 9 of this process are repeated, except the parties are switched. Endpoint B would send a packet to endpoint A; routing would direct it to gateway B. Gateway B would modify the packet appropriately and send it to gateway A. Gateway A would process and validate the packet to restore the original IP address, then send the packet to endpoint A.

Assuming that the IPsec connection between the gateways is sustained, eventually the IKE or IPsec SAs will approach one of the SA lifetime thresholds (maximum time or maximum bytes transmitted) as determined by the local policy on the respective gateways. The gateway with the shortest lifetime first determines whether the maximum SA lifetime is approaching and initiates the rekeying process using the existing IKE SA. If the IPsec SA is being rekeyed, both ends install the new inbound and outbound IPsec SA before removing the old inbound and outbound IPsec SA. Once valid encrypted traffic is received on the new inbound IPsec SA, the old inbound IPsec SA will be deleted. This ensures that there is no interruption of the traffic flow during IPsec SA rekeying. If the IKE SA is being rekeyed, both ends replace the IKE SA, and all IPsec SAs belonging to the old IKE SA are attached to the new IKE SA.

IPsec VPN products vary in functionality, including protocol and algorithm support. They also vary in breadth, depth, and completeness of features and security services. Management features, such as status reporting, logging, and auditing, should provide adequate capabilities for the organization to effectively operate and manage the IPsec VPN and extract detailed usage information. In the case of mesh encryption, too much logging can also be a concern. Traditionally, the management of IPsec products from different vendors has been problematic. Some recommendations and considerations include the following:

which should provide a non-proprietary management interface that can be used across all vendors.

This section provides information on troubleshooting IPsec VPNs.

Some IKE and IPsec interactions need careful attention to prevent the two subsystems from interfering with each other. Usually, these are handled by the IKE implementation. If an IPsec implementation insisted that all communication between two hosts be encrypted with IPsec, those two hosts would never be able to send non-IPsec packets, including IKE packets. Without allowing IKE packets, no IPsec SA can be negotiated and installed, and the two hosts would never be able to communicate. Similarly, if one host crashes and restarts, it needs to be able to send IKE packets that are not IPsec-encrypted, yet the remote endpoint still has a policy that only allows encrypted traffic to be received.

To work around this, IPsec implements a policy exception for UDP port 500 and 4500 packets and will skip processing these via the regular SPD processing. If the kernel does not override IKE packets for IPsec processing, the IKE daemon needs to have a policy specifically for the IKE ports used with the highest preference—higher than the IPsec SA processing policy preference. If TCP is used, UDP ports 500 and 4500 also need to have such a policy exception. Practically all IKE daemons perform this task on startup.

A more subtle requirement is the need to exclude IPv6 neighbor discovery. If two hosts in the same subnet have established an IPsec SA over IPv6, and one of these hosts crashes and reboots, that host will send an unencrypted neighbor host discovery ICMP packet in an attempt to find the other host on the local network. If the host that did not crash drops the unencrypted ICMP packet, the two hosts will not be able to set up a new IPsec SA. If the host that did not crash performs DPD, it might find out in a few minutes that it needs to renegotiate the IPsec SA. Otherwise, communication will be blocked until the IPsec SA rekey or expiry timer runs out. This could be an outage that lasts anywhere from 1-8 hours. Unfortunately, not all IKE daemons and IPsec implementations install the IPv6 neighbor discovery policy exception. The recommendation is to test this scenario when using a new IKE/IPsec implementation.⁴⁵

If a kernel receives a packet with an SPI for which it has no IPsec SA, it can send a message to the IKE process containing the IP address of the host that sent the IPsec packet. Such an IKE process may be able to recognize the peer based on its (static) IP address and initiate a new IKE exchange to try to set up a new IPsec SA that replaces the obsoleted IPsec SA on the host that did not crash. Not all kernels implement this mechanism to inform the IKE process.

The method for debugging IKE and IPsec configurations depends on the specific implementation. For new configurations that are not working properly, the first step should be for both endpoint administrators to verify the configuration options that they believe they have agreed upon. A checklist with the most common options to review can be found in Appendix A. A mismatch between basic IKE or IPsec parameters is most often the cause for new IPsec configurations not establishing properly.

Using a network monitoring tool such as tcpdump is not very useful for capturing and debugging the IKE negotiation because only information from the first IKE_SA_INIT exchange can be inspected, and the visible information only contains the DH groups. It is therefore unlikely that a misconfiguration can be detected at this point. All further captured IKE packets are encrypted, so they will not provide any additional information to diagnose the problem. It will be more helpful to enable additional logging or debugging. Remember to disable these settings again after the problem is resolved; otherwise, large amounts of logging information will continuously be produced.

If an administrator controls both endpoints that will be configured for IPsec, it is often the case that this administrator is located behind one of the gateways and is using a secure remote login tool, such as a web interface or SSH connection, to configure the remote endpoint. If a configuration mistake is made or a partial configuration is accidentally activated, the IPsec hosts will drop all non-IPsec traffic and lock out the administrator’s remote session. To prevent this problem, use a third host to indirectly log in to the remote IPsec endpoint for configuration.

The HMAC integrity algorithm may be implemented with three different hash functions: SHA-256, SHA-384, and SHA-512. SHA-2 means different things for different vendors. Some vendors use SHA-2 to mean only SHA2-256, while others mean it to be all variants of the SHA-2 family.

Care should be taken with sending DPD/liveness probes too often. If the remote client is a device that might enter sleep mode, it may not be able to respond to such probes. Another issue arises when the device’s link is congested while the IPsec connection is idle. This will trigger DPD/liveness probes that could be dropped due to traffic congestion. If repeatedly dropped, these packets will trigger a false positive warning about the remote IPsec endpoint connection being lost, causing the server to terminate the IKE and IPsec SAs and resulting in more packets to re-establish the VPN on an already congested link. Do not set DPD/liveness probes to values under one minute, which is in accordance with the recommendation in [23].

PFS and DH group negotiation issues can be tricky to diagnose. In IKEv2, the first IPsec SA is established with the IKE SA establishment, and it does not really use a separate DH key exchange for PFS (unlike IKEv1). Any mismatch in the DH group will only become apparent during a rekey message exchange hours later.

VPN gateways are also commonly used as NAT devices. If packets from the internal network are NAT’ed to the VPN server’s public IP before being considered for IPsec protection, the source IP will no longer match the IPsec policy, and the packet will not be sent out via IPsec. Instead, it could leak onto the internet without encryption or be caught by the firewall subsystem running on the VPN gateway.

In an IPv4-based network, machines within the same subnet use the address resolution protocol (ARP) to find the Ethernet address belonging to a local IP address. If remote access clients are being assigned IP addresses from the remote local area network (LAN), the VPN server needs to be configured to answer for all IP addresses that are reachable via the IPsec VPN since those remote VPN clients do not receive the local network ARP requests. This service is often called proxy ARP. Some IPsec implementations detect this automatically. For IPv6, this process is handled via IPv6 neighbor discovery, which would also need to be performed by the VPN server if the local IPv6 range is used for remote access clients.

The responder authenticates the initiator first and fully establishes the IPsec SA before the initiator receives the IKE_AUTH response packet. If the initiator determines that the responder failed to authenticate itself, the initiator can only notify the responder of this by immediately deleting the IKE SA since the responder believes this is a fully established IKE SA and IPsec SA. This sometimes confuses administrators when debugging a problem because from the responder’s point of view, this was a successful—but very short—IPsec connection.

IPsec implementations need to inspect packet streams to determine when a packet should be encrypted and when it should be transmitted unencrypted. One method is to use the routing table. If a route is pointing to a specific IPsec device, the IPsec implementation processes the packet based on its SPD/SAD rules. However, using routes can be fragile. Another subsystem could change the routing to accidentally or maliciously bypass the IPsec device, thus bypassing all encryption policies.

Another issue of routing-based policies is that administrators often use a single IPsec policy covering all possible IPv4 addresses (0.0.0.0/0) to all possible IPv4 addresses (0.0.0.0/0). Once the tunnel is established, routing is used to determine which packets to send over the IPsec connection. If a remote branch extends its network to use another subnet, such as 192.0.2.0/24, the only change needed is for the local branch to add a route for that IP range into the IPsec device. Firewall rules to limit the subnets allowed are omitted to allow this easy type of deployment, but this introduces a security problem as well as a compatibility problem. If the routes into the IPsec devices on both ends do not match, traffic will be encrypted in one direction but not in the other. At best, the IPsec gateway expecting encrypted packets will drop the unencrypted packets, and network connectivity will fail. Worse, the IPsec gateway could mistakenly route the unencrypted (and possibly modified) packets onto its local network.

Policy-based VPNs that cover only specific subnets rather than every address (0.0.0.0/0) are a better solution and are recommended over routing-based VPNs despite the additional management overhead required. Depending on the implementation, policy-based VPNs can be a bit harder to debug since it might not be obvious to the administrator where in the IP stack a packet is taken to be processed by the IPsec subsystem. This can lead to unexpected issues in hub-spoke deployments. For example, if a host with LAN IP address 10.0.2.1 and public IP 192.0.2.1 creates an IPsec tunnel to a remote host on IP 192.0.2.2 to cover traffic between 10.0.2.0/24 and 10.0.0.0/8, such an IPsec gateway might lose access to its own LAN since a packet with destination 10.0.2.13 will be sent over the IPsec tunnel because it falls within the destination IPsec policy range of 10.0.0.0/8. Routing-based VPNs do not have this issue because LAN packets do not pass through the routing table and instead find the target host to send the packet to via ARP.

One common implementation processes the packets for IPsec after the network monitoring hooks are consulted. This leads to debugging tools, such as the tcpdump tool, seeing the packet as leaving the host unencrypted, while, in fact, the packet is encrypted after it is shown to the network debugging tool.

The most common network issue when setting up IPsec is that a firewall on the VPN server or the network is blocking the IKE ports UDP 500 and 4500. If an IPsec connection works for simple ping commands but not when an application is trying to use the IPsec connection, the cause is most likely due to broken path MTU discovery. While this problem is not directly related to IPsec, it is often triggered because of the extra overhead of the ESP header making each 1500-byte original packet larger than 1500 bytes after the ESP header is added. The ESP packets would fragment, and, too often, some stateful router or firewall mistakenly drops these packets.

Problems with the Maximum Segment Size (MSS) can be encountered when the ESP packet contains a TCP packet. For TCP to work properly, it needs to be able to send ICMP packets with the “packet too big” notification, but ICMP is often blocked. Some IPsec policies might only allow TCP packets and prohibit ICMP packets. This also commonly manifests itself as an administrator who can log in over the IPsec connection using the SSH protocol, but as soon as the administrator tries to actually use this session, the connection freezes. Decreasing the MTU size of the IPsec interface can work around this issue. For TCP, a common workaround is to set a smaller TCP MSS size that ensures that packets are not bigger than the path MTU. This method is called TCP MSS clamping. Most implementations also allow the setting of a fixed value independent of the discovered path MTU. Common fixed values include 1340 or even 1200.

This section focuses on the planning and implementation of IPsec in an enterprise. As with any new technology deployment, IPsec planning and implementation should be addressed in a phased approach. A successful deployment of IPsec can be achieved by following a clear, step-by-step planning and implementation process. The use of a phased approach for deployment can minimize unforeseen issues and identify potential pitfalls early in the process. This model also allows for the incorporation of advances in new technology, as well as adapting IPsec to the ever-changing enterprise. This section explores each of the IPsec planning and implementation phases in depth, as follows:

Organizations should also implement other measures that support and complement IPsec implementations. These measures help to ensure that IPsec is implemented in an environment with the technical, management, and operational controls necessary to provide adequate security for the IPsec implementation. Examples of supporting measures include the following:

The purpose of this phase is to identify the need to protect communications and determine how that need can best be met. The first step is to determine which communications need to be protected (e.g., all communications between two networks, certain applications involving a particular server). The next step is to determine what protection measures (e.g., providing confidentiality, assuring integrity, authenticating the source) are needed for each type of communication. It is also important to identify other general and application-specific requirements, such as performance, and to think about future needs. For example, if it is likely that other types of communications will need protection in a year, those needs should also be considered.

After identifying all of the relevant needs, the organization should consider the possible technical solutions and select the one that best meets the identified needs. Although IPsec is typically a reasonable choice, other protocols such as TLS or SSH may be equally good or better in some cases. See Section 8 for descriptions of such protocols and guidance on when a particular protocol may be a viable alternative to IPsec. In some cases, IPsec is the only option—for example, if a gateway-to-gateway VPN is being established with a business partner that has already purchased and deployed an IPsec gateway for the connection. Another possibility is that the solution may need to support a protocol that is only provided by IPsec.

Assuming that IPsec is chosen as the solution’s protocol, the Identify Needs phase should result in the following:

Once the needs have been identified, and it has been determined that IPsec is the best solution, the next phase is to design a solution that meets the needs. This involves five major components, which are described in more detail starting in Section 7.2.1:

and remote access architectures), and host address space management considerations (for host-to-host and remote access architectures).

The decisions made regarding cryptography and packet filters are all documented in the IPsec policy. In its simplest form, an IPsec policy is a set of rules that govern the use of the IPsec protocol. It specifies the data to secure and the security method to use to secure that data. An IPsec policy determines the type of traffic that is allowed through IPsec endpoints and generally consists of a packet filter and a set of security parameters for traffic that matches the packet filter. Those parameters include the authentication and encryption scheme and tunnel settings. When communications occur, each packet filter can result in the establishment of one or more IPsec SAs that enable protected communications satisfying the security policy for that packet filter.

Other decisions should also be made during the design phase, such as setting IKE and IPsec SA lifetimes and identifying which DH group number is best. In addition to meeting the organization’s cryptographic requirements of NIST SP 800-131A [18] and FIPS 140 [13][14], design decisions should incorporate the organization’s logging and data management strategies, incident response and recovery plans, resource replication and failover needs, and current and future network characteristics, such as the use of wireless, NAT, and IPv6. Section 7.2.6 covers these considerations and design decisions in more detail.

The architecture of the IPsec implementation refers to the selection of devices and software to provide IPsec services and the placement of IPsec endpoints within the existing network infrastructure. These two considerations are often closely tied together; for example, a decision could be made to use the existing internet firewall as the IPsec gateway. This section explores three particular aspects of IPsec architecture: gateway placement, IPsec client software for hosts, and host address space management.

Due to the layered defense strategy used to protect enterprise networks, IPsec gateway placement is often a challenging task. As described later in this section, the gateway’s placement has security, functionality, and performance implications. The gateway’s placement may also have an effect on other network devices, such as firewalls, routers, and switches. Incorporating an IPsec gateway into a network architecture requires strong overall knowledge of the network and security policy. The following are major factors to consider for IPsec gateway placement:

In IPsec host-to-host and remote access architectures, each host must have an IPsec-compliant implementation installed and configured. Most operating systems on computers and mobile devices have built-in support for IPsec and only require configuration or an enterprise provisioning system that provides and installs the required configurations. However, some mobile devices or embedded devices do not have a built-in IPsec implementation. Also, some built-in clients might be lacking a feature required for a certain deployment or might not support an enterprise provisioning system. In such cases, a third-party client might need to be deployed instead. Third-party clients must be distributed and installed, then configured or provisioned.⁵⁰

Features that may be of interest when evaluating IPsec client software include support for the following:

Another important IPsec client feature is the ability to allow or prevent split tunneling. Split tunneling occurs when an IPsec client on an external network is not configured to send all of its traffic to the organization’s IPsec gateway. Requests with a destination on the organization’s network are sent to the IPsec gateway, and all other requests are sent directly to their destination without going through the IPsec tunnel. The client host is effectively communicating directly and simultaneously with the organization’s internal network and another network (typically, the internet). If the client host were compromised, a remote attacker could connect to the host surreptitiously and use its IPsec tunnel to gain unauthorized access to the organization’s network. This would not be possible if the IPsec client software had been configured to prohibit split tunneling. However, any compromise of an IPsec client host is problematic because an attacker could install utilities on the host that capture data, passwords, and other valuable information.

Prohibiting split tunneling can limit the potential impact of a compromise by preventing the attacker from taking advantage of the IPsec connection to enter the organization’s network; the attacker can only connect to the compromised system when it is not using IPsec. However, many hosts have multiple methods of connectivity, such as mobile data, wired LAN, and wireless LAN. If an attacker can connect to a network interface other than the one used for IPsec, it may be possible to use the IPsec tunnel even if split tunneling is prohibited. This can allow access to a more trusted network—the network protected by IPsec—from a less trusted network, such as an improperly secured wireless LAN. Accordingly, hosts should support being configured so that only the network interface used for IPsec is enabled when IPsec is in use. Some VPN clients can be configured to disable other network interfaces automatically. An alternative is to configure a personal firewall on the host so that it blocks unnecessary and unauthorized network traffic on all interfaces. Due to its security complications and risks, split tunneling is strongly discouraged.

As described in Section 7.2.6, not allowing split tunneling is also helpful in preventing the IPsec clients’ hosts from being compromised. If a user mistakenly tries to connect to a malicious site, the traffic would be forced to go through the VPN where an enterprise firewall or proxy server could filter malicious traffic. Some organizations prefer split tunneling because it prevents non-enterprise traffic from reaching the enterprise. It also reduces the internet bandwidth capacity needed by the enterprise to support its remote VPN clients. There may also be legal reasons why an enterprise prefers not to handle traffic unrelated to its organization.

There are other factors that may differentiate IPsec clients. For example, one client may provide substantially better performance or consume less of the host’s resources. Another consideration is the security of the client software itself, such as how frequently vulnerabilities are identified and how quickly patches are available. Client interoperability with other IPsec implementations is also a key concern; some client implementations only interoperate with their own vendor’s gateway implementation or with a limited number of other vendors’ gateway implementations. It is critical to ensure that the selected client will interoperate with each gateway implementation that it might encounter. Section 7.3.1 discusses this topic in more detail.

Organizations should also carefully consider how clients can be provisioned with IPsec client software and configuration settings, including policies. Many clients offer different features that can make client deployment, configuration, and management easier. For example, an administrator might be able to remotely set policy for clients rather than manually visiting each host. Some clients offer administrators the ability to lock out or disable certain configuration options or functionality so that users cannot inadvertently or intentionally circumvent the intended security. If administrators cannot distribute pre-configured IPsec clients or remotely control IPsec configuration settings, the administrators might need to manually configure each IPsec client or rely on users to follow instructions and configure the clients themselves. The latter approach is often challenging for non-technical users.

In remote access VPN architectures where the hosts are outside of the organization (e.g., mobile devices, remote workers), the VPN client will receive an additional IP address from the organization’s address space assigned as a virtual IP address to each external IPsec host. The client will then establish an IPsec connection that uses its real IP address in the external packet headers (so that the IPsec-encapsulated packets can be routed across public networks) and its virtual IP address in the internal packet headers (so that the packets can be routed across the organization’s internal networks and treated as internally generated).

Virtual addresses can be assigned from an address pool that resides on the VPN server. The VPN server can also use the DHCP relay protocol or a AAA service, such as RADIUS or Diameter, to obtain an IP address. A local pool can provide an easier indication that the IP address accessing a local resource is originating from a VPN client or is a client connecting from a certain region.

It is important to ensure that any addresses managed by the IPsec gateway are excluded from the ranges that other internal DHCP servers can assign to avoid address conflicts. Some vendors provide internal address assignment and authentication using proprietary functionality. This may present compatibility issues depending on the products being used.

When deploying a remote access VPN in a data center or cloud where the only service offered is the VPN server without any other local resources, non-routable IP addresses, such as those defined in RFC 1918 [41], can be used for the address pool of virtual IPs for the VPN clients. The VPN server then uses NAT to translate these IP addresses to its own public IP address. One potential issue with such a deployment is that some websites limit the number of users or connections coming from a single IP address. If dozens or hundreds of website users appear to all come from one VPN server public IP address, the website might block the IP address because it assumes that it is a malicious entity that obtained the credentials of many users. Using multiple public IP addresses on such a VPN server deployment could mitigate this problem.

The endpoints of a host-to-host and gateway-to-gateway IPsec architecture typically use the same authentication method to validate each other. Validation for remote access VPNs tend to use different mechanisms to authenticate each other, such as when the server is authenticated using a machine certificate, and clients are authenticated using EAP-TLS. IPsec implementations typically support a number of authentication methods. The most common methods are certificate-based digital signatures or raw public keys, EAP, and PSK. When using IKEv1, a group PSK combined with a username and password is also common. This section discusses the primary advantages and disadvantages of these methods.

PSKs should only be used for gateway-to-gateway scenarios that cross an administrative domain and only when based on generating strong and sufficiently long random PSKs with at least 112 bits of entropy. Using a public-key key pair (with or without certificates) based on RSA, DSA, or ECDSA is preferred over using PSKs, but if the implementations that need to interoperate do not share the same public key-based authentication method, PSKs are an appropriate alternative. Within an administrative domain, PSKs should not be used. For remote access VPN scenarios, EAP-TLS or machine certificate authentication should be used.

To use PSKs, the IPsec administrator needs to create a strong random secret key or password string that is then configured in both IPsec devices (the endpoints) of an IPsec connection.⁵² PSKs are the simplest authentication method to implement but are also, by far, the least secure. Administrators need to find IPsec products that provide key management capabilities for PSKs or implement their own key management mechanisms, such as generating, storing, deploying, auditing, and destroying keys. Proper key management can be quite resource intensive. Although it is easiest to create a single key that all endpoints share, this causes problems when a host should no longer have access; the key then needs to be changed on all other hosts. PSKs should also be updated periodically to reduce the potential impact of a compromised key. Another issue is that the key must be kept secret and transferred over secure channels. Individuals with access to an endpoint are almost always able to gain access to the PSK.⁵³ Depending on the key type, this could grant access from one, some, or all IP addresses. Using the same key for a group of endpoints also reduces accountability since anyone within the group can impersonate another member of the group. Therefore, NIST discourages the use of a PSK for more than two entities.

Because of scalability and security concerns, PSK authentication is generally an acceptable solution only for small-scale implementations with known IP addresses or small IP address ranges. The use of a single PSK for a group of hosts is strongly discouraged for all but the most highly controlled environments, such as a group of secure routers. PSKs are generally not recommended for remote access clients that have dynamic IP addresses because the keys cannot be restricted to a particular IP address or small range of IP addresses. PSKs are also frequently used during initial IPsec testing and implementation because of their simplicity. After the IPsec implementation is operating properly, the authentication method can be changed.

Certificates for IPsec are typically used in “machine certificate” and EAP-TLS-based authentication. The certificate owner produces a digital signature of the IKE exchange that proves its possession of the certificate’s private key and authenticates the IKE session.

A certificate identifies each device, and each device is configured to use certificates. User-specific certificates may be used instead of device-specific certificates, but some remote access

VPN configurations do not allow a single user to log onto multiple devices simultaneously, so it is always better to generate a certificate per device rather than per user.

Two IPsec endpoints will trust the validity of the presented peer certificate if a CA that they both trust has signed their certificates.⁵⁴ The certificates must be securely stored in the local certificate store on the IPsec hosts and gateways or on a secure hardware token. Using a certificate-based method allows much of the key administration to be offloaded to a central certificate server but still requires IPsec administrators to perform some key management activities, such as provisioning hosts with credentials, either through IPsec vendor-provided features or IPsec administrator-created capabilities. Many organizations implement a public key infrastructure (PKI) for managing certificates for IPsec VPNs and other applications, such as secure email and web access.⁵⁵ Certificates can be issued to limit their use using EKU attributes. Some IPsec hosts insist on IPsec-specific EKUs, while others accept the TLS-based EKUs (serverAuth or clientAuth) and some ignore all EKUs. The IETF PKI standard for IKE EKUs is specified in RFC 4945 [60]. A certificate issued for secure email might not be usable for IPsec on some of the VPN gateways deployed in an organization. Issuing certificates per device instead of per user avoids this issue and has the additional advantage that if a device is lost or stolen, not all of the user’s VPN access will need to be revoked.

Although the certificate authentication method scales well to large implementations and provides a much stronger security solution than PSKs, it does have some disadvantages. While certificates can be revoked and transmitted to the VPN servers via CRLs [61] in bulk or on demand via OCSP [62], these mechanisms typically provide no option for temporarily disabling a certificate. Additional complications can occur when the connection to the OCSP server itself is down or, worse, requires an IPsec tunnel to be negotiated that needs to use that OCSP server. Non-standard solutions using a AAA server or a pluggable authentication module (PAM authentication) are usually added for such use cases.

Another potential problem with the certificate authentication method involves packet fragmentation. Packets in an IKE negotiation are typically relatively small and do not need to be fragmented. By adding certificates to the negotiation, packets may become so large that they need to be fragmented, which is not supported by some IPsec implementations.

Raw public key digital signatures work the same way as certificate-based digital signatures, except that instead of trusting a certificate (directly or indirectly via a CA), the trust is placed in the public key itself. Keys are usually represented in base64 format or using just the SPKI part of a certificate.

Public keys can be distributed to the endpoints via trusted provisioning software or can be fetched on demand from DNSSEC or a directory service (e.g., Lightweight Directory Access Protocol [LDAP]) based on the ID presented during the IKE exchange. Instead of specifying the validity period in a certificate, these publishing services can simply remove the key when it is no longer needed. The public key for a particular ID specified in IKE resides in the DNS or directory service under that ID name. Revocation is accomplished by removing the public key from the publishing service’s database.

For resource-constrained embedded devices that authenticate using a single, hard-coded public key, a certificate by itself can be too large to be contained or operated on and serves no purpose since certificate validation is not performed.

One disadvantage of raw public keys is that there are not as many tools that support these because most IKE implementations have been written to be used with certificates or PSKs.

EAP support is included in IKEv2. Both older and newer EAP methods are supported. EAP can be used as the only authentication method or as a second authentication method. Often, different authentication methods are used: the server is authenticated using certificate-based authentication, and the client (typically a laptop or mobile device) is authenticated using an EAP method. EAP authentication allows additional types of authentications to be used, such as a username with a password (EAP-MSCHAPv2), a user (not host) certificate (EAP-TLS), or an EAP method supporting two-factor authentication. EAP authentication is mostly used for laptops and mobile phones.

Setting the cryptographic policy for confidentiality and integrity protection and for key exchange involves choosing encryption and integrity protection algorithms,⁵⁶ key lengths, DH groups for key exchange, and IKE and ESP lifetimes. For up-to-date policies and advice on these settings, see NIST SP 800-131A [18] and FIPS 140 [13][14] as well as the recommendations of the IETF for IKE [25] and ESP [63]. Note that these documents will be updated over time or be obsoleted for newer publications.

The IKE protocol sends just a few packets per hour, so it makes sense to be extra cautious and pick strong algorithms with large enough keys and, specifically, a strong DH group. Approved DH groups are identified in NIST SP 800-56A [64]. The bulk of the CPU power of an IPsec host will be spent on IPsec, not IKE. In IKE, the most CPU-intensive operation is the DH calculation. When an IPsec host has hundreds or thousands of IKE (re)connections, choosing the right DH group becomes very important.

The use of strong key sizes is recommended for IKE. The performance impact of larger key sizes is minimal because IKE traffic is negligible compared to IPsec traffic. For IPsec (ESP), the key size can have a significant impact on performance. In general, use larger key sizes for IPsec if performance is not an issue. For ESP, the choice of algorithms for confidentiality and integrity protection should also take performance into account. Using an AEAD algorithm that can provide both confidentiality and integrity protection in a single operation, such as AES-GCM, will give better performance than using non-AEAD algorithms that require separate operations (e.g., AES-CBC for encryption and HMAC for integrity protection). It is important to estimate the processing resources that the cryptographic computations will require during peak usage.

It is uncommon to use 192-bit AES keys, and this key length is optional in [25]. It is worth mentioning as well that in the future, an adversary with a quantum computer may be able to reduce the key strength of an AES key by a factor of two, in which case a 256-bit AES key may effectively provide around 128 bits of security in the quantum computer world.

AES-GCM (an AEAD algorithm) is often offloaded to hardware, making it significantly faster than AES-CBC (a non-AEAD algorithm) in combination with an HMAC. The CPU is typically the hardware component most affected by cryptographic operations. In some cases, a hardware-based cryptographic engine with a customized CPU (also known as a cryptographic accelerator) may be needed for greater throughput, but this may limit the algorithm options. Another potential issue is export restrictions involving the use of encryption algorithms in certain countries.⁵⁷ In addition, some IPsec components may not provide support for a particular algorithm or key size.

For integrity checking of non-AEAD algorithms, most IPsec implementations offer HMAC-SHA-1 or HMAC-SHA-2. Even though HMAC-SHA-1 is still a NIST-approved option, the HMAC-SHA-2 algorithms are recommended because they have stronger security than HMAC-SHA-1. HMAC-MD5 has never been a NIST-approved algorithm and shall not be used.

In some implementations of IPsec, the cryptographic policy settings are not immediately apparent to administrators. The default settings for encryption and integrity protection, as well as the details of each setting, are hidden in separate menus that are harder to access or are split among multiple locations. It is also challenging with some implementations to alter the settings once they have been located. For example, by having portions of the settings in multiple locations, administrators may need to go back and forth between different configuration screens to ensure that the settings are correct and consistent.

While network devices such as routers and firewalls will already be optimized for network performance, generic operating systems will require tuning for optimized network performance. Enough RAM should be made available to the network stack. CPU power saving and throttling should be disabled, and on non-uniform memory access (NUMA) systems, further optimizations might be possible. Consult the hardware vendor for specific instructions.

Network card settings can also have a large impact on throughput. Check that the network card’s transmit queue (txqueuelen) is set large enough to accommodate the amount of traffic. Check the network card settings for TCP segmentation offload (TSO), generic segmentation offload (GSO), checksum offloading, and virtual local area network (VLAN) settings. If using a network card with IPsec hardware acceleration support, follow the vendor’s instructions on how to optimize the host.

When using virtualization, ensure that the virtualization layer is using as much direct hardware access as possible. For performance, it will be better to configure a hardware network card inside a virtual machine than to configure the virtual machine with a virtual network card. On some hardware, this needs to be enabled in the Basic Input/Output System (BIOS). For example, on Intel systems, ensure that Intel Virtualization Technology for Directed I/O (Intel VT-d) is enabled. Ensure that the virtualization is not emulating a slightly different CPU than the real hardware because it will not be able to use the hardware virtualization instructions of the CPU and instead will have to perform full emulation in software.⁵⁸

Ideally, when not using IPsec, the system should be able to utilize line-speed unencrypted traffic. A popular network tool to perform network performance tests is iperf. Once the system is performing well without IPsec, IPsec can be enabled.

IPsec hosts that are busy will spend the bulk of their computational resources on encrypting and decrypting ESP traffic. The performance of the algorithms for IKE is less important since there are far fewer IKE packets than ESP packets in most deployments of IPsec VPNs.

If the host’s CPU usage is the limiting factor, it is particularly important to use the right algorithms. Using an AEAD algorithm for encryption and integrity protection is much faster than using two non-AEAD algorithms. The best algorithm choice will likely be AES-GCM because modern CPUs have hardware support for it. Both 256-bit and 128-bit AES keys currently provide strong protection, so when CPU load becomes an issue, one could consider switching from 256-bit to 128-bit keys, provided that this is allowed by the deployment policy. Otherwise, 256-bit keys are recommended.

If the host is running a few high-speed IPsec SAs, it could be that multiple CPUs on the host are not utilized properly to spread the cryptographic load of a single IPsec SA over multiple CPUs. When multiple CPUs are used for a single IPsec SA, there will be an increase in out-of-order packets being sent, and the replay window will need to be increased to accommodate this at both endpoints. IPsec replay protection can be disabled to test if that is the limiting factor for the server performance. This is less of a concern on busy servers that act as a remote access VPN since these will be serving many users’ IPsec SAs per CPU. For high-speed IPsec SAs, it is also important to use ESNs to avoid excessive rekeying.

If the application is sending packets close to the MTU size, using ESP encryption (which adds a few bytes in size compared to the unencrypted packet size) might lead to fragmentation, which will reduce performance. If the IPsec SA is a connection within a data center or over a dedicated fiber cable, it might be possible to increase the MTU size (e.g., to 9000 bytes) to prevent fragmentation. The MTU of the internal-facing network card can also be reduced to force the LAN to send packets that are smaller than 1500 bytes so that once the host encrypts the packet to send it out over the external interface, the ESP packet will not exceed an MTU of 1500 bytes. TCP MSS clamping can be used on both IPsec endpoints to ensure that TCP sessions will use a lower MTU that prevents fragmentation.

While IKE performance in most cases does not matter, it does matter for remote access VPN servers that have a continuous stream of clients connecting and disconnecting. If IKE uses too much of the CPU resources, ESP processing times will be impacted as well. If a remote access VPN server is too busy and has degraded to the point where an IKE session takes more than a few seconds to establish, the server will completely collapse under the load. IKE clients usually time out after 5 to 10 seconds and will start a new IKE attempt. This will put even more load on the already loaded server. That is, the load based on the number of IKE clients connecting will slowly rise until it hits a breaking point. If the IKE REDIRECT [44] extension is supported, the server can be configured to start redirecting clients to another server before it becomes too busy. See Section 3.8 for more information.

The most computationally expensive part of IKE is the DH calculation that is performed during a key exchange. DH implemented using ECP groups (elliptic curve group modulo a prime) takes fewer resources than the use of finite field groups (i.e., modular exponential or MODP groups), such as DH group 14. The DH 19, DH 20, and DH 21 ECP groups are also considered more secure than the MODP groups [65]. DH groups 1, 2, 5, and 22 are not NIST-approved because these groups do not supply the minimum of 112 bits of security. See NIST SP 800-56A [64] for further information about approved DH groups.

MOBIKE should be enabled on remote access VPN servers. Mobile devices will switch between WiFi and mobile data. Without MOBIKE, this requires a new IKE session for each network switch, which will increase the number of DH calculations that need to be supported. IKE clients on unreliable WiFi can result in many IKE sessions being reset and restarted. When MOBIKE is used, an encrypted informational exchange message is sent to modify the existing IKE and ESP sessions to use the new IP address of the other interface and avoid starting new sessions with new, expensive DH group calculations.

Liveness⁵⁹ probes can be used by a server to detect remote clients that have vanished without sending a delete notification. The timer for these probes should not be set too short or else the server will need to send frequent IKE packets with DPD probes for idle IKE clients. If the timeout value is set very short (on the order of a few seconds), there is the additional risk of IKE clients on unreliable networks not receiving the DPD probes. The server will disconnect the IKE client when a response to the probe is not returned. That client will experience packet loss and declare the IPsec connection dead. This will lead to the creation of another new IKE session and an increased load on the VPN server. In general, keeping a few IKE and IPsec states alive for vanished VPN clients takes very little memory and no CPU resources. A reasonable DPD timeout value is in the range of 10 to 60 minutes.

The IKE SA and IPsec SA lifetimes are not negotiated. Each endpoint decides when it wants to rekey or expire an existing SA. Using longer IKE SA and IPsec SA lifetimes can reduce the amount of IKE rekeying required. IKE rekeying and IPsec rekeying with PFS require a new DH calculation as well, so extending the IKE and IPsec lifetimes can help reduce the server load. Increased lifetimes must not result in keys encrypting too much data. The upper limit on the amount of data that can be safely encrypted depends on the specific encryption algorithm used.

Another option on busy servers with many remote access users is to support IKE session resumption [66]. A mobile device that is going to sleep can send the server a sleep notification to prevent DPD-based disconnections. The server and client keep the cryptographic state of the IKE session. When the device wakes up, it can send an encrypted session resumption request. This avoids the need for a new IKE session with the expensive DH calculation to establish a new connection; the server is triggered via a DPD timeout to delete the IKE and IPsec SA if the sleep period exceeds the timeout period.

If a provisioning system is used to generate and install configurations for the IKE clients, optimized settings could be pushed automatically to all IKE clients to ensure optimal performance. This would avoid manual configurations that, when performed by inexperienced users, could result in less optimized settings because the user did not enable or disable certain features.

Enabling IKE debugging can cause a lot of data to be logged. That in itself can cause a significant performance impact on the system. Always check to see if debugging has accidentally been left enabled on systems experiencing a high workload.

DDoS attacks are a separate issue of concern. Such attacks also put an additional load on the server, but the characteristics are different from a legitimate user load.

An attack from an authenticated user with valid credentials is assumed to be a readily solvable problem—simply revoke the user’s access to the VPN infrastructure. One exception to this is when anonymous IPsec is in use because in that case, the connection cannot be terminated or prevented based on the user credentials. Vendors of IPsec equipment supporting anonymous IPsec connections should take countermeasures by, for example, limiting the number of IPsec SA requests that are accepted or by limiting the number of rekeys or anonymous connections allowed based on an IP address.

IKEv2 has built-in protection against DDoS attacks, but IKEv1 does not. When the number of incomplete IKE sessions (sometimes called half-open IKE SAs) reaches a threshold, indicating a possible DDoS attack, IKEv2 can enable DDoS COOKIES. Each new IKE_SA_INIT request will be answered with a reply that only contains a COOKIE based on a local secret⁶⁰ and the client’s IP address and port. The client will have to resend its original IKE_SA_INIT request with the COOKIE added to the request. The server can calculate the value of the COOKIE without needing to store any state in memory for the original IKE_SA_INIT request. The IKE server will only perform the expensive DH calculations after the client has retransmitted its IKE_SA_INIT packet with the COOKIE, proving to the server that the client was not simply a spoofed IP packet.

Additionally, IKEv1 can be coerced into an amplification attack. With IKEv1, the responder and initiator are each responsible for retransmission when a packet is lost. A malicious user can send a single spoofed IKEv1 packet to an IKEv1 server and cause that IKEv1 server to send several retransmit packets to the spoofed IP address. Some IKEv1 implementations defend against this by never responding more than once to an initial IKEv1 request, but this can break legitimate IKEv1 clients using aggressive mode when there is actual packet loss happening.

The purpose of the packet filter is to specify how each type of incoming and outgoing traffic should be handled—whether the traffic should be permitted or denied (usually based on IP addresses, protocols, and ports) and how permitted traffic should be protected (if at all). By default, IPsec implementations typically provide protection for all traffic. In some cases, this may not be advisable for performance reasons. Encrypting traffic that does not need protection or is already protected (e.g., encrypted by another application) can be a significant waste of resources. For such traffic, the packet filter could specify the use of the null encryption algorithm for ESP, which would provide integrity checks and anti-replay protection, or the packet filter could simply pass along the traffic without any additional protection at all and fully depend on the application to provide data integrity. One caveat is that the more complex the packet filter becomes, the more likely it is that a configuration error may occur, which could permit traffic to traverse networks without sufficient protection.

An issue related to packet filters is that certain types of traffic are incompatible with IPsec. For example, IPsec cannot negotiate security for multicast and broadcast traffic.⁶¹ This means that some types of applications, such as multicast-based video conferencing, may not be compatible with IPsec. Attempting to use IPsec to secure such traffic often causes communication problems or impairs or breaks application functionality. Other traffic, such as multicast DNS (mDNS) and DNS Service Discovery (DNS-SD) broadcast requests, should not be forwarded to other networks because they have no meaning or relevance beyond the local network. For example,

ICMP error messages are often generated by an intermediate host such as a router, not by a tunnel endpoint; because the source IP address of the error message is the intermediate host’s address, these ICMP packets do not have confidentiality or integrity protection, and the receiving host cannot make security policy decisions based on unprotected packets. Packet filters should be configured to not apply IPsec protection to types of traffic that are incompatible with IPsec; rather, they should let the traffic pass through unprotected if that does not compromise security. If the IPsec gateway cannot block broadcasts and other traffic that should not be passed through it, it may also be effective to configure firewalls or routers near the IPsec gateway to block that particular type of traffic.

A particularly important consideration in design decisions is the identification and implementation of other security controls. Organizations should have other security controls in place that support and complement the IPsec implementation. For example, organizations should configure packet filtering devices (e.g., firewalls, routers) to restrict direct access to IPsec gateways. Organizations should have policies in place regarding the acceptable use of IPsec connections and software. Organizations may also set minimum security standards for IPsec endpoints, such as mandatory host hardening measures and patch levels, and specify security controls that must be employed by every endpoint.

For endpoints outside of the organization’s control—such as systems belonging to business partners, users’ home computers, and public internet access networks—organizations should recognize that some of the endpoints might violate the organization’s minimum security standards. For example, some of these external endpoints might be compromised by malware and other threats; malicious activity could then enter the organization’s networks from the endpoints through their IPsec connections. To minimize risk, organizations should restrict the access provided to external endpoints as much as possible and also ensure that policies, processes, and technologies are in place to detect and respond to suspicious activity. Organizations should be prepared to identify users or endpoint devices of interest and disable their IPsec access rapidly as needed.

IPsec packet filters can be helpful in limiting external IPsec endpoints’ access to the organization. Using packet filters to limit acceptable traffic to the minimum necessary for untrusted hosts, in addition to other network security measures (e.g., firewall rulesets, router access control lists), should be effective in preventing certain types of malicious activity from reaching their targets. Administrators may also need to temporarily suspend access for infected hosts until appropriate host security measures (e.g., antivirus software update, patch deployment) have resolved any infection-related issues. Another option in some environments is to automatically quarantine each remote host that establishes an IPsec connection, checking its host security control settings and then deciding if it should be permitted to use the organization’s networks and resources. It is advisable to perform these checks, not only for hosts connecting to the organization’s VPN from external locations but also for mobile systems connecting to the organization’s internal network that are also sometimes connected to external networks.

In addition to endpoint security, there are many other possible design considerations. The following items describe specific IPsec settings not addressed earlier in this section:

a remote access VPN with thousands of users might experience an additional workload if PFS is enabled on all VPN clients.

Design decisions should incorporate several other considerations, as described below:

Table 2 provides a checklist that summarizes the major design decisions made during the first two phases of the IPsec planning and implementation process.

Table 2: Design Decisions Checklist