IPsec VPN (IKEv2) with pfSense and OS X El Capitan

This note is meant to help troubleshoot OS X failing to connect to an IPsec VPN, particularly when using certificate-based authentication (EAP-TLS).

Update (16-Feb-2019): Incorporates new advice from the strongSwan Security Recommendations document and RFC 8247 on suitable algorithms for encryption & hashing.

Update (16-Feb-2019): Now includes some detail on the Windows 10 Set-VpnConnectionIPsecConfiguration PowerShell command, which makes it possible to use one common VPN server configuration for all three platforms I use.

Table of contents:

Settings that work

There are two areas which require specific configuration:

  1. The VPN server encryption & hash algorithms, and
  2. The OS X VPN client settings.

Encryption & hash algorithms; DH groups

By default, OS X El Capitan supports a limited number of valid possible configurations. Here are the possible configurations for each phase, and which settings I’d recommend implementing on the VPN server. They are listed in the order they were sent by the client.

Table 1: OS X El Capitan supported Phase 1 possible encryption, hash algorithm, and DH group options.
PhaseEncryptionHashDH GroupNotes
Phase 1AES-CBC-256SHA25614 (2048 bit) Recommended.
AES-CBC-256SHA25619 (nist ecp256) Recommended if you’re unconcerned about the use of elliptic curves.
AES-CBC-256SHA2565 (1536 bit) Not recommended (DH group 5 is no longer considered secure).
Also DH group 5 is unsupported by the strongSwan client for Android.
AES-CBC-128SHA12 (1024 bit) Not recommended (DH group 2 is no longer considered secure).
3DESSHA12 (1024 bit) Not recommended (3DES and DH group 2 are no longer considered secure).
Table 2: OS X El Capitan supported Phase 2 possible encryption and hash algorithm options.
PhaseEncryptionHashNotes
Phase 2AES-CBC-256SHA256 Recommended.
AES-CBC-128SHA1 Not recommended (SHA1 is no longer considered secure).
3DESSHA1 Not recommended (3DES is no longer considered secure).

VPN server settings for OS X and Android (strongSwan) clients

In order to use these settings, here is what I implemented in pfSense:

Table 3: VPN server settings chosen for security, compatibility, and performance.
PhaseSettingsNotes
Phase 1Encryption algorithm: AES-CBC-256
Hash algorithm: SHA256
DH group: 14 (2048 bit)
Recommended.
Chosen to avoid DH group 2 (1024 bit).
Phase 2Encryption algorithm:
AES-CBC-256 or AES-GCM-128 (128 bits)
Hash algorithm: SHA256
Recommended.
Chosen to avoid SHA1.

OS X VPN client settings

Because I’m using certificate-based authentication (EAP-TLS), some of the settings which are typically left blank must be filled in with the correct values for the connection to be successful:

Table 4: OS X VPN client settings required for certificate-based authentication.
SettingExplanation
Server Address: host name or IP addressAs long as your server certificate contains SANs for both host name and IP address, you can use either here.
Remote ID: the CN of your server certificateThis must be the CN of your server certificate, usually its fully-qualified domain name.
Local ID: the CN of your user certificateThis must be the CN of your user certificate; in my case, this is just a username.

Remember to import your user certificate (with private key), import the CA certificate and any intermediate certificates, and mark the CA as trusted.

Background of my VPN setup and goals

I’m using the following server and client components:

Here are the goals for this setup in rough order of importance:

Initial VPN setup instructions

Initial setup is based on this documentation from the pfSense wiki:

This was tested and working well with my Nexus 6P (Android) but I hadn’t attempted to use it with OS X yet.

Troubleshooting OS X VPN client issues

Here are some issues I encountered and how I resolved them.

Obstacle 1: The OS X VPN client GUI provides no help whatsoever

The first thing you notice when using the OS X VPN client is that connection failures are completely silent. The only way you could tell that it failed was that the word “Connected” doesn’t appear. This is of course very unhelpful, and Apple should be a bit embarrassed to fail the user in this way.

My advice for troubleshooting the VPN connection is to keep the Console application open and visible, pointed at ‘system.log’.

Obstacle 2: The OS X ‘system.log’ file is full of red herrings

Take, for example, the following error message seen in ‘system.log’:

Jul 27 19:46:37 isomer nesessionmanager[12598]: Failed to find the VPN app for plugin type com.apple.neplugin.IKEv2

I have no idea what this means, but it doesn't seem to have any relevance to whether or not the connection was successful.

It’s not just the odd error message, either; there are sometimes full backtraces in ‘system.log’, too:

Jul 27 19:46:38 isomer neagent[12699]: BUG in libdispatch client: kevent[EVFILT_READ] delete: "Bad file descriptor" - 0x9
Jul 27 19:46:38 isomer kernel[0]: SIOCPROTODETACH_IN6: ipsec0 error=6
Jul 27 19:46:38 isomer symptomsd[369]: nw_interface_create_with_name netutil_ifname_to_ifindex(ipsec0) failed, dumping backtrace:
[...snip...]

None of this is helpful (at least, not to me).

Eventually I clued in that most of the interesting errors came from neagent. If you’re having trouble finding the ‘real’ error, try filtering by that.

Revelation 1: My VPN client can’t do DNS lookups

The first head-scratcher is that, for reasons not known to me, the VPN client wouldn’t do DNS lookups. The error seen in ‘system.log’ is:

Jul 27 19:32:57 isomer neagent[12599]: IKEv2 Plugin: ikev2_resolve_server_name: failed to query DNS
Jul 27 19:32:57 isomer neagent[12599]: IKEv2 Plugin: Connect: Attempt to query DNS failed

The fix here is simple: use the IP address for the server address.

Update (12-Sep-2017): Others have reported seeing this issue. I did some testing with Wireshark and noticed that, in these cases, no DNS query attempts were even made.

More testing revealed the server address field must be at least ten characters long. There’s also some other “mystery validation” happening here; my hostname is eleven characters and is still being rejected. If I modify the hostname slightly (swap a few characters), DNS resolution works. Probably this is a bug.

Revelation 2: The Phase 1 settings of the server and client must match

The next issue was more elusive. I had a hard time reading ‘system.log’, so I pulled out my favourite utility, tcpdump(1), on the client, and had a look.

fission@isomer[s008]:~% sudo tcpdump -nvvi en0 port 500
tcpdump: listening on en0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:41:29.971881 IP (tos 0x0, ttl 64, id 3664, offset 0, flags [none], proto UDP (17), length 632)
    192.168.43.5.500 > 203.0.113.10.500: [udp sum ok] isakmp 2.0 msgid 00000000 cookie 4b0bd8cb99e5f6ac->0000000000000000: parent_sa ikev2_init[I]:
    (sa: len=216
        (p: #1 protoid=isakmp transform=4 len=44
            (t: #1 type=encr id=aes (type=keylen value=0100))
            (t: #2 type=prf id=#5 )
            (t: #3 type=integ id=#12 )
            (t: #4 type=dh id=modp2048 ))
        (p: #2 protoid=isakmp transform=4 len=44
            (t: #1 type=encr id=aes (type=keylen value=0100))
            (t: #2 type=prf id=#5 )
            (t: #3 type=integ id=#12 )
            (t: #4 type=dh id=#19 ))
        (p: #3 protoid=isakmp transform=4 len=44
            (t: #1 type=encr id=aes (type=keylen value=0100))
            (t: #2 type=prf id=#5 )
            (t: #3 type=integ id=#12 )
            (t: #4 type=dh id=modp1536 ))
        (p: #4 protoid=isakmp transform=4 len=44
            (t: #1 type=encr id=aes (type=keylen value=0080))
            (t: #2 type=prf id=hmac-sha )
            (t: #3 type=integ id=hmac-sha )
            (t: #4 type=dh id=modp1024 ))
        (p: #5 protoid=isakmp transform=4 len=40
            (t: #1 type=encr id=3des )
            (t: #2 type=prf id=hmac-sha )
            (t: #3 type=integ id=hmac-sha )
            (t: #4 type=dh id=modp1024 )))
    (v2ke: len=256 group=modp2048)
    (nonce: len=16 nonce=(5195098add8e0e74f6bb7ff30ae40d72) )
    (n: prot_id=#0 type=16406(status))
    (n: prot_id=#0 type=16388(nat_detection_source_ip))
    (n: prot_id=#0 type=16389(nat_detection_destination_ip))
    (n: prot_id=#0 type=16430(status))
19:41:30.923281 IP (tos 0x0, ttl 56, id 40183, offset 0, flags [none], proto UDP (17), length 64)
    203.0.113.10.500 > 192.168.43.5.500: [udp sum ok] isakmp 2.0 msgid 00000000 cookie 4b0bd8cb99e5f6ac->0000000000000000: parent_sa ikev2_init[R]:
    (n: prot_id=#0 type=14(no_protocol_chosen))

Interesting – the client has offered a series of possible encryption types, hash algorithms, and DH groups, but the server has responded with no_protocol_chosen. This is good information, as it indicates that things went off the rails early on.

This helped me to find the client error message in ‘system.log’:

Jul 27 19:41:31 isomer neagent[12638]: Failed to process IKE SA Init packet

The server logs for that time period say:

Jul 27 19:41:30	charon	02[CFG] <44> received proposals: IKE:AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048, IKE:AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/ECP_256, IKE:AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_1536, IKE:AES_CBC_128/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024, IKE:3DES_CBC/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024
Jul 27 19:41:30	charon	02[CFG] <44> configured proposals: IKE:AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_1024
Jul 27 19:41:30	charon	02[IKE] <44> received proposals inacceptable

Table 1 above has a breakdown of the various Phase 1 options supported by the OS X VPN client, which did not match what the server was offering: at that time, the Phase 1 configuration only included AES 256 with SHA256 and DH group 2 (1024 bit). However, that was not a configuration the OS X VPN client supported.

To fix this, I had to find a set of Phase 1 options which were compatible with both OS X and strongSwan for Android. Here are the Phase 1 possibilities for strongSwan for Android:

Table 5: strongSwan for Android supported Phase 1 possible encryption, hash algorithm, and DH group options.
PhaseEncryptionHashDH Group
Phase 1 AES-CBC-128
AES-CBC-192
AES-CBC-256
3DES
SHA256
SHA384
SHA512
MD5
SHA1
AES-XCBC
19 (nist ecp256)
20 (nist ecp384)
21 (nist ecp521)
28 (brainpool ecp256)
29 (brainpool ecp384)
30 (brainpool ecp512)
31 (curve25519)
15 (3072 bit)
16 (4096 bit)
18 (8192 bit)
14 (2048 bit)
2 (1024 bit)
AES-GCM-128 (128 bits)
AES-GCM-192 (128 bits)
AES-GCM-256 (128 bits)
ChaCha20/Poly1305
AES-GCM-128 (96 bits)
AES-GCM-192 (96 bits)
AES-GCM-256 (96 bits)
AES-GCM-128 (64 bits)
AES-GCM-192 (64 bits)
AES-GCM-256 (64 bits)
SHA256
SHA384
SHA512
MD5
SHA1
AES-XCBC
19 (nist ecp256)
20 (nist ecp384)
21 (nist ecp521)
28 (brainpool ecp256)
29 (brainpool ecp384)
30 (brainpool ecp512)
31 (curve25519)
15 (3072 bit)
16 (4096 bit)
18 (8192 bit)
14 (2048 bit)
2 (1024 bit)

Interestingly, the strongSwan for Android client supports every single Phase 1 configuration that the OS X VPN client supports, except DH group 5 (1536 bit).

The solution here is to change the VPN server to use DH group 14 (2048 bit), which creates an option set compatible with both clients. Other valid – though suboptimal – choices would have been to choose DH group 19 (nist ecp256), or to choose AES 128 encryption.

Revelation 3: The remote ID and local ID must match the certificates

Most OS X VPN guides I found on the Internet are geared for username & password-based authentication; I didn’t yet see one that really explains the validation process for certificates.

As mentioned in table 4, both the server and client certificate CNs are required to match the remote ID and local ID fields, respectively, in the VPN client configuration. Initially I had the remote ID set correctly, but I didn’t realize I needed to set the local ID as well. This got me the following error in ‘system.log’:

Jul 28 22:53:07 isomer neagent[14549]: [eaptls_plugin.c:397] eaptls_handshake(): SSLHandshake failed, errSSLPeerCertUnknown (-9829)
Jul 28 22:53:07 isomer neagent[14549]: Failed to process IKE Auth (EAP) packet

On the server side, the errors look like this:

Jul 27 23:03:44	charon	05[TLS] <con1|66> no trusted certificate found for '192.168.43.5' to verify TLS peer
Jul 27 23:03:44	charon	05[TLS] <con1|66> sending fatal TLS alert 'certificate unknown'
[...snip...]
Jul 27 23:03:44	charon	05[IKE] <con1|66> EAP method EAP_TLS failed for peer 192.168.43.5

The solution is to specify the user certificate’s CN for the local ID field in the VPN configuration.

Coincidentally, if you fail to specify the server CN for the remote ID field, you get this:

Jul 28 22:55:03 isomer neagent[14562]: Failed to create IKE Auth packet

The server sees a timeout, since the OS X VPN client simply gives up. Notice the thirty second gap between log entries:

Jul 28 22:55:03	charon	10[NET] <77> sending packet: from 203.0.113.10[500] to 192.168.43.5[500] (493 bytes)
Jul 28 22:55:33	charon	10[JOB] <77> deleting half open IKE_SA after timeout

Revelation 4: The Phase 2 settings of the server and client must match

After these three fixes, I saw this in ‘system.log’:

Jul 27 23:08:07 isomer neagent[12983]: Received error: Error (No Proposal Chosen)
Jul 27 23:08:07 isomer neagent[12983]: Failed to process IKE Auth packet

And on the server side:

Jul 27 23:08:07	charon	12[CFG] <con1|67> received proposals: ESP:AES_CBC_256/HMAC_SHA2_256_128/NO_EXT_SEQ, ESP:AES_CBC_256/HMAC_SHA2_256_128/NO_EXT_SEQ, ESP:AES_CBC_256/HMAC_SHA2_256_128/NO_EXT_SEQ, ESP:AES_CBC_128/HMAC_SHA1_96/NO_EXT_SEQ, ESP:3DES_CBC/HMAC_SHA1_96/NO_EXT_SEQ
Jul 27 23:08:07	charon	12[CFG] <con1|67> configured proposals: ESP:AES_CBC_256/HMAC_SHA1_96/NO_EXT_SEQ, ESP:AES_CBC_256/HMAC_SHA1_96/NO_EXT_SEQ
Jul 27 23:08:07	charon	12[IKE] <con1|67> no acceptable proposal found
Jul 27 23:08:07	charon	12[IKE] <con1|67> failed to establish CHILD_SA, keeping IKE_SA

This looks a lot like the Phase 1 failure, except the proposals now start with ESP: instead of IKE:, indicating that we are attempting to establish an ESP tunnel. Table 2 shows the Phase 2 options compatible with the OS X VPN client. And here are the Phase 2 options supported by strongSwan for Android:

Table 6: strongSwan for Android supported Phase 2 possible encryption and hash algorithm options.
PhaseEncryptionHash
Phase 2 AES-GCM-128 (128 bits)
AES-GCM-256 (128 bits)
ChaCha20/Poly1305
(none required)
AES-CBC-128 SHA256
AES-CBC-256 SHA384
AES-CBC-128
AES-CBC-192
AES-CBC-256
SHA1
SHA256
SHA384
SHA512

At that time, the Phase 2 configuration only included AES 256 with SHA1, which wasn’t acceptable to OS X – it requires either AES 256 with SHA256, or AES 128 with SHA1.

Since strongSwan for Android supports AES-GCM-128 and -256, I originally enabled AES-CBC-128 with SHA1 for OS X, and AES-GCM-128 (128 bits) for Android. Eighteen months later, SHA1 no longer seems like such a great idea, so I enabled AES-CBC-256 for OS X and changed from SHA1 to SHA256, leaving AES-GCM-128 (128 bits) for Android (if it chooses to use it).

Common error messages

Hopefully placing all of these errors in one place will help others to troubleshoot faster.

Table 7: Common OS X VPN client error messages and possible solutions.
Error messagePossible cause / resolution
IKEv2 Plugin: ikev2_resolve_server_name: failed to query DNS
IKEv2 Plugin: Connect: Attempt to query DNS failed
DNS resolution failed.
Try using the server IP address.
Failed to process IKE SA Init packet Phase 1 didn’t complete, possibly due to a Phase 1 options mismatch.
Ensure the server supports one of the Phase 1 configurations supported by your client.
Failed to create IKE Auth packet This message indicates that the client can’t even begin the authentication process.
Be sure to specify the server certificate’s CN in the Remote ID field.
eaptls_handshake(): SSLHandshake failed, errSSLPeerCertUnknown (-9829)
Failed to process IKE Auth (EAP) packet
The client didn’t present the server with a valid user certificate.
Be sure to specify the user certificate’s CN in the Local ID field.
Received error: Error (No Proposal Chosen)
Failed to process IKE Auth packet
A suitable Phase 2 encryption method and hash algorithm couldn’t be determined.
Ensure the server supports a Phase 2 configuration supported by your client.

Notice the subtle difference between the Failed... messages, each of which indicates the phase reached by the client.

Conclusion

The upshot: choosing Phase 1 and Phase 2 encryption, hash algorithms, and DH groups can be difficult due to the lack of documentation for the options that each client supports.

Once the server and client are communicating, I’ve found the IPsec IKEv2 VPN with certificate-based authentication (EAP-TLS) to be very fast & reliable. However, if you want to avoid some of this setup mess, OpenVPN is probably a good way to go.

References

  1. pfSense wiki: IKEv2 with EAP-TLS
  2. strongSwan: IKEv2 Cipher Suites
  3. strongSwan: Advanced Cipher Suite Examples
  4. Weak Diffie-Hellman and the Logjam Attack

Appendix

Windows 10 default encryption & hash algorithms; DH groups

To the surprise of no one, Windows 10 – by default – has the least flexbility of any of the three VPN clients reviewed here:

Table 8: Windows 10 default Phase 1 encryption, hash algorithm, and DH group options.
PhaseData encryptionEncryptionHashDH Group
Phase 1No encryption allowed
or
Optional encryption
3DESSHA12 (1024 bit)
3DESSHA2562 (1024 bit)
3DESSHA3842 (1024 bit)
AES-CBC-128SHA12 (1024 bit)
AES-CBC-128SHA2562 (1024 bit)
AES-CBC-128SHA3842 (1024 bit)
AES-CBC-192SHA12 (1024 bit)
AES-CBC-192SHA2562 (1024 bit)
AES-CBC-192SHA3842 (1024 bit)
AES-CBC-256SHA12 (1024 bit)
AES-CBC-256SHA2562 (1024 bit)
AES-CBC-256SHA3842 (1024 bit)
Phase 1Require encryption
or
Maximum strength encryption
3DESSHA12 (1024 bit)
AES-CBC-256SHA12 (1024 bit)
3DESSHA2562 (1024 bit)
AES-CBC-256SHA2562 (1024 bit)
3DESSHA3842 (1024 bit)
AES-CBC-256SHA3842 (1024 bit)

The jump between Optional encryption and Require encryption strikes me as very odd; it removes some AES ciphers, but leaves 3DES! The biggest disappointment, though, is that only DH group 2 (1024 bit) is enabled. In 2019, this is a rather poor state of affairs.

In Phase 2, there are a few different possibilities, depending on what is selected for the Data encryption field:

Table 9: Windows 10 default Phase 2 encryption and hash algorithm options.
PhaseData encryptionEncryptionHash
Phase 2No encryption allowedNULLSHA1
Optional encryptionAES-CBC-256SHA1
AES-CBC-128SHA1
3DESSHA1
DESSHA1
NULLSHA1
Require encryption
or
Maximum strength encryption
AES-CBC-256SHA1
3DESSHA1

Again, perfectly good ciphers disappear inexplicably between Optional encryption and Require encryption. And again, there is no difference between Require encryption and Maximum strength encryption.

Comparison of VPN client software capabilities

Now that the supported encryption & hash algorithms and DH groups has been enumerated for each client, here is a comparison chart:

Table 10: Comparison of VPN client software capabilities (partial).
PhaseEncryptionHashDH groupWindowsOS XAndroid
(strongSwan)
Optional*Require
Phase 1 AES 256SHA25614 (2048 bit) --
AES 256SHA2562 (1024 bit) -
AES 256SHA114 (2048 bit) ---
AES 256SHA12 (1024 bit) -
AES 128SHA25614 (2048 bit) ---
AES 128SHA2562 (1024 bit) --
AES 128SHA114 (2048 bit) ---
AES 128SHA12 (1024 bit) -
3DESSHA25614 (2048 bit) ---
3DESSHA2562 (1024 bit) -
3DESSHA114 (2048 bit) ---
3DESSHA12 (1024 bit)
Phase 2 AES 256SHA256 --
AES 256SHA1 -
AES 128SHA256 ---
AES 128SHA1 -
3DESSHA256 ----
3DESSHA1 -
* Optional = Optional encryption (connect even if no encryption)
Require = Require encryption (disconnect if server declines)

As you can see, very few options are simultaneously supported by all three clients – none of them any good. What to do?

Configuring Windows 10 IPsec settings using Set-VpnConnectionIPsecConfiguration

The article How to configure Diffie Hellman protocol over IKEv2 VPN connections led me to the Set-VpnConnectionIPsecConfiguration PowerShell cmdlet, which allows you to set the IPsec encryption parameters for a given IPsec configuration.

Setting up a VPN client in Windows 10 is beyond the scope of this document; however, I will provide the command I used and a brief explanation. (The documentation for the cmdlet itself is rather good, once you come to grips with the Microsoft-invented names for everything.)

Here is how I set the connection to use AES-GCM-128 (128 bits):

Set-VpnConnectionIPsecConfiguration -ConnectionName atom-vpn -AuthenticationTransformConstants GCMAES128 -CipherTransformConstants GCMAES128 -DHGroup Group14 -EncryptionMethod AES256 -IntegrityCheckMethod SHA256 -PfsGroup None -Force

And this will use AES 256:

Set-VpnConnectionIPsecConfiguration -ConnectionName atom-vpn -AuthenticationTransformConstants SHA256 -CipherTransformConstants AES256 -DHGroup Group14 -EncryptionMethod AES256 -IntegrityCheckMethod SHA256 -PfsGroup None -Force

Here follows a description of the options, translated from Microsoft-speak into more common terminology:

Table 11: Set-VpnConnectionIPsecConfiguration parameter description
PhaseSettingParameterExample
Phase 1Encryption algorithm-EncryptionMethodAES256
Hash algorithm-IntegrityCheckMethodSHA256
DH group-DHGroupGroup14
Phase 2Encryption algorithm-CipherTransformConstantsAES256
Hash algorithm-AuthenticationTransformConstantsSHA256
PFS group-PfsGroupNone

If you set CipherTransformConstants to a GCM algorithm, you must also set AuthenticationTransformConstants to match, otherwise you’ll get the error:

The IPsec cipher transform is not compatible with the policy.

Last update: 2019-02-16