IPsec VPN (IKEv2) with pfSense and OS X El Capitan

This note is meant to help troubleshoot OS X failing to connect to an IPsec VPN, particularly when using certificate-based authentication (EAP-TLS).

Table of contents:

Settings that work

There are two areas which require specific configuration:

  1. The VPN server encryption & hash algorithms, and
  2. The OS X VPN client settings.

Encryption & hash algorithms; DH groups

By default, OS X El Capitan supports a limited number of valid possible configurations. Here are the possible configurations for each phase, and which settings I’d recommend implementing on the VPN server. They are listed in the order they were sent by the client.

Table 1: OS X supported Phase 1 possible encryption, hash algorithm, and DH group options.
PhaseEncryptionHashDH GroupNotes
Phase 1AES-CBC-256SHA25614 (2048 bit) Recommended.
AES-CBC-256SHA25619 (nist ecp256)Could be used if you’re unconcerned about elliptic curves.
AES-CBC-256SHA2565 (1536 bit)DH group 5 (1536 bit) is unsupported by the strongSwan client for Android.
AES-CBC-128SHA12 (1024 bit) If the first option set is impractical, this is a good second choice.
3DESSHA12 (1024 bit) Not recommended (3DES encryption is too insecure).
Table 2: OS X supported Phase 2 possible encryption and hash algorithm options.
PhaseEncryptionHashNotes
Phase 2AES-CBC-256SHA256 Recommended when the highest degree of security is required.
AES-CBC-128SHA1 Recommended for most use cases.
3DESSHA1 Not recommended (3DES encryption is too insecure).

VPN server settings for OS X and Android (strongSwan) clients

In order to use these settings, here is what I implemented in pfSense:

Table 3: VPN server settings chosen for security, compatibility, and performance.
PhaseSettingsNotes
Phase 1Encryption algorithm: AES 256
Hash algorithm: SHA256
DH group: 14 (2048 bit)
Recommended.
Chosen to avoid DH group 2 (1024 bit).
Provides compatibility with strongSwan for Android.
Phase 2Encryption algorithm: AES 128
Hash algorithm: SHA1
Recommended.
I feel this is a good balance between security & performance.

OS X VPN client settings

Because I’m using certificate-based authentication (EAP-TLS), some of the settings which are typically left blank must be filled in with the correct values for the connection to be successful:

Table 4: OS X VPN client settings required for certificate-based authentication.
SettingExplanation
Server Address: host name or IP addressAs long as your server certificate contains SANs for both host name and IP address, you can use either here.
Remote ID: the CN of your server certificateThis must be the CN of your server certificate, usually its fully-qualified domain name.
Local ID: the CN of your user certificateThis must be the CN of your user certificate; in my case, this is just a username.

Remember to import your user certificate (with private key), import the CA certificate and any intermediate certificates, and mark the CA as trusted.

Background of my VPN setup and goals

I’m using the following server and client components:

Here are the goals for this setup in rough order of importance:

Initial VPN setup instructions

Initial setup is based on this documentation from the pfSense wiki:

This was tested and working well with my Nexus 6P (Android) but I hadn’t attempted to use it with OS X yet.

Troubleshooting OS X VPN client issues

Here are some issues I encountered and how I resolved them.

Obstacle 1: The OS X VPN client GUI provides no help whatsoever

The first thing you notice when using the OS X VPN client is that connection failures are completely silent. The only way you could tell that it failed was that the word “Connected” doesn’t appear. This is of course very unhelpful, and Apple should be a bit embarrassed to fail the user in this way.

My advice for troubleshooting the VPN connection is to keep the Console application open and visible, pointed at ‘system.log’.

Obstacle 2: The OS X ‘system.log’ file is full of red herrings

Take, for example, the following error message seen in ‘system.log’:

Jul 27 19:46:37 isomer nesessionmanager[12598]: Failed to find the VPN app for plugin type com.apple.neplugin.IKEv2

I have no idea what this means, but it doesn't seem to have any relevance to whether or not the connection was successful.

It’s not just the odd error message, either; there are sometimes full backtraces in ‘system.log’, too:

Jul 27 19:46:38 isomer neagent[12699]: BUG in libdispatch client: kevent[EVFILT_READ] delete: "Bad file descriptor" - 0x9
Jul 27 19:46:38 isomer kernel[0]: SIOCPROTODETACH_IN6: ipsec0 error=6
Jul 27 19:46:38 isomer symptomsd[369]: nw_interface_create_with_name netutil_ifname_to_ifindex(ipsec0) failed, dumping backtrace:
[...snip...]

None of this is helpful (at least, not to me).

Eventually I clued in that most of the interesting errors came from neagent. If you’re having trouble finding the ‘real’ error, try filtering by that.

Revelation 1: My VPN client can’t do DNS lookups

The first head-scratcher is that, for reasons not known to me, the VPN client wouldn’t do DNS lookups. The error seen in ‘system.log’ is:

Jul 27 19:32:57 isomer neagent[12599]: IKEv2 Plugin: ikev2_resolve_server_name: failed to query DNS
Jul 27 19:32:57 isomer neagent[12599]: IKEv2 Plugin: Connect: Attempt to query DNS failed

The fix here is simple: use the IP address for the server address.

Update (12-Sep-2017): Others have reported seeing this issue. I did some testing with Wireshark and noticed that, in these cases, no DNS query attempts were even made.

More testing revealed the server address field must be at least ten characters long. There’s also some other “mystery validation” happening here; my hostname is eleven characters and is still being rejected. If I modify the hostname slightly (swap a few characters), DNS resolution works. Probably this is a bug.

Revelation 2: The Phase 1 settings of the server and client must match

The next issue was more elusive. I had a hard time reading ‘system.log’, so I pulled out my favourite utility, tcpdump(1), on the client, and had a look.

fission@isomer[s008]:~% sudo tcpdump -nvvi en0 port 500
tcpdump: listening on en0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:41:29.971881 IP (tos 0x0, ttl 64, id 3664, offset 0, flags [none], proto UDP (17), length 632)
    192.168.43.5.500 > 203.0.113.10.500: [udp sum ok] isakmp 2.0 msgid 00000000 cookie 4b0bd8cb99e5f6ac->0000000000000000: parent_sa ikev2_init[I]:
    (sa: len=216
        (p: #1 protoid=isakmp transform=4 len=44
            (t: #1 type=encr id=aes (type=keylen value=0100))
            (t: #2 type=prf id=#5 )
            (t: #3 type=integ id=#12 )
            (t: #4 type=dh id=modp2048 ))
        (p: #2 protoid=isakmp transform=4 len=44
            (t: #1 type=encr id=aes (type=keylen value=0100))
            (t: #2 type=prf id=#5 )
            (t: #3 type=integ id=#12 )
            (t: #4 type=dh id=#19 ))
        (p: #3 protoid=isakmp transform=4 len=44
            (t: #1 type=encr id=aes (type=keylen value=0100))
            (t: #2 type=prf id=#5 )
            (t: #3 type=integ id=#12 )
            (t: #4 type=dh id=modp1536 ))
        (p: #4 protoid=isakmp transform=4 len=44
            (t: #1 type=encr id=aes (type=keylen value=0080))
            (t: #2 type=prf id=hmac-sha )
            (t: #3 type=integ id=hmac-sha )
            (t: #4 type=dh id=modp1024 ))
        (p: #5 protoid=isakmp transform=4 len=40
            (t: #1 type=encr id=3des )
            (t: #2 type=prf id=hmac-sha )
            (t: #3 type=integ id=hmac-sha )
            (t: #4 type=dh id=modp1024 )))
    (v2ke: len=256 group=modp2048)
    (nonce: len=16 nonce=(5195098add8e0e74f6bb7ff30ae40d72) )
    (n: prot_id=#0 type=16406(status))
    (n: prot_id=#0 type=16388(nat_detection_source_ip))
    (n: prot_id=#0 type=16389(nat_detection_destination_ip))
    (n: prot_id=#0 type=16430(status))
19:41:30.923281 IP (tos 0x0, ttl 56, id 40183, offset 0, flags [none], proto UDP (17), length 64)
    203.0.113.10.500 > 192.168.43.5.500: [udp sum ok] isakmp 2.0 msgid 00000000 cookie 4b0bd8cb99e5f6ac->0000000000000000: parent_sa ikev2_init[R]:
    (n: prot_id=#0 type=14(no_protocol_chosen))

Interesting – the client has offered a series of possible encryption types, hash algorithms, and DH groups, but the server has responded with no_protocol_chosen. This is good information, as it indicates that things went off the rails early on.

This helped me to find the client error message in ‘system.log’:

Jul 27 19:41:31 isomer neagent[12638]: Failed to process IKE SA Init packet

The server logs for that time period say:

Jul 27 19:41:30	charon	02[CFG] <44> received proposals: IKE:AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048, IKE:AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/ECP_256, IKE:AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_1536, IKE:AES_CBC_128/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024, IKE:3DES_CBC/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024
Jul 27 19:41:30	charon	02[CFG] <44> configured proposals: IKE:AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_1024
Jul 27 19:41:30	charon	02[IKE] <44> received proposals inacceptable

Table 1 above has a breakdown of the various Phase 1 options supported by the OS X VPN client, which did not match what the server was offering: the server offered only AES 256 with SHA256 and DH group 2 (1024 bit). However, that was not a configuration the OS X VPN client supported.

To fix this, I had to find a set of Phase 1 options which were compatible with both OS X and strongSwan for Android. Here are the Phase 1 possibilities for strongSwan for Android:

Table 5: strongSwan for Android supported Phase 1 possible encryption, hash algorithm, and DH group options.
PhaseEncryptionHashDH Group
Phase 1 AES-CBC-128
AES-CBC-192
AES-CBC-256
3DES
SHA256
SHA384
SHA512
MD5
SHA1
AES-XCBC
19 (nist ecp256)
20 (nist ecp384)
21 (nist ecp521)
28 (brainpool ecp256)
29 (brainpool ecp384)
30 (brainpool ecp512)
31 (curve25519)
15 (3072 bit)
16 (4096 bit)
18 (8192 bit)
14 (2048 bit)
2 (1024 bit)
AES-GCM-128 (128 bits)
AES-GCM-192 (128 bits)
AES-GCM-256 (128 bits)
ChaCha20/Poly1305
AES-GCM-128 (96 bits)
AES-GCM-192 (96 bits)
AES-GCM-256 (96 bits)
AES-GCM-128 (64 bits)
AES-GCM-192 (64 bits)
AES-GCM-256 (64 bits)
SHA256
SHA384
SHA512
MD5
SHA1
AES-XCBC
19 (nist ecp256)
20 (nist ecp384)
21 (nist ecp521)
28 (brainpool ecp256)
29 (brainpool ecp384)
30 (brainpool ecp512)
31 (curve25519)
15 (3072 bit)
16 (4096 bit)
18 (8192 bit)
14 (2048 bit)
2 (1024 bit)

Interestingly, the strongSwan for Android client supports every single Phase 1 configuration that the OS X VPN client supports, except DH group 5 (1536 bit).

The solution here is to change the VPN server to use DH group 14 (2048 bit), which creates an option set compatible with both clients. Other valid – though suboptimal – choices would have been to choose DH group 19 (nist ecp256), or to choose AES 128 encryption.

Revelation 3: The remote ID and local ID must match the certificates

Most OS X VPN guides I found on the Internet are geared for username & password-based authentication; I didn’t yet see one that really explains the validation process for certificates.

As mentioned in table 4, both the server and client certificate CNs are required to match the remote ID and local ID fields, respectively, in the VPN client configuration. Initially I had the remote ID set correctly, but I didn’t realize I needed to set the local ID as well. This got me the following error in ‘system.log’:

Jul 28 22:53:07 isomer neagent[14549]: [eaptls_plugin.c:397] eaptls_handshake(): SSLHandshake failed, errSSLPeerCertUnknown (-9829)
Jul 28 22:53:07 isomer neagent[14549]: Failed to process IKE Auth (EAP) packet

On the server side, the errors look like this:

Jul 27 23:03:44	charon	05[TLS] <con1|66> no trusted certificate found for '192.168.43.5' to verify TLS peer
Jul 27 23:03:44	charon	05[TLS] <con1|66> sending fatal TLS alert 'certificate unknown'
[...snip...]
Jul 27 23:03:44	charon	05[IKE] <con1|66> EAP method EAP_TLS failed for peer 192.168.43.5

The solution is to specify the user certificate’s CN for the local ID field in the VPN configuration.

Coincidentally, if you fail to specify the server CN for the remote ID field, you get this:

Jul 28 22:55:03 isomer neagent[14562]: Failed to create IKE Auth packet

The server sees a timeout, since the OS X VPN client simply gives up. Notice the thirty second gap between log entries:

Jul 28 22:55:03	charon	10[NET] <77> sending packet: from 203.0.113.10[500] to 192.168.43.5[500] (493 bytes)
Jul 28 22:55:33	charon	10[JOB] <77> deleting half open IKE_SA after timeout

Revelation 4: The Phase 2 settings of the server and client must match

After these three fixes, I saw this in ‘system.log’:

Jul 27 23:08:07 isomer neagent[12983]: Received error: Error (No Proposal Chosen)
Jul 27 23:08:07 isomer neagent[12983]: Failed to process IKE Auth packet

And on the server side:

Jul 27 23:08:07	charon	12[CFG] <con1|67> received proposals: ESP:AES_CBC_256/HMAC_SHA2_256_128/NO_EXT_SEQ, ESP:AES_CBC_256/HMAC_SHA2_256_128/NO_EXT_SEQ, ESP:AES_CBC_256/HMAC_SHA2_256_128/NO_EXT_SEQ, ESP:AES_CBC_128/HMAC_SHA1_96/NO_EXT_SEQ, ESP:3DES_CBC/HMAC_SHA1_96/NO_EXT_SEQ
Jul 27 23:08:07	charon	12[CFG] <con1|67> configured proposals: ESP:AES_CBC_256/HMAC_SHA1_96/NO_EXT_SEQ, ESP:AES_CBC_256/HMAC_SHA1_96/NO_EXT_SEQ
Jul 27 23:08:07	charon	12[IKE] <con1|67> no acceptable proposal found
Jul 27 23:08:07	charon	12[IKE] <con1|67> failed to establish CHILD_SA, keeping IKE_SA

This looks a lot like the Phase 1 failure, except the proposals now start with ESP: instead of IKE:, indicating that we are attempting to establish an ESP tunnel. Table 2 shows the Phase 2 options compatible with the OS X VPN client. And here are the Phase 2 options supported by strongSwan for Android:

Table 6: strongSwan for Android supported Phase 2 possible encryption and hash algorithm options.
PhaseEncryptionHash
Phase 2 AES-GCM-128 (128 bits)
AES-GCM-256 (128 bits)
ChaCha20/Poly1305
(none required)
AES-CBC-128 SHA256
AES-CBC-256 SHA384
AES-CBC-128
AES-CBC-192
AES-CBC-256
SHA1
SHA256
SHA384
SHA512

Because a single VPN server can offer multiple Phase 2 encryption methods, there are a few potential solutions. One solution is for me to change the Phase 2 encryption algorithm from AES 256 to AES (auto) to allow both AES 128 and AES 256. If I had wanted to force AES 256, I could have instead enabled the hash algorithm SHA256.

Since strongSwan for Android supports AES-GCM-128 and -256, another option is to switch to AES-CBC-128 for OS X, and enable AES-GCM-128 for Android. This is the setup I used in the end.

Common error messages

Hopefully placing all of these errors in one place will help others to troubleshoot faster.

Table 7: Common OS X VPN client error messages and possible solutions.
Error messagePossible cause / resolution
IKEv2 Plugin: ikev2_resolve_server_name: failed to query DNS
IKEv2 Plugin: Connect: Attempt to query DNS failed
DNS resolution failed.
Try using the server IP address.
Failed to process IKE SA Init packet Phase 1 didn’t complete, possibly due to a Phase 1 options mismatch.
Ensure the server supports one of the Phase 1 configurations supported by your client.
Failed to create IKE Auth packet This message indicates that the client can’t even begin the authentication process.
Be sure to specify the server certificate’s CN in the Remote ID field.
eaptls_handshake(): SSLHandshake failed, errSSLPeerCertUnknown (-9829)
Failed to process IKE Auth (EAP) packet
The client didn’t present the server with a valid user certificate.
Be sure to specify the user certificate’s CN in the Local ID field.
Received error: Error (No Proposal Chosen)
Failed to process IKE Auth packet
A suitable Phase 2 encryption method and hash algorithm couldn’t be determined.
Ensure the server supports a Phase 2 configuration supported by your client.

Notice the subtle difference between the Failed... messages, each of which indicates the phase reached by the client.

Conclusion

The upshot: choosing Phase 1 and Phase 2 encryption, hash algorithms, and DH groups can be difficult due to the lack of documentation for the options that each client supports.

Once the server and client are communicating, I’ve found the IPsec IKEv2 VPN with certificate-based authentication (EAP-TLS) to be very fast & reliable. However, if you want to avoid some of this setup mess, OpenVPN is probably a good way to go.

References

  1. pfSense wiki: IKEv2 with EAP-TLS
  2. strongSwan: IKEv2 Cipher Suites
  3. strongSwan: Advanced Cipher Suite Examples
  4. Weak Diffie-Hellman and the Logjam Attack

Appendix

Windows 10 encryption & hash algorithms; DH groups

To the surprise of no one, Windows 10 has the least flexbility of any of the three VPN clients reviewed here:

Table 8: Windows 10 supported Phase 1 possible encryption, hash algorithm, and DH group options.
PhaseData encryptionEncryptionHashDH Group
Phase 1No encryption allowed
or
Optional encryption
3DESSHA12 (1024 bit)
3DESSHA2562 (1024 bit)
3DESSHA3842 (1024 bit)
AES-CBC-128SHA12 (1024 bit)
AES-CBC-128SHA2562 (1024 bit)
AES-CBC-128SHA3842 (1024 bit)
AES-CBC-192SHA12 (1024 bit)
AES-CBC-192SHA2562 (1024 bit)
AES-CBC-192SHA3842 (1024 bit)
AES-CBC-256SHA12 (1024 bit)
AES-CBC-256SHA2562 (1024 bit)
AES-CBC-256SHA3842 (1024 bit)
Phase 1Require encryption
or
Maximum strength encryption
3DESSHA12 (1024 bit)
AES-CBC-256SHA12 (1024 bit)
3DESSHA2562 (1024 bit)
AES-CBC-256SHA2562 (1024 bit)
3DESSHA3842 (1024 bit)
AES-CBC-256SHA3842 (1024 bit)

The jump between Optional encryption and Require encryption strikes me as very odd; it removes some AES ciphers, but leaves 3DES! The biggest disappointment, though, is that only DH group 2 (1024 bit) is supported. What a pity.

In Phase 2, there are a few different possibilities, depending on what is selected for the Data encryption field:

Table 9: Windows 10 supported Phase 2 possible encryption and hash algorithm options.
PhaseData encryptionEncryptionHash
Phase 2No encryption allowedNULLSHA1
Optional encryptionAES-CBC-256SHA1
AES-CBC-128SHA1
3DESSHA1
DESSHA1
NULLSHA1
Require encryption
or
Maximum strength encryption
AES-CBC-256SHA1
3DESSHA1

Again, perfectly good ciphers disappear inexplicably between Optional encryption and Require encryption. And again, there is no difference between Require encryption and Maximum strength encryption.

Comparison of VPN client software capabilities

Now that the supported encryption & hash algorithms and DH groups has been enumerated for each client, here is a comparison chart:

Table 10: Comparison of VPN client software capabilities (partial).
PhaseEncryptionHashDH groupWindowsOS XAndroid
(strongSwan)
Optional*Require
Phase 1 AES 256SHA25614 (2048 bit) --
AES 256SHA2562 (1024 bit) -
AES 256SHA114 (2048 bit) ---
AES 256SHA12 (1024 bit) -
AES 128SHA25614 (2048 bit) ---
AES 128SHA2562 (1024 bit) --
AES 128SHA114 (2048 bit) ---
AES 128SHA12 (1024 bit) -
3DESSHA25614 (2048 bit) ---
3DESSHA2562 (1024 bit) -
3DESSHA114 (2048 bit) ---
3DESSHA12 (1024 bit)
Phase 2 AES 256SHA256 --
AES 256SHA1 -
AES 128SHA256 ---
AES 128SHA1 -
3DESSHA256 ----
3DESSHA1 -
* Optional = Optional encryption (connect even if no encryption)
Require = Require encryption (disconnect if server declines)

As you can see, very few options are simultaneously supported by all three clients.

VPN server settings for OS X, Android (strongSwan), and Windows 10 clients

In the end, I decided to create a separate VPN server instance for Windows 10, as I wanted to avoid 3DES and use a stronger DH group than group 2 (1024 bit). But if you wanted to support OS X, strongSwan for Android, and Windows 10 all in the same VPN server, here are the settings you’d have to use:

Table 11: VPN server settings compatible with OS X, Android (strongSwan), and Windows 10.
PhaseSettingsNotes
Phase 1Encryption algorithm: 3DES
Hash algorithm: SHA1
DH group: 2 (1024 bit)
Not recommended (3DES encryption is too insecure).
Suggestion: create a separate VPN server for Windows clients with AES 256 in Phase 1.
Phase 2Encryption algorithm: AES (auto)
Hash algorithm: SHA1
or
Encryption algorithm: AES 256
Hash algorithm: SHA1 and SHA256
There is no one encryption and hash algorithm pair that works universally,
unless you use Optional encryption in Windows 10, which seems like a bad idea.

You either have to allow AES 128 (OS X) and 256 (Windows 10) with SHA1,
or AES 256 with SHA1 (Windows 10) and SHA256 (OS X).

If you decide to create a separate VPN server instance for Windows 10 clients, here are the settings I’d suggest:

Table 12: VPN server settings targeting Windows 10 clients.
PhaseSettingsNotes
Phase 1Encryption algorithm: AES 256
Hash algorithm: SHA256
DH group: 2 (1024 bit)
DH group 2 (1024 bit) is suboptimal, but it’s the only supported option.
This is also compatible with strongSwan for Android (but not OS X).
Phase 2Encryption algorithm: AES 256
Hash algorithm: SHA1
If you plan to use this VPN server with other client types,
you could include other encryption and hash algorithms, too.