IPsec VPN (IKEv2) with pfSense and OS X El Capitan

Table of contents:

There are two areas which require specific configuration:

1. The VPN server encryption & hash algorithms, and
2. The OS X VPN client settings.

By default, OS X El Capitan supports a limited number of valid possible configurations. Here are the possible configurations for each phase, and which settings I’d recommend implementing on the VPN server. They are listed in the order they were sent by the client.

In order to use these settings, here is what I implemented in pfSense:

Because I’m using certificate-based authentication (EAP-TLS), some of the settings which are typically left blank must be filled in with the correct values for the connection to be successful:

Remember to import your user certificate (with private key), import the CA certificate and any intermediate certificates, and mark the CA as trusted.

I’m using the following server and client components:

• VPN server: pfSense 2.3.4 running strongSwan 5.5.1
• VPN client 1: Nexus 6P running Android Nougat 7.1.2
• VPN client 2: MacBook Pro (Late 2013) running OS X El Capitan 10.11.6

Here are the goals for this setup in rough order of importance:

• Use strong encryption & hash algorithms. After reading some documents about how weak the 1024-bit DH group is, it makes sense to switch to a 2048-bit group. Unfortunately the NSA’s involvement with elliptic curves makes me want to avoid them for the time being.
• Tunnel all traffic. Jim Pingle, a pfSense developer, says: “The [IKEv2] client behaviour [for split tunnelling] differs a lot. Some clients will honour the request; some won’t. All OpenVPN [clients] will work just fine with split tunnelling, but IPsec is inconsistent. Windows native IKEv2 client will just tunnel everything no matter what you try to do; iOS and OS X will honour it; for Android, it depends on the type of client. It’s best to just plan on tunnelling everything across the VPN if you can.”
• Authenticate using certificates. Certificate-based authentication (EAP-TLS) in VPN seems analogous to key-based authentication in SSH; a bit better security than ol’ username & password.
• Use only one server configuration. Although I have a couple of publicly-routable IP addresses, I didn’t want to stand up multiple IPsec VPN configurations; I wanted to find settings that would work the same with every client.

Initial setup is based on this documentation from the pfSense wiki:

• IKEv2 with EAP-TLS

This was tested and working well with my Nexus 6P (Android) but I hadn’t attempted to use it with OS X yet.

Here are some issues I encountered and how I resolved them.

The first thing you notice when using the OS X VPN client is that connection failures are completely silent. The only way you could tell that it failed was that the word “Connected” doesn’t appear. This is of course very unhelpful, and Apple should be a bit embarrassed to fail the user in this way.

My advice for troubleshooting the VPN connection is to keep the Console application open and visible, pointed at ‘system.log’.

Take, for example, the following error message seen in ‘system.log’:

Jul 27 19:46:37 isomer nesessionmanager[12598]: Failed to find the VPN app for plugin type com.apple.neplugin.IKEv2

I have no idea what this means, but it doesn't seem to have any relevance to whether or not the connection was successful.

It’s not just the odd error message, either; there are sometimes full backtraces in ‘system.log’, too:

Jul 27 19:46:38 isomer neagent[12699]: BUG in libdispatch client: kevent[EVFILT_READ] delete: "Bad file descriptor" - 0x9
Jul 27 19:46:38 isomer kernel[0]: SIOCPROTODETACH_IN6: ipsec0 error=6
Jul 27 19:46:38 isomer symptomsd[369]: nw_interface_create_with_name netutil_ifname_to_ifindex(ipsec0) failed, dumping backtrace:
[...snip...]

None of this is helpful (at least, not to me).

Eventually I clued in that most of the interesting errors came from neagent. If you’re having trouble finding the ‘real’ error, try filtering by that.

The first head-scratcher is that, for reasons not known to me, the VPN client wouldn’t do DNS lookups. The error seen in ‘system.log’ is:

Jul 27 19:32:57 isomer neagent[12599]: IKEv2 Plugin: ikev2_resolve_server_name: failed to query DNS
Jul 27 19:32:57 isomer neagent[12599]: IKEv2 Plugin: Connect: Attempt to query DNS failed

The fix here is simple: use the IP address for the server address.

Update (12-Sep-2017): Others have reported seeing this issue. I did some testing with Wireshark and noticed that, in these cases, no DNS query attempts were even made.

More testing revealed the server address field must be at least ten characters long. There’s also some other “mystery validation” happening here; my hostname is eleven characters and is still being rejected. If I modify the hostname slightly (swap a few characters), DNS resolution works. Probably this is a bug.

The next issue was more elusive. I had a hard time reading ‘system.log’, so I pulled out my favourite utility, tcpdump(1), on the client, and had a look.

fission@isomer[s008]:~% sudo tcpdump -nvvi en0 port 500
tcpdump: listening on en0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:41:29.971881 IP (tos 0x0, ttl 64, id 3664, offset 0, flags [none], proto UDP (17), length 632)
    192.168.43.5.500 > 203.0.113.10.500: [udp sum ok] isakmp 2.0 msgid 00000000 cookie 4b0bd8cb99e5f6ac->0000000000000000: parent_sa ikev2_init[I]:
    (sa: len=216
        (p: #1 protoid=isakmp transform=4 len=44
            (t: #1 type=encr id=aes (type=keylen value=0100))
            (t: #2 type=prf id=#5 )
            (t: #3 type=integ id=#12 )
            (t: #4 type=dh id=modp2048 ))
        (p: #2 protoid=isakmp transform=4 len=44
            (t: #1 type=encr id=aes (type=keylen value=0100))
            (t: #2 type=prf id=#5 )
            (t: #3 type=integ id=#12 )
            (t: #4 type=dh id=#19 ))
        (p: #3 protoid=isakmp transform=4 len=44
            (t: #1 type=encr id=aes (type=keylen value=0100))
            (t: #2 type=prf id=#5 )
            (t: #3 type=integ id=#12 )
            (t: #4 type=dh id=modp1536 ))
        (p: #4 protoid=isakmp transform=4 len=44
            (t: #1 type=encr id=aes (type=keylen value=0080))
            (t: #2 type=prf id=hmac-sha )
            (t: #3 type=integ id=hmac-sha )
            (t: #4 type=dh id=modp1024 ))
        (p: #5 protoid=isakmp transform=4 len=40
            (t: #1 type=encr id=3des )
            (t: #2 type=prf id=hmac-sha )
            (t: #3 type=integ id=hmac-sha )
            (t: #4 type=dh id=modp1024 )))
    (v2ke: len=256 group=modp2048)
    (nonce: len=16 nonce=(5195098add8e0e74f6bb7ff30ae40d72) )
    (n: prot_id=#0 type=16406(status))
    (n: prot_id=#0 type=16388(nat_detection_source_ip))
    (n: prot_id=#0 type=16389(nat_detection_destination_ip))
    (n: prot_id=#0 type=16430(status))
19:41:30.923281 IP (tos 0x0, ttl 56, id 40183, offset 0, flags [none], proto UDP (17), length 64)
    203.0.113.10.500 > 192.168.43.5.500: [udp sum ok] isakmp 2.0 msgid 00000000 cookie 4b0bd8cb99e5f6ac->0000000000000000: parent_sa ikev2_init[R]:
    (n: prot_id=#0 type=14(no_protocol_chosen))

Interesting – the client has offered a series of possible encryption types, hash algorithms, and DH groups, but the server has responded with no_protocol_chosen. This is good information, as it indicates that things went off the rails early on.

This helped me to find the client error message in ‘system.log’:

Jul 27 19:41:31 isomer neagent[12638]: Failed to process IKE SA Init packet

The server logs for that time period say:

Jul 27 19:41:30	charon	02[CFG] <44> received proposals: IKE:AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048, IKE:AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/ECP_256, IKE:AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_1536, IKE:AES_CBC_128/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024, IKE:3DES_CBC/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024
Jul 27 19:41:30	charon	02[CFG] <44> configured proposals: IKE:AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_1024
Jul 27 19:41:30	charon	02[IKE] <44> received proposals inacceptable

Table 1 above has a breakdown of the various Phase 1 options supported by the OS X VPN client, which did not match what the server was offering: at that time, the Phase 1 configuration only included AES 256 with SHA256 and DH group 2 (1024 bit). However, that was not a configuration the OS X VPN client supported.

To fix this, I had to find a set of Phase 1 options which were compatible with both OS X and strongSwan for Android. Here are the Phase 1 possibilities for strongSwan for Android:

Interestingly, the strongSwan for Android client supports every single Phase 1 configuration that the OS X VPN client supports, except DH group 5 (1536 bit).

The solution here is to change the VPN server to use DH group 14 (2048 bit), which creates an option set compatible with both clients. Other valid – though suboptimal – choices would have been to choose DH group 19 (nist ecp256), or to choose AES 128 encryption.

Most OS X VPN guides I found on the Internet are geared for username & password-based authentication; I didn’t yet see one that really explains the validation process for certificates.

As mentioned in table 4, both the server and client certificate CNs are required to match the remote ID and local ID fields, respectively, in the VPN client configuration. Initially I had the remote ID set correctly, but I didn’t realize I needed to set the local ID as well. This got me the following error in ‘system.log’:

Jul 28 22:53:07 isomer neagent[14549]: [eaptls_plugin.c:397] eaptls_handshake(): SSLHandshake failed, errSSLPeerCertUnknown (-9829)
Jul 28 22:53:07 isomer neagent[14549]: Failed to process IKE Auth (EAP) packet

On the server side, the errors look like this:

Jul 27 23:03:44	charon	05[TLS] <con1|66> no trusted certificate found for '192.168.43.5' to verify TLS peer
Jul 27 23:03:44	charon	05[TLS] <con1|66> sending fatal TLS alert 'certificate unknown'
[...snip...]
Jul 27 23:03:44	charon	05[IKE] <con1|66> EAP method EAP_TLS failed for peer 192.168.43.5

The solution is to specify the user certificate’s CN for the local ID field in the VPN configuration.

Coincidentally, if you fail to specify the server CN for the remote ID field, you get this:

Jul 28 22:55:03 isomer neagent[14562]: Failed to create IKE Auth packet

The server sees a timeout, since the OS X VPN client simply gives up. Notice the thirty second gap between log entries:

Jul 28 22:55:03	charon	10[NET] <77> sending packet: from 203.0.113.10[500] to 192.168.43.5[500] (493 bytes)
Jul 28 22:55:33	charon	10[JOB] <77> deleting half open IKE_SA after timeout

After these three fixes, I saw this in ‘system.log’:

Jul 27 23:08:07 isomer neagent[12983]: Received error: Error (No Proposal Chosen)
Jul 27 23:08:07 isomer neagent[12983]: Failed to process IKE Auth packet

And on the server side:

Jul 27 23:08:07	charon	12[CFG] <con1|67> received proposals: ESP:AES_CBC_256/HMAC_SHA2_256_128/NO_EXT_SEQ, ESP:AES_CBC_256/HMAC_SHA2_256_128/NO_EXT_SEQ, ESP:AES_CBC_256/HMAC_SHA2_256_128/NO_EXT_SEQ, ESP:AES_CBC_128/HMAC_SHA1_96/NO_EXT_SEQ, ESP:3DES_CBC/HMAC_SHA1_96/NO_EXT_SEQ
Jul 27 23:08:07	charon	12[CFG] <con1|67> configured proposals: ESP:AES_CBC_256/HMAC_SHA1_96/NO_EXT_SEQ, ESP:AES_CBC_256/HMAC_SHA1_96/NO_EXT_SEQ
Jul 27 23:08:07	charon	12[IKE] <con1|67> no acceptable proposal found
Jul 27 23:08:07	charon	12[IKE] <con1|67> failed to establish CHILD_SA, keeping IKE_SA

This looks a lot like the Phase 1 failure, except the proposals now start with ESP: instead of IKE:, indicating that we are attempting to establish an ESP tunnel. Table 2 shows the Phase 2 options compatible with the OS X VPN client. And here are the Phase 2 options supported by strongSwan for Android:

At that time, the Phase 2 configuration only included AES 256 with SHA1, which wasn’t acceptable to OS X – it requires either AES 256 with SHA256, or AES 128 with SHA1.

Since strongSwan for Android supports AES-GCM-128 and -256, I originally enabled AES-CBC-128 with SHA1 for OS X, and AES-GCM-128 (128 bits) for Android. Eighteen months later, SHA1 no longer seems like such a great idea, so I enabled AES-CBC-256 for OS X and changed from SHA1 to SHA256, leaving AES-GCM-128 (128 bits) for Android (if it chooses to use it).

Hopefully placing all of these errors in one place will help others to troubleshoot faster.

Notice the subtle difference between the Failed... messages, each of which indicates the phase reached by the client.

The upshot: choosing Phase 1 and Phase 2 encryption, hash algorithms, and DH groups can be difficult due to the lack of documentation for the options that each client supports.

Once the server and client are communicating, I’ve found the IPsec IKEv2 VPN with certificate-based authentication (EAP-TLS) to be very fast & reliable. However, if you want to avoid some of this setup mess, OpenVPN is probably a good way to go.

1. pfSense wiki: IKEv2 with EAP-TLS
2. strongSwan: IKEv2 Cipher Suites
3. strongSwan: Advanced Cipher Suite Examples
4. Weak Diffie-Hellman and the Logjam Attack

To the surprise of no one, Windows 10 – by default – has the least flexbility of any of the three VPN clients reviewed here:

The jump between Optional encryption and Require encryption strikes me as very odd; it removes some AES ciphers, but leaves 3DES! The biggest disappointment, though, is that only DH group 2 (1024 bit) is enabled. In 2019, this is a rather poor state of affairs.

In Phase 2, there are a few different possibilities, depending on what is selected for the Data encryption field:

Again, perfectly good ciphers disappear inexplicably between Optional encryption and Require encryption. And again, there is no difference between Require encryption and Maximum strength encryption.

Now that the supported encryption & hash algorithms and DH groups has been enumerated for each client, here is a comparison chart:

As you can see, very few options are simultaneously supported by all three clients – none of them any good. What to do?

The article How to configure Diffie Hellman protocol over IKEv2 VPN connections led me to the Set-VpnConnectionIPsecConfiguration PowerShell cmdlet, which allows you to set the IPsec encryption parameters for a given IPsec configuration.

Setting up a VPN client in Windows 10 is beyond the scope of this document; however, I will provide the command I used and a brief explanation. (The documentation for the cmdlet itself is rather good, once you come to grips with the Microsoft-invented names for everything.)

Here is how I set the connection to use AES-GCM-128 (128 bits):

Set-VpnConnectionIPsecConfiguration -ConnectionName atom-vpn -AuthenticationTransformConstants GCMAES128 -CipherTransformConstants GCMAES128 -DHGroup Group14 -EncryptionMethod AES256 -IntegrityCheckMethod SHA256 -PfsGroup None -Force

And this will use AES 256:

Set-VpnConnectionIPsecConfiguration -ConnectionName atom-vpn -AuthenticationTransformConstants SHA256 -CipherTransformConstants AES256 -DHGroup Group14 -EncryptionMethod AES256 -IntegrityCheckMethod SHA256 -PfsGroup None -Force

Here follows a description of the options, translated from Microsoft-speak into more common terminology:

If you set CipherTransformConstants to a GCM algorithm, you must also set AuthenticationTransformConstants to match, otherwise you’ll get the error:

The IPsec cipher transform is not compatible with the policy.

Feel free to contact me with any questions, comments, or feedback.