VyOS DMVPN AWS

Received some feedback from a reader concerning VyOS DMVPN on an instance hosted at Amazon Web Services (AWS).

Syed wrote:  “Thanks for your post. Is the VPN IPSEC supported tested/verified behind NAT? NHRP is working fine, but not VPN IPSEC.”

I suggested that in a NAT situation, like you’d find at AWS, we might have to alter the tun0 local-ip setting to reflect the public ip address, rather than the eth0 address…

He responded: “No, the local-ip should be private, and remote ip will be public. Private ip should be translated to public when it traverses the 1:1 Nat or Default Gateway of AWS VPC, as I have tested the same setup with other cisco device and the same scenario is working perfectly. I believe it could be due to an IPSEC bug behind NAT – not sure we support IPSEC behind NAT on VyOS.”

It was an interesting enough topic that I decided to spin up three AWS instances today to see if I could get it working.  Here’s the topology:

VyOS DMVPN AWS
VyOS DMVPN AWS

I have created and applied a Security Group that allows all traffic to and from the public addresses of each of these hosts, and they are successfully able to reach each other.

Hub Config:

set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth0 duplex 'auto'
set interfaces ethernet eth0 hw-id '12:2c:2b:63:37:fe'
set interfaces ethernet eth0 smp_affinity 'auto'
set interfaces ethernet eth0 speed 'auto'
set interfaces loopback lo address '10.0.1.1/24'
set interfaces tunnel tun0 address '10.0.0.1/24'
set interfaces tunnel tun0 encapsulation 'gre'
set interfaces tunnel tun0 local-ip '172.31.48.229'
set interfaces tunnel tun0 multicast 'enable'
set interfaces tunnel tun0 parameters ip key '1'
set protocols bgp 65000 neighbor 10.0.0.11 peer-group 'DMVPNPEERS'
set protocols bgp 65000 neighbor 10.0.0.12 peer-group 'DMVPNPEERS'
set protocols bgp 65000 network '10.0.1.0/24'
set protocols bgp 65000 parameters router-id '10.0.0.1'
set protocols bgp 65000 peer-group DMVPNPEERS 'passive'
set protocols bgp 65000 peer-group DMVPNPEERS remote-as '65000'
set protocols bgp 65000 peer-group DMVPNPEERS 'route-reflector-client'
set protocols bgp 65000 peer-group DMVPNPEERS soft-reconfiguration 'inbound'
set protocols bgp 65000 peer-group DMVPNPEERS update-source '10.0.0.1'
set protocols nhrp tunnel tun0 cisco-authentication 'SECRET'
set protocols nhrp tunnel tun0 holding-time '300'
set protocols nhrp tunnel tun0 multicast 'dynamic'
set protocols nhrp tunnel tun0 'redirect'
set vpn ipsec esp-group ESP-HUB compression 'disable'
set vpn ipsec esp-group ESP-HUB lifetime '1800'
set vpn ipsec esp-group ESP-HUB mode 'tunnel'
set vpn ipsec esp-group ESP-HUB pfs 'dh-group2'
set vpn ipsec esp-group ESP-HUB proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-HUB proposal 1 hash 'sha1'
set vpn ipsec ike-group IKE-HUB key-exchange 'ikev1'
set vpn ipsec ike-group IKE-HUB lifetime '3600'
set vpn ipsec ike-group IKE-HUB proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-HUB proposal 1 hash 'sha1'
set vpn ipsec ipsec-interfaces interface 'tun0'
set vpn ipsec profile DMVPN authentication mode 'pre-shared-secret'
set vpn ipsec profile DMVPN authentication pre-shared-secret 'SECRET'
set vpn ipsec profile DMVPN bind tunnel 'tun0'
set vpn ipsec profile DMVPN esp-group 'ESP-HUB'
set vpn ipsec profile DMVPN ike-group 'IKE-HUB'

Spoke1 Config:

set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth0 duplex 'auto'
set interfaces ethernet eth0 hw-id '12:d8:e9:00:ac:a2'
set interfaces ethernet eth0 smp_affinity 'auto'
set interfaces ethernet eth0 speed 'auto'
set interfaces loopback lo address '10.0.2.1/24'
set interfaces tunnel tun0 address '10.0.0.11/24'
set interfaces tunnel tun0 encapsulation 'gre'
set interfaces tunnel tun0 local-ip '172.31.55.24'
set interfaces tunnel tun0 multicast 'enable'
set interfaces tunnel tun0 parameters ip key '1'
set protocols bgp 65000 neighbor 10.0.0.1 remote-as '65000'
set protocols bgp 65000 neighbor 10.0.0.1 soft-reconfiguration 'inbound'
set protocols bgp 65000 neighbor 10.0.0.1 timers holdtime '15'
set protocols bgp 65000 neighbor 10.0.0.1 timers keepalive '5'
set protocols bgp 65000 neighbor 10.0.0.1 update-source '10.0.0.11'
set protocols bgp 65000 network '10.0.2.0/24'
set protocols bgp 65000 parameters router-id '10.0.0.11'
set protocols nhrp tunnel tun0 cisco-authentication 'SECRET'
set protocols nhrp tunnel tun0 map 10.0.0.1/24 nbma-address '54.172.39.26'
set protocols nhrp tunnel tun0 map 10.0.0.1/24 'register'
set protocols nhrp tunnel tun0 multicast 'nhs'
set protocols nhrp tunnel tun0 'redirect'
set protocols nhrp tunnel tun0 'shortcut'
set vpn ipsec esp-group ESP-SPOKE compression 'disable'
set vpn ipsec esp-group ESP-SPOKE lifetime '1800'
set vpn ipsec esp-group ESP-SPOKE mode 'tunnel'
set vpn ipsec esp-group ESP-SPOKE pfs 'dh-group2'
set vpn ipsec esp-group ESP-SPOKE proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-SPOKE proposal 1 hash 'sha1'
set vpn ipsec ike-group IKE-SPOKE key-exchange 'ikev1'
set vpn ipsec ike-group IKE-SPOKE lifetime '3600'
set vpn ipsec ike-group IKE-SPOKE proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-SPOKE proposal 1 hash 'sha1'
set vpn ipsec ipsec-interfaces interface 'tun0'
set vpn ipsec profile DMVPN authentication mode 'pre-shared-secret'
set vpn ipsec profile DMVPN authentication pre-shared-secret 'SECRET'
set vpn ipsec profile DMVPN bind tunnel 'tun0'
set vpn ipsec profile DMVPN esp-group 'ESP-SPOKE'
set vpn ipsec profile DMVPN ike-group 'IKE-SPOKE'

Spoke2 Config:

set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth0 duplex 'auto'
set interfaces ethernet eth0 hw-id '12:4d:b1:65:87:00'
set interfaces ethernet eth0 smp_affinity 'auto'
set interfaces ethernet eth0 speed 'auto'
set interfaces loopback lo address '10.0.3.1/24'
set interfaces tunnel tun0 address '10.0.0.12/24'
set interfaces tunnel tun0 encapsulation 'gre'
set interfaces tunnel tun0 local-ip '172.31.63.36'
set interfaces tunnel tun0 multicast 'enable'
set interfaces tunnel tun0 parameters ip key '1'
set protocols bgp 65000 neighbor 10.0.0.1 remote-as '65000'
set protocols bgp 65000 neighbor 10.0.0.1 soft-reconfiguration 'inbound'
set protocols bgp 65000 neighbor 10.0.0.1 timers holdtime '15'
set protocols bgp 65000 neighbor 10.0.0.1 timers keepalive '5'
set protocols bgp 65000 neighbor 10.0.0.1 update-source '10.0.0.12'
set protocols bgp 65000 network '10.0.3.0/24'
set protocols bgp 65000 parameters router-id '10.0.0.12'
set protocols nhrp tunnel tun0 cisco-authentication 'SECRET'
set protocols nhrp tunnel tun0 map 10.0.0.1/24 nbma-address '54.172.39.26'
set protocols nhrp tunnel tun0 map 10.0.0.1/24 'register'
set protocols nhrp tunnel tun0 multicast 'nhs'
set protocols nhrp tunnel tun0 'redirect'
set protocols nhrp tunnel tun0 'shortcut'
set vpn ipsec esp-group ESP-SPOKE compression 'disable'
set vpn ipsec esp-group ESP-SPOKE lifetime '1800'
set vpn ipsec esp-group ESP-SPOKE mode 'tunnel'
set vpn ipsec esp-group ESP-SPOKE pfs 'dh-group2'
set vpn ipsec esp-group ESP-SPOKE proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-SPOKE proposal 1 hash 'sha1'
set vpn ipsec esp-group ESP-SPOKE proposal 2 encryption '3des'
set vpn ipsec esp-group ESP-SPOKE proposal 2 hash 'md5'
set vpn ipsec ike-group IKE-SPOKE key-exchange 'ikev1'
set vpn ipsec ike-group IKE-SPOKE lifetime '3600'
set vpn ipsec ike-group IKE-SPOKE proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-SPOKE proposal 1 hash 'sha1'
set vpn ipsec ike-group IKE-SPOKE proposal 2 encryption 'aes128'
set vpn ipsec ike-group IKE-SPOKE proposal 2 hash 'sha1'
set vpn ipsec ipsec-interfaces interface 'tun0'
set vpn ipsec profile DMVPN authentication mode 'pre-shared-secret'
set vpn ipsec profile DMVPN authentication pre-shared-secret 'SECRET'
set vpn ipsec profile DMVPN bind tunnel 'tun0'
set vpn ipsec profile DMVPN esp-group 'ESP-SPOKE'
set vpn ipsec profile DMVPN ike-group 'IKE-SPOKE'

With a baseline DMVPN config, setting the tunnel local-ip to the actual eth0 address, NHRP works.

  • VyOS2 registers to VyOS1
    • Successful pings in both directions
    • BGP session establishes
  • VyOS3 registers to VyOS1
    • Successful ping in both directions
    • BGP session establishes
  • VyOS2 & VyOS3 can ping each other

The network is all working in terms of reachability and forwarding, but there doesn’t appear to be any IPSEC happening, just as Syed had observed, and reason seems to be related to my initial supposition – there’s a mismatch between the crypto relationship, and the source interface.  Here’s a snippet from the logs:

Oct 16 00:53:14 VyOS2-AWS pluto[2747]: "10.0.0.11-to-10.0.0.12" #8:
                            Peer ID is ID_IPV4_ADDR: '172.31.63.36'
Oct 16 00:53:14 VyOS2-AWS pluto[2747]: "10.0.0.11-to-10.0.0.12" #8:
                            we require peer to have ID '52.91.245.55',
                            but peer declares '172.31.63.36'
Oct 16 00:53:14 VyOS2-AWS pluto[2747]: "10.0.0.11-to-10.0.0.12" #8:
                            sending encrypted notification INVALID_ID_INFORMATION
                            to 52.91.245.55:500

So clearly, the linkage between NHRP and the Crypto is missing.  Now elsewhere in the logs, I see this:

Oct 15 17:43:59 VyOS2-AWS pluto[3516]: Starting IKEv1 pluto daemon (strongSwan 4.5.2)
                                       THREADS SMARTCARD VENDORID CISCO_QUIRKS
Oct 15 17:43:59 VyOS2-AWS ipsec_starter[3514]: pluto (3516) started after 20 ms
Oct 15 17:43:59 VyOS2-AWS pluto[3516]: including NAT-Traversal patch
                                       (Version 0.6c) [disabled]

My best guess at this point is that last message is the reason this is failing.  My reading of that log message says that the NAT-Traversal patch is included in the crypto packages, but for whatever the reason, it’s not enabled.  I need to find out if there’s a command we should be enabling within our router config, or if there’s something else that needs to happen.

This should be a command, but so far, I haven’t had any luck finding it.

Share this post:
Facebooktwittergoogle_plusredditpinterestlinkedintumblrmail

war Written by:

8 Comments

  1. October 16, 2016
    Reply

    It may be a stupid question, but, have you tried to enable NAT-T in the config explicitly?

    set vpn ipsec nat-traversal enable
    set vpn ipsec nat-networks allowed-network 0.0.0.0/0

    • October 16, 2016
      Reply

      Not a stupid question at all… I didn’t have either of those lines. Syed seems to have had NAT-T enabled, but not the second line allowing 0.0.0.0/0.

      When I turned it on, there was definitely a change. The logs now show NAT-T:

      Oct 16 20:04:30 VyOS1-AWS pluto[2749]: “vpnprof-tunnel-tun0″[4] 52.91.245.55:4500 #6: responding to Main Mode from unknown peer 52.91.245.55:4500
      Oct 16 20:04:30 VyOS1-AWS pluto[2749]: “vpnprof-tunnel-tun0″[4] 52.91.245.55:4500 #6: NAT-Traversal: Result using RFC 3947: both are NATed
      Oct 16 20:04:30 VyOS1-AWS pluto[2749]: “vpnprof-tunnel-tun0″[4] 52.91.245.55:4500 #6: Peer ID is ID_IPV4_ADDR: ‘172.31.63.36’

      Show vpn ipsec sa shows:

      vyos@VyOS1-AWS:~$ sh vpn ips sa
      Peer ID / IP Local ID / IP
      ———— ————-
      52.91.245.55 172.31.48.229

      Tunnel State Bytes Out/In Encrypt Hash NAT-T A-Time L-Time Proto
      —— —– ————- ——- —- —– —— —— —–
      tun0[4] down n/a n/a n/a yes 0 1800 gre

      Peer ID / IP Local ID / IP
      ———— ————-
      54.159.116.229 172.31.48.229

      Tunnel State Bytes Out/In Encrypt Hash NAT-T A-Time L-Time Proto
      —— —– ————- ——- —- —– —— —— —–
      tun0[2] down n/a n/a n/a yes 0 1800 gre

      I’d have expected to see some manner of byte count for encrypted traffic, but NHRP is definitely completing correctly now, and the BGP sessions have come up through the Tun0 interfaces.

      It appears to be working – but how can I verify that the traffic is actually being encrypted?

  2. syed
    October 17, 2016
    Reply

    Thanks War for putting ur time and effort but seems like its a bug/defect and not sure if “set vpn ipsec nat-networks allowed-network 0.0.0.0/0” is required or not . I believe once we have NAT Traversal enable and working then this check of comparing PEER ID ID_IPV4_ADDR should understand peer/spoke is behind nat and allow and accept the communication as Legitimate.

    Oct 16 00:53:14 VyOS2-AWS pluto[2747]: “10.0.0.11-to-10.0.0.12” #8:
    Peer ID is ID_IPV4_ADDR: ‘172.31.63.36’
    Oct 16 00:53:14 VyOS2-AWS pluto[2747]: “10.0.0.11-to-10.0.0.12” #8:
    we require peer to have ID ‘52.91.245.55’,
    but peer declares ‘172.31.63.36’
    Oct 16 00:53:14 VyOS2-AWS pluto[2747]: “10.0.0.11-to-10.0.0.12” #8:
    sending encrypted notification INVALID_ID_INFORMATION
    to 52.91.245.55:500

    And secondly @war if u want to make sure if the packet is encrypted or decrypted atleast in Cisco “show crypto ipsec sa ” will tell u the number of packets encrypted and decrypted but in case of vyos or vyatta show vpn ipsec sa will simply show u the number of traffic incrementing but not the packet count .

    I came across the similar issue in Vyatta 5400 and seems like there is bug too.Not sure if vyos Developer have tested or aware of this issue.

    Brocade 5400 vRouter / VSE-9088
    Connection between vRouter Hub and Cisco ASA spoke fails if the spoke is behind NAT access gateway.

    Regards
    Syed.

    • October 17, 2016
      Reply

      There’s an equivalent command on VyOS: ‘show vpn ipsec sa’

      It usually shows the amount of encrypted traffic, though in this case, it’s not reporting the tunnel as being up, nor are the packet/traffic counters incrementing.

      It’s odd.

      • November 3, 2016
        Reply

        Reproduced and have the same behavior here.
        mGRE tunnels are up, can ping across them, BGP is up, but show vpn ipsec sa states down.

        is there any chance this traffic is flowing without encryption ?

        • November 5, 2016
          Reply

          There’s definitely a chance, but it’s equally possible that it’s just a bug in the reporting/counter. There are extra moving parts in this configuration, and it’s possible that the code isn’t sure how to account for them.

          I tore down my AWS environment. When I get some time, I’ll build a similar environment here at the house where I have some additional control and visibility, and try to take a closer look at the raw traffic.

          If yours is still up, maybe you can give something like this a try:

          • Modify your crypto parameters to use a NULL algorithm
          • Transfer a plain text file through the tunnel
          • Get a packet capture of traffic to see what you can see
          • Modify the crypto parameters to use a non-NULL algorithm
          • Transfer the same text file again
          • Get another packet capture
          • Compare the results

          To my mind, this would prove that the traffic is or isn’t being encrypted in flight, and we’d know that we were just looking at a bug in the way crypto traffic was tracked/counted.

    • November 11, 2017
      Reply

      Good stuff. Was disappointed that there were no replies.

      Did you happen to log it as an issue on the VyOS Issue tracker?

Leave a Reply

Your email address will not be published. Required fields are marked *