VyOS Crypto Throughput – Initial Results

I added the crypto and tunnel configurations to the test environment today, and have some preliminary numbers.

Here’s what the environment looks like now, with the tunnels added:

 

VyOS Environment with Tunnels
VyOS Environment with Tunnels

 

Here’s more detail about the way the VMs are provisioned:

 

VyOS VM Provisioning
VyOS VM Provisioning

 

My hypervisors are repurposed Dell R720s, w/(2)Xeon E5-2670 -8 cores each @ 2.6Ghz, and 256GB of RAM.

I gave the VyOS VMs:

  • 1 CPU Core
  • 1 GB of RAM
  • 8 GB of disk
  • 2 VMXNET3 Network Adapters
    • One External (Internet)
    • One Internal (Test VLAN)
  • 2GB Video RAM (Reduced from the default of 4GB)
  • All four VyOS instances are provisioned identically

The test hosts sitting behind the VyOS instances are provisioned the same way.

 


Tip: When standing up a VM for VyOS, select “Debian 7 / 64-bit.”  If you choose “Other Linux / 64-bit,” VMware only gives you the venerable E1000 network adapters, which can be performance limited.  You want the VMXNET3 adapters which have no problem with 10Gig.


 

I built GRE tunnels, and the associated crypto, as a full mesh between each of the router instances.

Here’s the configuration I added to VyOS-RTR1:

 

# Crypto Parameters
edit vpn ipsec
set esp-group ESP-GRE compression 'disable'
set esp-group ESP-GRE lifetime '1800'
set esp-group ESP-GRE mode 'tunnel'
set esp-group ESP-GRE pfs 'enable'
set esp-group ESP-GRE proposal 1 encryption 'aes256'
set esp-group ESP-GRE proposal 1 hash 'sha1'
set ike-group IKE-GRE ikev2-reauth 'no'
set ike-group IKE-GRE key-exchange 'ikev1'
set ike-group IKE-GRE lifetime '3600'
set ike-group IKE-GRE proposal 1 encryption 'aes256'
set ike-group IKE-GRE proposal 1 hash 'sha1'
top

# TUNNEL 12 (VyOS1 - VyOS2)
edit vpn ipsec site-to-site peer xxx.xxx.xxx.222
set authentication mode pre-shared-secret
set authentication pre-shared-secret vy0svpnk3y
set connection-type initiate
set ike-group IKE-GRE
set local-address xxx.xxx.xxx.111
set tunnel 12 allow-nat-networks disable
set tunnel 12 allow-public-networks disable
set tunnel 12 esp-group ESP-GRE
set tunnel 12 local prefix 10.0.1.0/24
set tunnel 12 remote prefix 10.0.2.0/24
top
edit interfaces tunnel tun12
set address 10.0.12.1/30
set encapsulation gre
set local-ip xxx.xxx.xxx.111
set remote-ip xxx.xxx.xxx.222
top

# TUNNEL 13 (VyOS1 - VyOS3)
edit vpn ipsec site-to-site peer xxx.xxx.xxx.333
set authentication mode pre-shared-secret
set authentication pre-shared-secret vy0svpnk3y
set connection-type initiate
set ike-group IKE-GRE
set local-address xxx.xxx.xxx.111
set tunnel 13 allow-nat-networks disable
set tunnel 13 allow-public-networks disable
set tunnel 13 esp-group ESP-GRE
set tunnel 13 local prefix 10.0.1.0/24
set tunnel 13 remote prefix 10.0.3.0/24
top
edit interfaces tunnel tun13
set address 10.0.13.1/30
set encapsulation gre
set local-ip xxx.xxx.xxx.111
set remote-ip xxx.xxx.xxx.333
top

# TUNNEL 14 (VyOS1 - VyOS4)
edit vpn ipsec site-to-site peer xxx.xxx.xxx.444
set authentication mode pre-shared-secret
set authentication pre-shared-secret vy0svpnk3y
set connection-type initiate
set ike-group IKE-GRE
set local-address xxx.xxx.xxx.111
set tunnel 14 allow-nat-networks disable
set tunnel 14 allow-public-networks disable
set tunnel 14 esp-group ESP-GRE
set tunnel 14 local prefix 10.0.1.0/24
set tunnel 14 remote prefix 10.0.4.0/24
top
edit interfaces tunnel tun14
set address 10.0.14.1/30
set encapsulation gre
set local-ip xxx.xxx.xxx.111
set remote-ip xxx.xxx.xxx.444
top

# Static Routes Across Tunnels
set protocols static route 10.0.2.0/24 next-hop 10.0.12.2
set protocols static route 10.0.3.0/24 next-hop 10.0.13.2
set protocols static route 10.0.4.0/24 next-hop 10.0.14.2
commit
save

 

After I’d repeated this process on each of the other routers, making the appropriate addressing substitutions, I checked all of the crypto security associations:

 

vyos-rtr1:~$ sh vpn ike sa
Peer ID / IP                            Local ID / IP
------------                            -------------
xxx.xxx.xxx.222                         xxx.xxx.xxx.111 
    State  Encrypt  Hash    D-H Grp  NAT-T  A-Time  L-Time
    -----  -------  ----    -------  -----  ------  ------
    up     aes256   sha1    5        no     1882    3600
Peer ID / IP                            Local ID / IP
------------                            -------------
xxx.xxx.xxx.333                         xxx.xxx.xxx.111 
    State  Encrypt  Hash    D-H Grp  NAT-T  A-Time  L-Time
    -----  -------  ----    -------  -----  ------  ------
    up     aes256   sha1    5        no     1623    3600
Peer ID / IP                            Local ID / IP
------------                            -------------
xxx.xxx.xxx.444                         xxx.xxx.xxx.111 
    State  Encrypt  Hash    D-H Grp  NAT-T  A-Time  L-Time
    -----  -------  ----    -------  -----  ------  ------
    up     aes256   sha1    5        no     1366    3600


vyos-rtr1:~$ sh vpn ipsec sa
Peer ID / IP                            Local ID / IP
------------                            -------------
xxx.xxx.xxx.222                         xxx.xxx.xxx.111
    Tunnel  State  Bytes Out/In   Encrypt  Hash    NAT-T  A-Time  L-Time  Proto
    ------  -----  -------------  -------  ----    -----  ------  ------  -----
    12      up     0.0/0.0        aes256   sha1    no     1103    1800    all
Peer ID / IP                            Local ID / IP
------------                            -------------
xxx.xxx.xxx.333                         xxx.xxx.xxx.111
    Tunnel  State  Bytes Out/In   Encrypt  Hash    NAT-T  A-Time  L-Time  Proto
    ------  -----  -------------  -------  ----    -----  ------  ------  -----
    13      up     0.0/0.0        aes256   sha1    no     887     1800    all
Peer ID / IP                            Local ID / IP
------------                            -------------
xxx.xxx.xxx.444                         xxx.xxx.xxx.111
    Tunnel  State  Bytes Out/In   Encrypt  Hash    NAT-T  A-Time  L-Time  Proto
    ------  -----  -------------  -------  ----    -----  ------  ------  -----
    14      up     0.0/0.0        aes256   sha1    no     459     1800    all

 

I checked each of the routers in the same way – and actually discovered I’d missed a couple of substitutions in one of my configs.  Trust me, you want to check this early in the process, rather than drive yourself crazy.

Once that all checked out, I had good connectivity between each and every one of the test hosts.

I fired up iperf on Hosts 2, 3, and 4:

 

iperf -s

 

On Host 1, I tested against each:

 

vyos-host1:~$ iperf -c 10.0.2.10
------------------------------------------------------------
Client connecting to 10.0.2.10, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 10.0.1.10 port 45890 connected with 10.0.2.10 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   802 MBytes   672 Mbits/sec

vyos-host1:~$ iperf -c 10.0.3.10
------------------------------------------------------------
Client connecting to 10.0.3.10, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 10.0.1.10 port 34632 connected with 10.0.3.10 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   333 MBytes   278 Mbits/sec

vyos-host1:~$ iperf -c 10.0.4.10
------------------------------------------------------------
Client connecting to 10.0.4.10, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 10.0.1.10 port 47618 connected with 10.0.4.10 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   354 MBytes   296 Mbits/sec

 

The first test, between Hosts 1 & 2, is local.  This is between systems on unique hypervisors in the same rack, connected via local 10GbE switching.  With a longer test duration, you can see some fluctuation, but on average, the results are pretty good:

 

vyos-host1:~$ iperf -c 10.0.2.10 -t 60 -i 10
------------------------------------------------------------
Client connecting to 10.0.2.10, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 10.0.1.10 port 45900 connected with 10.0.2.10 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   876 MBytes   735 Mbits/sec
[  3] 10.0-20.0 sec   896 MBytes   752 Mbits/sec High
[  3] 20.0-30.0 sec   682 MBytes   572 Mbits/sec
[  3] 30.0-40.0 sec   674 MBytes   565 Mbits/sec
[  3] 40.0-50.0 sec   560 MBytes   470 Mbits/sec Low
[  3] 50.0-60.0 sec   850 MBytes   713 Mbits/sec
[  3]  0.0-60.0 sec  4.43 GBytes   635 Mbits/sec Average

 

The second and third tests, between Hosts 1 & 3, and Host 1 & 4, are remote tests.  These are hosted in different co-lo facilities, approximately 1000 miles apart.

Again, increasing the test duration shows similar fluctuation:

 

vyos-host1:~$ iperf -c 10.0.3.10 -t 60 -i 10
------------------------------------------------------------
Client connecting to 10.0.3.10, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 10.0.1.10 port 34642 connected with 10.0.3.10 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   236 MBytes   198 Mbits/sec Low
[  3] 10.0-20.0 sec   239 MBytes   201 Mbits/sec
[  3] 20.0-30.0 sec   296 MBytes   248 Mbits/sec
[  3] 30.0-40.0 sec   294 MBytes   246 Mbits/sec
[  3] 40.0-50.0 sec   401 MBytes   337 Mbits/sec High
[  3] 50.0-60.0 sec   401 MBytes   336 Mbits/sec
[  3]  0.0-60.0 sec  1.82 GBytes   261 Mbits/sec Average

 

Keep in mind that while these hypervisors are largely vacant at this point, the site itself is in production, so there is some contention on the Internet circuits.  It could be the source of much of the variability in throughput.

Also, another observation I made.  At this point, it looks to me like maximum crypto throughout is somewhat linear, based on CPU core clock speed.

When I ran this test at home, on a 2.0Ghz Xeon-D, I was getting crypto numbers in the 420-450Mbps range.  Today, on a 2.6Ghz Xeon, the average was around 635 Mbps.  That’s a clock speed increase of about 1.3x, and a crypto throughput increase of about 1.4x.

My expectation is that adding additional cores will not have any impact on crypto throughput of a single stream, as crypto is generally bound to a single core.  In the real world, multiple streams may benefit from additional cores.  I plan to adjust some variables, and run some more tests.  If I see anything interesting, I’ll be sure to share it.

 

Share this post:
Facebooktwittergoogle_plusredditpinterestlinkedintumblrmail
war Written by:

13 Comments

  1. September 13, 2016
    Reply

    Just wondering if you willing to test csr1000v if we can get 60-day trial?
    Will be not official real life test!

    • September 13, 2016
      Reply

      Certainly.

      • September 13, 2016
        Reply

        Cool.
        I will need UDIs for devices. I will ask community to get three other trials keys, but it should be a great experience!
        Ping me on IRC
        Thanks!

  2. October 20, 2016
    Reply

    By the way, were the CPUs AES-NI capable and was it enabled?

    • October 20, 2016
      Reply

      I assume so… This is ESX on Dell Poweredge 720s, with Intel(R) Xeon(R) CPU E5-2670s, but I don’t “own” the hypervisor in this case. The chip certainly supports AES-NI. I believe that VMware would make use of it by default.

      vyos-rtr1:~$ grep aes /proc/cpuinfo

      flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm ida arat pln pts dtherm

  3. syed
    April 13, 2017
    Reply

    Hi War ,
    Is the above configuration is for Gre over IPsec ? As i am trying Gre Over IPsec and its working fine and the moment i tried the same configs with the VyOS behind Nat Ipsec is breaking.

    IKE is Up and IPsec down.
    Using the following from this link.

    https://emrlab.wordpress.com/2013/07/16/ipsec-protected-gre-tunnel-vyatta-to-cisco-nat-ospf-2/

    Problem Description:
    =============
    Gre over Ipsec b/w Vyos and Vyatta not working , IKE is up but IPsec down.

    GRE-IPSEC B/w VYOS and Vyatta:
    ====================

    Topology:
    =========

    VYOS(172.31.61.122)—1:1NAT GW —Y.Y.Y.Y———————GRE-IPSEC——————(X.X.X.X)—VYATTA

    WHERE X.X.X.X & Y.Y.Y.Y ARE PUBLIC IPs

    VYOS-STATIC-NAT-AWS:
    ====================

    wanclouds@VyOS-AMI-ZAYAD:~$ show configuration commands | grep vpn
    set vpn ipsec esp-group ESP-1W0 lifetime ‘86400’
    set vpn ipsec esp-group ESP-1W0 mode ‘transport’
    set vpn ipsec esp-group ESP-1W0 pfs ‘dh-group5’
    set vpn ipsec esp-group ESP-1W0 proposal 1 encryption ‘3des’
    set vpn ipsec esp-group ESP-1W0 proposal 1 hash ‘md5’
    set vpn ipsec ike-group IKE-1W0 lifetime ‘86400’
    set vpn ipsec ike-group IKE-1W0 proposal 1 dh-group ‘5’
    set vpn ipsec ike-group IKE-1W0 proposal 1 encryption ‘aes256’
    set vpn ipsec ike-group IKE-1W0 proposal 1 hash ‘md5’
    set vpn ipsec ipsec-interfaces interface ‘eth0’
    set vpn ipsec nat-traversal ‘enable’
    set vpn ipsec site-to-site peer X.X.X.X authentication id ‘419b9c8ee2544d598bf209173640f934’
    set vpn ipsec site-to-site peer X.X.X.X authentication mode ‘pre-shared-secret’
    set vpn ipsec site-to-site peer X.X.X.X authentication pre-shared-secret ‘62066c88582a411390965d7827d2780c’
    set vpn ipsec site-to-site peer X.X.X.X authentication remote-id ‘419b9c8ee2544d598bf209173640f934’
    set vpn ipsec site-to-site peer X.X.X.X default-esp-group ‘ESP-1W0’
    set vpn ipsec site-to-site peer X.X.X.X ike-group ‘IKE-1W0’
    set vpn ipsec site-to-site peer X.X.X.X local-address ‘172.31.61.122’
    set vpn ipsec site-to-site peer X.X.X.X tunnel 0 protocol ‘gre’
    wanclouds@VyOS-AMI-ZAYAD:~$
    wanclouds@VyOS-AMI-ZAYAD:~$
    wanclouds@VyOS-AMI-ZAYAD:~$ show configuration commands | grep tunnel
    set interfaces tunnel tun0 address ‘172.168.100.198/24’
    set interfaces tunnel tun0 encapsulation ‘gre’
    set interfaces tunnel tun0 local-ip ‘172.31.61.122’
    set interfaces tunnel tun0 multicast ‘enable’
    set interfaces tunnel tun0 remote-ip ‘X.X.X.X’
    set vpn ipsec site-to-site peer X.X.X.X tunnel 0 protocol ‘gre’
    wanclouds@VyOS-AMI-ZAYAD:~$
    wanclouds@VyOS-AMI-ZAYAD:~$
    wanclouds@VyOS-AMI-ZAYAD:~$ show log
    log login
    wanclouds@VyOS-AMI-ZAYAD:~$ show vpn ike sa
    Peer ID / IP Local ID / IP
    ———— ————-
    X.X.X.X 172.31.61.122

    State Encrypt Hash D-H Grp NAT-T A-Time L-Time
    —– ——- —- ——- —– —— ——
    up aes256 md5 5 yes 3658 86400

    wanclouds@VyOS-AMI-ZAYAD:~$ show vpn ipsec sa
    Peer ID / IP Local ID / IP
    ———— ————-
    X.X.X.X 172.31.61.122

    Tunnel State Bytes Out/In Encrypt Hash NAT-T A-Time L-Time Proto
    —— —– ————- ——- —- —– —— —— —–
    0 down n/a n/a n/a yes 0 86400 gre

    wanclouds@VyOS-AMI-ZAYAD:~$ show log
    log login
    wanclouds@VyOS-AMI-ZAYAD:~$ show log tail -20
    Apr 11 21:30:45 VyOS-AMI-ZAYAD pluto[12059]: “peer-X.X.X.X-tunnel-0” #410: max number of retransmissions (2) reached STATE_QUICK_I1. No acceptable response to our first Quick Mode message: perhaps peer likes no proposal
    Apr 11 21:30:45 VyOS-AMI-ZAYAD pluto[12059]: “peer-X.X.X.X-tunnel-0” #410: starting keying attempt 37 of an unlimited number
    Apr 11 21:30:45 VyOS-AMI-ZAYAD pluto[12059]: “peer-X.X.X.X-tunnel-0” #428: initiating Quick Mode PSK+ENCRYPT+PFS+UP to replace #410 {using isakmp#15}
    Apr 11 21:30:45 VyOS-AMI-ZAYAD pluto[12059]: “peer-X.X.X.X-tunnel-0” #15: ignoring informational payload, type INVALID_MESSAGE_ID
    Apr 11 21:30:45 VyOS-AMI-ZAYAD pluto[12059]: “peer-X.X.X.X-tunnel-0” #15: next payload type of ISAKMP Hash Payload has an unknown value: 58
    Apr 11 21:30:45 VyOS-AMI-ZAYAD pluto[12059]: “peer-X.X.X.X-tunnel-0” #15: malformed payload in packet
    Apr 11 21:30:47 VyOS-AMI-ZAYAD pluto[12059]: “peer-X.X.X.X-tunnel-0” #15: ignoring informational payload, type INVALID_MESSAGE_ID
    Apr 11 21:30:54 VyOS-AMI-ZAYAD pluto[12059]: last message repeated 3 times
    Apr 11 21:30:54 VyOS-AMI-ZAYAD pluto[12059]: “peer-X.X.X.X-tunnel-0” #411: max number of retransmissions (2) reached STATE_QUICK_I1. No acceptable response to our first Quick Mode message: perhaps peer likes no proposal
    Apr 11 21:30:54 VyOS-AMI-ZAYAD pluto[12059]: “peer-X.X.X.X-tunnel-0” #411: starting keying attempt 42 of an unlimited number
    Apr 11 21:30:54 VyOS-AMI-ZAYAD pluto[12059]: “peer-X.X.X.X-tunnel-0” #429: initiating Quick Mode PSK+ENCRYPT+PFS+UP to replace #411 {using isakmp#15}
    Apr 11 21:30:54 VyOS-AMI-ZAYAD pluto[12059]: “peer-X.X.X.X-tunnel-0” #15: next payload type of ISAKMP Hash Payload has an unknown value: 72
    Apr 11 21:30:54 VyOS-AMI-ZAYAD pluto[12059]: “peer-X.X.X.X-tunnel-0” #15: malformed payload in packet
    Apr 11 21:30:56 VyOS-AMI-ZAYAD pluto[12059]: “peer-X.X.X.X-tunnel-0” #15: ignoring informational payload, type INVALID_MESSAGE_ID
    Apr 11 21:30:56 VyOS-AMI-ZAYAD pluto[12059]: “peer-X.X.X.X-tunnel-0” #412: max number of retransmissions (2) reached STATE_QUICK_I1. No acceptable response to our first Quick Mode message: perhaps peer likes no proposal
    Apr 11 21:30:56 VyOS-AMI-ZAYAD pluto[12059]: “peer-X.X.X.X-tunnel-0” #412: starting keying attempt 15 of an unlimited number
    Apr 11 21:30:56 VyOS-AMI-ZAYAD pluto[12059]: “peer-X.X.X.X-tunnel-0” #430: initiating Quick Mode PSK+ENCRYPT+PFS+UP to replace #412 {using isakmp#15}
    Apr 11 21:30:56 VyOS-AMI-ZAYAD pluto[12059]: “peer-X.X.X.X-tunnel-0” #15: ignoring informational payload, type INVALID_MESSAGE_ID
    Apr 11 21:30:56 VyOS-AMI-ZAYAD pluto[12059]: “peer-X.X.X.X-tunnel-0” #15: byte 2 of ISAKMP Hash Payload must be zero, but is not
    Apr 11 21:30:56 VyOS-AMI-ZAYAD pluto[12059]: “peer-X.X.X.X-tunnel-0” #15: malformed payload in packet
    wanclouds@VyOS-AMI-ZAYAD:~$
    wanclouds@VyOS-AMI-ZAYAD:~$

    VYATTA-PUBLIC-IP:
    ===============

    vyatta@gw-melbourne1-02-06-2016:~$ show configuration commands | grep vpn
    set vpn ipsec esp-group ESP-1W0 lifetime ‘86400’
    set vpn ipsec esp-group ESP-1W0 mode ‘transport’
    set vpn ipsec esp-group ESP-1W0 pfs ‘dh-group5’
    set vpn ipsec esp-group ESP-1W0 proposal 1 encryption ‘3des’
    set vpn ipsec esp-group ESP-1W0 proposal 1 hash ‘md5’
    set vpn ipsec ike-group IKE-1W0 lifetime ‘86400’
    set vpn ipsec ike-group IKE-1W0 proposal 1 dh-group ‘5’
    set vpn ipsec ike-group IKE-1W0 proposal 1 encryption ‘aes256’
    set vpn ipsec ike-group IKE-1W0 proposal 1 hash ‘md5’
    set vpn ipsec ipsec-interfaces interface ‘bond1v1’
    set vpn ipsec nat-traversal ‘enable’
    set vpn ipsec site-to-site peer Y.Y.Y.Y authentication id ‘419b9c8ee2544d598bf209173640f934’
    set vpn ipsec site-to-site peer Y.Y.Y.Y authentication mode ‘pre-shared-secret’
    set vpn ipsec site-to-site peer Y.Y.Y.Y authentication pre-shared-secret ‘62066c88582a411390965d7827d2780c’
    set vpn ipsec site-to-site peer Y.Y.Y.Y authentication remote-id ‘419b9c8ee2544d598bf209173640f934’
    set vpn ipsec site-to-site peer Y.Y.Y.Y default-esp-group ‘ESP-1W0’
    set vpn ipsec site-to-site peer Y.Y.Y.Y ike-group ‘IKE-1W0’
    set vpn ipsec site-to-site peer Y.Y.Y.Y local-address ‘X.X.X.X’
    set vpn ipsec site-to-site peer Y.Y.Y.Y tunnel 0 protocol ‘gre’
    vyatta@gw-melbourne1-02-06-2016:~$
    vyatta@gw-melbourne1-02-06-2016:~$
    vyatta@gw-melbourne1-02-06-2016:~$
    vyatta@gw-melbourne1-02-06-2016:~$ show configuration commands | grep tunnel
    set interfaces tunnel tun0 address ‘172.168.100.163/24’
    set interfaces tunnel tun0 encapsulation ‘gre’
    set interfaces tunnel tun0 local-ip ‘X.X.X.X’
    set interfaces tunnel tun0 multicast ‘enable’
    set interfaces tunnel tun0 remote-ip ‘Y.Y.Y.Y’
    set vpn ipsec site-to-site peer Y.Y.Y.Y tunnel 0 protocol ‘gre’
    vyatta@gw-melbourne1-02-06-2016:~$
    vyatta@gw-melbourne1-02-06-2016:~$
    vyatta@gw-melbourne1-02-06-2016:~$ show vpn ike sa
    Peer ID / IP Local ID / IP
    ———— ————-
    Y.Y.Y.Y X.X.X.X

    State Encrypt Hash D-H Grp NAT-T A-Time L-Time
    —– ——- —- ——- —– —— ——
    up aes256 md5 5 yes 3377 86400

    vyatta@gw-melbourne1-02-06-2016:~$
    vyatta@gw-melbourne1-02-06-2016:~$ show vpn ipsec sa
    Peer ID / IP Local ID / IP
    ———— ————-
    Y.Y.Y.Y X.X.X.X

    Tunnel State Bytes Out/In Encrypt Hash NAT-T A-Time L-Time Proto
    —— —– ————- ——- —- —– —— —— —–
    0 down n/a n/a n/a yes 0 86400 gre

    vyatta@gw-melbourne1-02-06-2016:~$ show log tail -25
    Apr 11 16:32:18 gw-melbourne1-02-06-2016 pluto[7603]: “peer-Y.Y.Y.Y-tunnel-0” #1: sending encrypted notification INVALID_MESSAGE_ID to Y.Y.Y.Y:4500
    Apr 11 16:32:18 gw-melbourne1-02-06-2016 pluto[7603]: “peer-Y.Y.Y.Y-tunnel-0” #1: cannot respond to IPsec SA request because no connection is known for X.X.X.X:4500[419b9c8ee2544d598bf209173640f934]:47/0…Y.Y.Y.Y:4500[419b9c8ee2544d598bf209173640f934]:47/0===172.31.61.122/32
    Apr 11 16:32:18 gw-melbourne1-02-06-2016 pluto[7603]: “peer-Y.Y.Y.Y-tunnel-0” #1: sending encrypted notification INVALID_ID_INFORMATION to Y.Y.Y.Y:4500
    Apr 11 16:32:19 gw-melbourne1-02-06-2016 sshd[10183]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.172.60 user=root
    Apr 11 16:32:20 gw-melbourne1-02-06-2016 pluto[7603]: “peer-Y.Y.Y.Y-tunnel-0” #1: Quick Mode I1 message is unacceptable because it uses a previously used Message ID 0x8e4cb23d (perhaps this is a duplicated packet)
    Apr 11 16:32:20 gw-melbourne1-02-06-2016 pluto[7603]: “peer-Y.Y.Y.Y-tunnel-0” #1: sending encrypted notification INVALID_MESSAGE_ID to Y.Y.Y.Y:4500
    Apr 11 16:32:21 gw-melbourne1-02-06-2016 sshd[10181]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.172.60 user=root
    Apr 11 16:32:21 gw-melbourne1-02-06-2016 pluto[7603]: “peer-Y.Y.Y.Y-tunnel-0” #1: Quick Mode I1 message is unacceptable because it uses a previously used Message ID 0x80a5a69d (perhaps this is a duplicated packet)
    Apr 11 16:32:21 gw-melbourne1-02-06-2016 pluto[7603]: “peer-Y.Y.Y.Y-tunnel-0” #1: sending encrypted notification INVALID_MESSAGE_ID to Y.Y.Y.Y:4500
    Apr 11 16:32:23 gw-melbourne1-02-06-2016 pluto[7603]: “peer-Y.Y.Y.Y-tunnel-0” #1: cannot respond to IPsec SA request because no connection is known for X.X.X.X:4500[419b9c8ee2544d598bf209173640f934]:47/0…Y.Y.Y.Y:4500[419b9c8ee2544d598bf209173640f934]:47/0===172.31.61.122/32
    Apr 11 16:32:23 gw-melbourne1-02-06-2016 pluto[7603]: “peer-Y.Y.Y.Y-tunnel-0” #1: sending encrypted notification INVALID_ID_INFORMATION to Y.Y.Y.Y:4500
    Apr 11 16:32:24 gw-melbourne1-02-06-2016 pluto[7603]: “peer-Y.Y.Y.Y-tunnel-0” #1: Quick Mode I1 message is unacceptable because it uses a previously used Message ID 0xe95143e1 (perhaps this is a duplicated packet)
    Apr 11 16:32:24 gw-melbourne1-02-06-2016 pluto[7603]: “peer-Y.Y.Y.Y-tunnel-0” #1: sending encrypted notification INVALID_MESSAGE_ID to Y.Y.Y.Y:4500
    Apr 11 16:32:25 gw-melbourne1-02-06-2016 pluto[7603]: “peer-Y.Y.Y.Y-tunnel-0” #1: Quick Mode I1 message is unacceptable because it uses a previously used Message ID 0xac9835cc (perhaps this is a duplicated packet)
    Apr 11 16:32:25 gw-melbourne1-02-06-2016 pluto[7603]: “peer-Y.Y.Y.Y-tunnel-0” #1: sending encrypted notification INVALID_MESSAGE_ID to Y.Y.Y.Y:4500
    Apr 11 16:32:26 gw-melbourne1-02-06-2016 pluto[7603]: “peer-Y.Y.Y.Y-tunnel-0” #1: Quick Mode I1 message is unacceptable because it uses a previously used Message ID 0x1c6d8a04 (perhaps this is a duplicated packet)
    Apr 11 16:32:26 gw-melbourne1-02-06-2016 pluto[7603]: “peer-Y.Y.Y.Y-tunnel-0” #1: sending encrypted notification INVALID_MESSAGE_ID to Y.Y.Y.Y:4500
    Apr 11 16:32:28 gw-melbourne1-02-06-2016 pluto[7603]: “peer-Y.Y.Y.Y-tunnel-0” #1: Quick Mode I1 message is unacceptable because it uses a previously used Message ID 0x10df76cd (perhaps this is a duplicated packet)
    Apr 11 16:32:28 gw-melbourne1-02-06-2016 pluto[7603]: “peer-Y.Y.Y.Y-tunnel-0” #1: sending encrypted notification INVALID_MESSAGE_ID to Y.Y.Y.Y:4500
    Apr 11 16:32:32 gw-melbourne1-02-06-2016 pluto[7603]: “peer-Y.Y.Y.Y-tunnel-0” #1: Quick Mode I1 message is unacceptable because it uses a previously used Message ID 0x83384514 (perhaps this is a duplicated packet)
    Apr 11 16:32:32 gw-melbourne1-02-06-2016 pluto[7603]: “peer-Y.Y.Y.Y-tunnel-0” #1: sending encrypted notification INVALID_MESSAGE_ID to Y.Y.Y.Y:4500
    Apr 11 16:32:32 gw-melbourne1-02-06-2016 pluto[7603]: “peer-Y.Y.Y.Y-tunnel-0” #1: cannot respond to IPsec SA request because no connection is known for X.X.X.X:4500[419b9c8ee2544d598bf209173640f934]:47/0…Y.Y.Y.Y:4500[419b9c8ee2544d598bf209173640f934]:47/0===172.31.61.122/32
    Apr 11 16:32:32 gw-melbourne1-02-06-2016 pluto[7603]: “peer-Y.Y.Y.Y-tunnel-0” #1: sending encrypted notification INVALID_ID_INFORMATION to Y.Y.Y.Y:4500
    Apr 11 16:32:33 gw-melbourne1-02-06-2016 pluto[7603]: “peer-Y.Y.Y.Y-tunnel-0” #1: Quick Mode I1 message is unacceptable because it uses a previously used Message ID 0x142e7918 (perhaps this is a duplicated packet)
    Apr 11 16:32:33 gw-melbourne1-02-06-2016 pluto[7603]: “peer-Y.Y.Y.Y-tunnel-0” #1: sending encrypted notification INVALID_MESSAGE_ID to Y.Y.Y.Y:4500
    vyatta@gw-melbourne1-02-06-2016:~$

  4. Maugli
    May 10, 2017
    Reply

    Hello, I’ve carefully read you article your design looks stable, I’m curious if you ever had an issue with tcp mtu/mss clamping on such configuration, since you’re using GRE tunnel and second question why you choose mode tunnel instead of mode transport.
    I’m asking because I tried several configurations already and I”m getting weired issue with tcp over such tunnels, UDP can pass about 300 mbit/s while tcp not more than 100 mbit/s. I’ll be glad if you could share your experience with me

    • May 14, 2017
      Reply

      Tunnel mode is implied when using DMVPN, which is just mGRE, NHRP, and IPSec rolled together. Technically, I should be doing TCP MSS adjust on the tunnel interfaces, but hadn’t rolled that in yet.
      DMVPN wouldn’t work without the GRE. It’s the mapping of the tunnel endpoints to tunnel addresses that makes the “magic” happen, with any two endpoints being able to discover each other, and forward encrypted traffic without hopping through an intermediate site.

  5. Maugli
    May 15, 2017
    Reply

    Thanks for your response, it would be interesting to get any stability measurement results for real production environment with TCP MSS adjusted, since I’m still having periodical issues with TCP streams on common ipip/gre over ipsec tunnel and I’m wondering if you also will get similar results

    • May 15, 2017
      Reply

      Sorry – I wasn’t paying attention to which post this was… This was for normal GRE/IPSec tunnels. I’d been doing a bunch of DMVPN related stuff, and assumed this was related.

      To once again answer your original question, this time in context, I prefer GRE tunnels over transport mode because I like the logical construct of an interface, and the corresponding point-to-point behavior. In almost all cases, I’m intending to run a routing protocol between the two endpoints, and tunnels facilitate this cleanly.

  6. Senthil
    May 24, 2017
    Reply

    Hi war,
    Is the throughput same if you use plain site-to-site ipsec without gre tunnel as we are using couple of vyatta 6.6 virtual router and are facing issue with ipsec vpn throughout , we experience packet loss once the traffic in ipsec touches about 30Mbps

    • June 3, 2017
      Reply

      In terms of pure crypto throughput, it should be the same. An issue that frequently arises when using GRE though, is MTU size, which can result in your router having to perform fragmentation. If your router gets into the fragmentation business, it can seriously tank performance.

      You can try running iperf with the -M option, which can be used to adjust the MSS down, somewhere in the 1400 byte range. This will be small enough to avoid any fragmentation issues.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.