Re: [vpp-dev] IPsec with AES-NI MB cryptodev

Tue, 19 Dec 2017 01:45:28 -0800 

Hey Matt,

Good stuff.
I think the change makes sense, remove any L4 checksum offload when doing IPsec. I wonder why we have those offloads on in the first place, but that is a different issue. 

Regards,
Sergio

On 18/12/2017 16:41, Matthew Smith wrote:

Hi Sergio,

I think I identified the problem.

When the UDP packet arrives, there are packet checksum offload flags set on the 
buffer (PKT_TX_TCP_CKSUM, PKT_TX_SCTP_CKSUM). Those can be seen in the previous 
trace I sent. I’m not sure why a UDP packet is resulting in those 2 particular 
flags being set. Maybe the trace function is showing the wrong thing, or maybe the 
PMD is doing something weird. I think after encapsulation & encryption the 
buffer still carries those original offload flags when it gets transmitted so the 
PMD tells the hardware to calculate the checksum. The UDP checksum is at the same 
offset in an IP packet as the lower 16 bits of an ESP sequence number is so those 2 
bytes are being overwritten by the hardware.

The ICMP packet buffers don’t have any offload flags set, which is why those 
packets are being correctly delivered.

I only saw this behavior with the DPDK cryptodev and not the default openssl 
encryption, because the openssl encrypt function esp_encrypt_node_fn() 
allocates a new buffer and initializes the flags to 
VLIB_BUFFER_TOTAL_LENGTH_VALID. When I applied the following patch to reset the 
flags in the same way the default crypto encrypt function does, I stopped 
seeing the ‘auth failed’ on UDP packets and saw those packets arrive at the 
host on the other side of the tunnel.

diff --git a/src/plugins/dpdk/ipsec/esp_encrypt.c 
b/src/plugins/dpdk/ipsec/esp_encrypt.c
index b4873d4..7bf24a4 100644
--- a/src/plugins/dpdk/ipsec/esp_encrypt.c
+++ b/src/plugins/dpdk/ipsec/esp_encrypt.c
@@ -265,6 +265,8 @@ dpdk_esp_encrypt_node_fn (vlib_main_t * vm,
          iv_size = cipher_alg->iv_len;
          trunc_size = auth_alg->trunc_size;
+ b0->flags = VLIB_BUFFER_TOTAL_LENGTH_VALID; 
+
          if (sa0->is_tunnel)
            {
              if (!is_ipv6 && !sa0->is_tunnel_ip6)   /* ip4inip4 */


I’m not sure if that’s all that needs to change. VPP also crashed a short while 
after I tested with that patch and I haven’t looked into the cause of the crash 
yet, so there may be more to properly fixing the issue than adding that 
statement.

-Matt




On Dec 18, 2017, at 4:01 AM, Gonzalez Monroy, Sergio 
<sergio.gonzalez.mon...@intel.com> wrote:

Hi Matt,

Could you add verbose to the trace? ie. 'trace add dpdk-input 10 verbose'

Thanks,
Sergio

On 15/12/2017 15:11, Matthew Smith wrote:

Hi Sergio,

Here is the sending side trace:

Packet 1

10:54:40:291456: dpdk-input
   VirtualFunctionEthernet0/6/0 rx queue 0
   buffer 0x6e4f: current data 14, length 84, free-list 0, clone-count 0, 
totlen-nifb 0, trace 0x0
                  l4-cksum-computed l4-cksum-correct l2-hdr-offset 0 
l3-hdr-offset 14
   PKT MBUF: port 0, nb_segs 1, pkt_len 98
     buf_len 2176, data_len 98, ol_flags 0x0, data_off 128, phys_addr 0x2c5b9440
     packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
     Packet Offload Flags
   IP4: 0e:c6:15:62:1f:9e -> 0e:7e:f2:5d:0d:b6
   ICMP: 10.1.3.235 -> 10.0.1.253
     tos 0x00, ttl 64, length 84, checksum 0x78df
     fragment id 0xa7e1, flags DONT_FRAGMENT
   ICMP echo_request checksum 0x5047
10:54:40:291473: ip4-input
   ICMP: 10.1.3.235 -> 10.0.1.253
     tos 0x00, ttl 64, length 84, checksum 0x78df
     fragment id 0xa7e1, flags DONT_FRAGMENT
   ICMP echo_request checksum 0x5047
10:54:40:291485: ip4-lookup
   fib 0 dpo-idx 2 flow hash: 0x00000000
   ICMP: 10.1.3.235 -> 10.0.1.253
     tos 0x00, ttl 64, length 84, checksum 0x78df
     fragment id 0xa7e1, flags DONT_FRAGMENT
   ICMP echo_request checksum 0x5047
10:54:40:291491: ip4-rewrite
   tx_sw_if_index 3 dpo-idx 2 : ipv4 via 0.0.0.0 ipsec0:  flow hash: 0x00000000
   00000000: 45000054a7e140003f0179df0a0103eb0a0001fd08005047549c00013de0335a
   00000020: 00000000210e020000000000101112131415161718191a1b1c1d1e1f
10:54:40:291495: ipsec-if-output
   IPSec: spi 3181023528 seq 28
10:54:40:291500: dpdk-esp-encrypt
   cipher aes-cbc-128 auth sha1-96
   IPSEC_ESP: 10.1.2.79 -> 10.0.0.213
     tos 0x00, ttl 254, length 152, checksum 0xc0d3
     fragment id 0x0000
   ESP: spi 3181023528, seq 29
10:54:40:291514: dpdk-crypto-input
   status: success

Packet 2

10:54:45:312374: dpdk-input
   VirtualFunctionEthernet0/6/0 rx queue 0
   buffer 0x6e76: current data 14, length 112, free-list 0, clone-count 0, 
totlen-nifb 0, trace 0x1
                  l4-cksum-computed l4-cksum-correct l2-hdr-offset 0 
l3-hdr-offset 14
   PKT MBUF: port 0, nb_segs 1, pkt_len 126
     buf_len 2176, data_len 126, ol_flags 0x0, data_off 128, phys_addr 
0x2c5b9e00
     packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
     Packet Offload Flags
       PKT_TX_TCP_CKSUM (0x0000) TCP cksum of TX pkt. computed by NIC
       PKT_TX_SCTP_CKSUM (0x0000) SCTP cksum of TX pkt. computed by NIC
   IP4: 0e:c6:15:62:1f:9e -> 0e:7e:f2:5d:0d:b6
   UDP: 10.1.3.235 -> 10.0.1.253
     tos 0x00, ttl 64, length 112, checksum 0x417c
     fragment id 0x1f19
   UDP: 55980 -> 53
     length 92, checksum 0x6c45
10:54:45:312393: ip4-input
   UDP: 10.1.3.235 -> 10.0.1.253
     tos 0x00, ttl 64, length 112, checksum 0x417c
     fragment id 0x1f19
   UDP: 55980 -> 53
     length 92, checksum 0x6c45
10:54:45:312400: ip4-lookup
   fib 0 dpo-idx 2 flow hash: 0x00000000
   UDP: 10.1.3.235 -> 10.0.1.253
     tos 0x00, ttl 64, length 112, checksum 0x417c
     fragment id 0x1f19
   UDP: 55980 -> 53
     length 92, checksum 0x6c45
10:54:45:312406: ip4-rewrite
   tx_sw_if_index 3 dpo-idx 2 : ipv4 via 0.0.0.0 ipsec0:  flow hash: 0x00000000
   00000000: 450000701f1900003f11427c0a0103eb0a0001fddaac0035005c6c45c0210120
   00000020: 0001000000000001337361666b6a687361666b6a6873666b6a686173
10:54:45:312409: ipsec-if-output
   IPSec: spi 3181023528 seq 29
10:54:45:312414: dpdk-esp-encrypt
   cipher aes-cbc-128 auth sha1-96
   IPSEC_ESP: 10.1.2.79 -> 10.0.0.213
     tos 0x00, ttl 254, length 184, checksum 0xc0b3
     fragment id 0x0000
   ESP: spi 3181023528, seq 30
10:54:45:312425: dpdk-crypto-input
   status: success


And the receiving side:

Packet 1

10:54:17:240273: dpdk-input
   VirtualFunctionEthernet0/7/0 rx queue 0
   buffer 0x2925: current data 14, length 152, free-list 0, clone-count 0, 
totlen-nifb 0, trace 0x0
                  l4-cksum-computed l4-cksum-correct l2-hdr-offset 0 
l3-hdr-offset 14
   PKT MBUF: port 1, nb_segs 1, pkt_len 166
     buf_len 2176, data_len 166, ol_flags 0x0, data_off 128, phys_addr 
0x2c4a49c0
     packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
     Packet Offload Flags
   IP4: 1a:85:68:80:00:01 -> 0e:d2:b9:22:74:fe
   IPSEC_ESP: 10.1.2.79 -> 10.0.0.213
     tos 0x00, ttl 252, length 152, checksum 0x2ed9
     fragment id 0x0000
10:54:17:240290: ip4-input
   IPSEC_ESP: 10.1.2.79 -> 10.0.0.213
     tos 0x00, ttl 252, length 152, checksum 0x2ed9
     fragment id 0x0000
10:54:17:240295: ip4-lookup
   fib 0 dpo-idx 7 flow hash: 0x00000000
   IPSEC_ESP: 10.1.2.79 -> 10.0.0.213
     tos 0x00, ttl 252, length 152, checksum 0x2ed9
     fragment id 0x0000
10:54:17:240301: ip4-local
     IPSEC_ESP: 10.1.2.79 -> 10.0.0.213
       tos 0x00, ttl 252, length 152, checksum 0x2ed9
       fragment id 0x0000
10:54:17:240304: ipsec-if-input
   IPSec: spi 3181023528 seq 29
10:54:17:240311: dpdk-esp-decrypt
   cipher aes-cbc-128 auth sha1-96
   ESP: spi 3181023528, seq 29
10:54:17:240323: dpdk-crypto-input
   status: success
10:54:17:240335: dpdk-esp-decrypt-post
   cipher aes-cbc-128 auth sha1-96
   ICMP: 10.1.3.235 -> 10.0.1.253
     tos 0x00, ttl 63, length 84, checksum 0x79df
     fragment id 0xa7e1, flags DONT_FRAGMENT
   ICMP echo_request checksum 0x5047
10:54:17:240338: ip4-input-no-checksum
   ICMP: 10.1.3.235 -> 10.0.1.253
     tos 0x00, ttl 63, length 84, checksum 0x79df
     fragment id 0xa7e1, flags DONT_FRAGMENT
   ICMP echo_request checksum 0x5047
10:54:17:240339: ip4-lookup
   fib 0 dpo-idx 4 flow hash: 0x00000000
   ICMP: 10.1.3.235 -> 10.0.1.253
     tos 0x00, ttl 63, length 84, checksum 0x79df
     fragment id 0xa7e1, flags DONT_FRAGMENT
   ICMP echo_request checksum 0x5047
10:54:17:240341: ip4-rewrite
   tx_sw_if_index 1 dpo-idx 4 : ipv4 via 10.0.1.253 
VirtualFunctionEthernet0/6/0: 0efe85797fa60e0481cb96080800 flow hash: 0x00000000
   00000000: 0efe85797fa60e0481cb9608080045000054a7e140003e017adf0a0103eb0a00
   00000020: 01fd08005047549c00013de0335a00000000210e0200000000001011
10:54:17:240344: VirtualFunctionEthernet0/6/0-output
   VirtualFunctionEthernet0/6/0
   IP4: 0e:04:81:cb:96:08 -> 0e:fe:85:79:7f:a6
   ICMP: 10.1.3.235 -> 10.0.1.253
     tos 0x00, ttl 62, length 84, checksum 0x7adf
     fragment id 0xa7e1, flags DONT_FRAGMENT
   ICMP echo_request checksum 0x5047
10:54:17:240353: VirtualFunctionEthernet0/6/0-tx
   VirtualFunctionEthernet0/6/0 tx queue 0
   buffer 0x2925: current data 44, length 98, free-list 0, clone-count 0, 
totlen-
nifb 0, trace 0x0
                  l4-cksum-computed l4-cksum-correct l2-hdr-offset 0 
l3-hdr-offset 14
   PKT MBUF: port 1, nb_segs 1, pkt_len 98
     buf_len 2176, data_len 98, ol_flags 0x0, data_off 172, phys_addr 0x2c4a49c0
     packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
     Packet Offload Flags
   IP4: 0e:04:81:cb:96:08 -> 0e:fe:85:79:7f:a6
   ICMP: 10.1.3.235 -> 10.0.1.253
     tos 0x00, ttl 62, length 84, checksum 0x7adf
     fragment id 0xa7e1, flags DONT_FRAGMENT
   ICMP echo_request checksum 0x5047

Packet 2

10:54:22:261159: dpdk-input
   VirtualFunctionEthernet0/7/0 rx queue 0
   buffer 0xd8d4: current data 14, length 184, free-list 0, clone-count 0, 
totlen-nifb 0, trace 0x1
                  l4-cksum-computed l4-cksum-correct l2-hdr-offset 0 
l3-hdr-offset 14
   PKT MBUF: port 1, nb_segs 1, pkt_len 198
     buf_len 2176, data_len 198, ol_flags 0x0, data_off 128, phys_addr 
0x24763580
     packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
     Packet Offload Flags
   IP4: 1a:85:68:80:00:01 -> 0e:d2:b9:22:74:fe
   IPSEC_ESP: 10.1.2.79 -> 10.0.0.213
     tos 0x00, ttl 252, length 184, checksum 0x2eb9
     fragment id 0x0000
10:54:22:261177: ip4-input
   IPSEC_ESP: 10.1.2.79 -> 10.0.0.213
     tos 0x00, ttl 252, length 184, checksum 0x2eb9
     fragment id 0x0000
10:54:22:261183: ip4-lookup
   fib 0 dpo-idx 7 flow hash: 0x00000000
   IPSEC_ESP: 10.1.2.79 -> 10.0.0.213
     tos 0x00, ttl 252, length 184, checksum 0x2eb9
     fragment id 0x0000
10:54:22:261188: ip4-local
     IPSEC_ESP: 10.1.2.79 -> 10.0.0.213
       tos 0x00, ttl 252, length 184, checksum 0x2eb9
       fragment id 0x0000
10:54:22:261192: ipsec-if-input
   IPSec: spi 3181023528 seq 44738
10:54:22:261198: dpdk-esp-decrypt
   cipher aes-cbc-128 auth sha1-96
   ESP: spi 3181023528, seq 44738
10:54:22:261205: dpdk-crypto-input
   status: auth failed


-Matt



On Dec 15, 2017, at 4:46 AM, Gonzalez Monroy, Sergio 
<sergio.gonzalez.mon...@intel.com> wrote:

Hi,

Could you show a full verbose trace of both packets from both endpoints?

Thanks,
Sergio

On 14/12/2017 21:26, Matthew Smith wrote:

Hi,

I’m experimenting with IPsec using the AES-NI MB cryptodev and seeing some 
weird behavior.

I have a tunnel up. Both ends of the tunnel are VPP instances, both configured 
to use the AES-NI MB cryptodev. The SAs are established with AES-CBC-128 and 
SHA1.

I can ping from a host on one side of the tunnel to a host on the other side. 
The packets successfully make it across the tunnel, as do the replies. When I 
try to send TCP or UDP packets in either direction, the VPP instance on the 
receiving side fails to decrypt the packet. The trace shows that the auth 
failed. Of note in the packet traces, when TCP or UDP packets arrive at the 
receiving side, the sequence numbers printed in the trace are wrong - i.e. they 
are not what the sending side lists as the current sequence number.

If I ping, then try to send a DNS request, then ping again, I see the following:

successful ping:

02:21:37:455916: dpdk-esp-decrypt
   cipher aes-cbc-128 auth sha1-96
   ESP: spi 1333702992, seq 42
02:21:37:455921: dpdk-crypto-input
   status: success

not successful DNS packet:

02:21:40:710125: dpdk-esp-decrypt
   cipher aes-cbc-128 auth sha1-96
   ESP: spi 1333702992, seq 4641
02:21:40:710129: dpdk-crypto-input
   status: auth failed

successful ping:

02:21:43:857313: dpdk-esp-decrypt
   cipher aes-cbc-128 auth sha1-96
   ESP: spi 1333702992, seq 44
02:21:43:857318: dpdk-crypto-input
   status: success

Before I spend several hours figuring out what’s going on, I thought I’d check 
whether anyone else has seen this issue before or has any theories on what 
might be causing it.

Thanks!
-Matt

_______________________________________________
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev


_______________________________________________
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev

Reply via email to