Hi Sergio, Thanks!
I just submitted https://gerrit.fd.io/r/#/c/9878/ <https://gerrit.fd.io/r/#/c/9878/>. It’s slightly different than the last patch I pasted last time, it just turns off the offload flags instead of resetting all flags. I’m testing with the ENA PMD (drivers/net/ena). I think the PMD sets the offload flags when it receives the packets originally (ena_ethdev.c - ena_rx_mbuf_prepare()). I haven’t looked at offloads with any other PMDs so I don’t know whether that’s a common practice. I am presuming that it’s not common or this problem probably would have occurred before. -Matt > On Dec 19, 2017, at 3:45 AM, Gonzalez Monroy, Sergio > <[email protected]> wrote: > > Hey Matt, > > Good stuff. > > I think the change makes sense, remove any L4 checksum offload when doing > IPsec. > I wonder why we have those offloads on in the first place, but that is a > different issue. > > Regards, > Sergio > > On 18/12/2017 16:41, Matthew Smith wrote: >> Hi Sergio, >> >> I think I identified the problem. >> >> When the UDP packet arrives, there are packet checksum offload flags set on >> the buffer (PKT_TX_TCP_CKSUM, PKT_TX_SCTP_CKSUM). Those can be seen in the >> previous trace I sent. I’m not sure why a UDP packet is resulting in those 2 >> particular flags being set. Maybe the trace function is showing the wrong >> thing, or maybe the PMD is doing something weird. I think after >> encapsulation & encryption the buffer still carries those original offload >> flags when it gets transmitted so the PMD tells the hardware to calculate >> the checksum. The UDP checksum is at the same offset in an IP packet as the >> lower 16 bits of an ESP sequence number is so those 2 bytes are being >> overwritten by the hardware. >> >> The ICMP packet buffers don’t have any offload flags set, which is why those >> packets are being correctly delivered. >> >> I only saw this behavior with the DPDK cryptodev and not the default openssl >> encryption, because the openssl encrypt function esp_encrypt_node_fn() >> allocates a new buffer and initializes the flags to >> VLIB_BUFFER_TOTAL_LENGTH_VALID. When I applied the following patch to reset >> the flags in the same way the default crypto encrypt function does, I >> stopped seeing the ‘auth failed’ on UDP packets and saw those packets arrive >> at the host on the other side of the tunnel. >> >> diff --git a/src/plugins/dpdk/ipsec/esp_encrypt.c >> b/src/plugins/dpdk/ipsec/esp_encrypt.c >> index b4873d4..7bf24a4 100644 >> --- a/src/plugins/dpdk/ipsec/esp_encrypt.c >> +++ b/src/plugins/dpdk/ipsec/esp_encrypt.c >> @@ -265,6 +265,8 @@ dpdk_esp_encrypt_node_fn (vlib_main_t * vm, >> iv_size = cipher_alg->iv_len; >> trunc_size = auth_alg->trunc_size; >> + b0->flags = VLIB_BUFFER_TOTAL_LENGTH_VALID; >> + >> if (sa0->is_tunnel) >> { >> if (!is_ipv6 && !sa0->is_tunnel_ip6) /* ip4inip4 */ >> >> >> I’m not sure if that’s all that needs to change. VPP also crashed a short >> while after I tested with that patch and I haven’t looked into the cause of >> the crash yet, so there may be more to properly fixing the issue than adding >> that statement. >> >> -Matt >> >> >> >>> On Dec 18, 2017, at 4:01 AM, Gonzalez Monroy, Sergio >>> <[email protected]> wrote: >>> >>> Hi Matt, >>> >>> Could you add verbose to the trace? ie. 'trace add dpdk-input 10 verbose' >>> >>> Thanks, >>> Sergio >>> >>> On 15/12/2017 15:11, Matthew Smith wrote: >>>> Hi Sergio, >>>> >>>> Here is the sending side trace: >>>> >>>> Packet 1 >>>> >>>> 10:54:40:291456: dpdk-input >>>> VirtualFunctionEthernet0/6/0 rx queue 0 >>>> buffer 0x6e4f: current data 14, length 84, free-list 0, clone-count 0, >>>> totlen-nifb 0, trace 0x0 >>>> l4-cksum-computed l4-cksum-correct l2-hdr-offset 0 >>>> l3-hdr-offset 14 >>>> PKT MBUF: port 0, nb_segs 1, pkt_len 98 >>>> buf_len 2176, data_len 98, ol_flags 0x0, data_off 128, phys_addr >>>> 0x2c5b9440 >>>> packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 >>>> Packet Offload Flags >>>> IP4: 0e:c6:15:62:1f:9e -> 0e:7e:f2:5d:0d:b6 >>>> ICMP: 10.1.3.235 -> 10.0.1.253 >>>> tos 0x00, ttl 64, length 84, checksum 0x78df >>>> fragment id 0xa7e1, flags DONT_FRAGMENT >>>> ICMP echo_request checksum 0x5047 >>>> 10:54:40:291473: ip4-input >>>> ICMP: 10.1.3.235 -> 10.0.1.253 >>>> tos 0x00, ttl 64, length 84, checksum 0x78df >>>> fragment id 0xa7e1, flags DONT_FRAGMENT >>>> ICMP echo_request checksum 0x5047 >>>> 10:54:40:291485: ip4-lookup >>>> fib 0 dpo-idx 2 flow hash: 0x00000000 >>>> ICMP: 10.1.3.235 -> 10.0.1.253 >>>> tos 0x00, ttl 64, length 84, checksum 0x78df >>>> fragment id 0xa7e1, flags DONT_FRAGMENT >>>> ICMP echo_request checksum 0x5047 >>>> 10:54:40:291491: ip4-rewrite >>>> tx_sw_if_index 3 dpo-idx 2 : ipv4 via 0.0.0.0 ipsec0: flow hash: >>>> 0x00000000 >>>> 00000000: >>>> 45000054a7e140003f0179df0a0103eb0a0001fd08005047549c00013de0335a >>>> 00000020: 00000000210e020000000000101112131415161718191a1b1c1d1e1f >>>> 10:54:40:291495: ipsec-if-output >>>> IPSec: spi 3181023528 seq 28 >>>> 10:54:40:291500: dpdk-esp-encrypt >>>> cipher aes-cbc-128 auth sha1-96 >>>> IPSEC_ESP: 10.1.2.79 -> 10.0.0.213 >>>> tos 0x00, ttl 254, length 152, checksum 0xc0d3 >>>> fragment id 0x0000 >>>> ESP: spi 3181023528, seq 29 >>>> 10:54:40:291514: dpdk-crypto-input >>>> status: success >>>> >>>> Packet 2 >>>> >>>> 10:54:45:312374: dpdk-input >>>> VirtualFunctionEthernet0/6/0 rx queue 0 >>>> buffer 0x6e76: current data 14, length 112, free-list 0, clone-count 0, >>>> totlen-nifb 0, trace 0x1 >>>> l4-cksum-computed l4-cksum-correct l2-hdr-offset 0 >>>> l3-hdr-offset 14 >>>> PKT MBUF: port 0, nb_segs 1, pkt_len 126 >>>> buf_len 2176, data_len 126, ol_flags 0x0, data_off 128, phys_addr >>>> 0x2c5b9e00 >>>> packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 >>>> Packet Offload Flags >>>> PKT_TX_TCP_CKSUM (0x0000) TCP cksum of TX pkt. computed by NIC >>>> PKT_TX_SCTP_CKSUM (0x0000) SCTP cksum of TX pkt. computed by NIC >>>> IP4: 0e:c6:15:62:1f:9e -> 0e:7e:f2:5d:0d:b6 >>>> UDP: 10.1.3.235 -> 10.0.1.253 >>>> tos 0x00, ttl 64, length 112, checksum 0x417c >>>> fragment id 0x1f19 >>>> UDP: 55980 -> 53 >>>> length 92, checksum 0x6c45 >>>> 10:54:45:312393: ip4-input >>>> UDP: 10.1.3.235 -> 10.0.1.253 >>>> tos 0x00, ttl 64, length 112, checksum 0x417c >>>> fragment id 0x1f19 >>>> UDP: 55980 -> 53 >>>> length 92, checksum 0x6c45 >>>> 10:54:45:312400: ip4-lookup >>>> fib 0 dpo-idx 2 flow hash: 0x00000000 >>>> UDP: 10.1.3.235 -> 10.0.1.253 >>>> tos 0x00, ttl 64, length 112, checksum 0x417c >>>> fragment id 0x1f19 >>>> UDP: 55980 -> 53 >>>> length 92, checksum 0x6c45 >>>> 10:54:45:312406: ip4-rewrite >>>> tx_sw_if_index 3 dpo-idx 2 : ipv4 via 0.0.0.0 ipsec0: flow hash: >>>> 0x00000000 >>>> 00000000: >>>> 450000701f1900003f11427c0a0103eb0a0001fddaac0035005c6c45c0210120 >>>> 00000020: 0001000000000001337361666b6a687361666b6a6873666b6a686173 >>>> 10:54:45:312409: ipsec-if-output >>>> IPSec: spi 3181023528 seq 29 >>>> 10:54:45:312414: dpdk-esp-encrypt >>>> cipher aes-cbc-128 auth sha1-96 >>>> IPSEC_ESP: 10.1.2.79 -> 10.0.0.213 >>>> tos 0x00, ttl 254, length 184, checksum 0xc0b3 >>>> fragment id 0x0000 >>>> ESP: spi 3181023528, seq 30 >>>> 10:54:45:312425: dpdk-crypto-input >>>> status: success >>>> >>>> >>>> And the receiving side: >>>> >>>> Packet 1 >>>> >>>> 10:54:17:240273: dpdk-input >>>> VirtualFunctionEthernet0/7/0 rx queue 0 >>>> buffer 0x2925: current data 14, length 152, free-list 0, clone-count 0, >>>> totlen-nifb 0, trace 0x0 >>>> l4-cksum-computed l4-cksum-correct l2-hdr-offset 0 >>>> l3-hdr-offset 14 >>>> PKT MBUF: port 1, nb_segs 1, pkt_len 166 >>>> buf_len 2176, data_len 166, ol_flags 0x0, data_off 128, phys_addr >>>> 0x2c4a49c0 >>>> packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 >>>> Packet Offload Flags >>>> IP4: 1a:85:68:80:00:01 -> 0e:d2:b9:22:74:fe >>>> IPSEC_ESP: 10.1.2.79 -> 10.0.0.213 >>>> tos 0x00, ttl 252, length 152, checksum 0x2ed9 >>>> fragment id 0x0000 >>>> 10:54:17:240290: ip4-input >>>> IPSEC_ESP: 10.1.2.79 -> 10.0.0.213 >>>> tos 0x00, ttl 252, length 152, checksum 0x2ed9 >>>> fragment id 0x0000 >>>> 10:54:17:240295: ip4-lookup >>>> fib 0 dpo-idx 7 flow hash: 0x00000000 >>>> IPSEC_ESP: 10.1.2.79 -> 10.0.0.213 >>>> tos 0x00, ttl 252, length 152, checksum 0x2ed9 >>>> fragment id 0x0000 >>>> 10:54:17:240301: ip4-local >>>> IPSEC_ESP: 10.1.2.79 -> 10.0.0.213 >>>> tos 0x00, ttl 252, length 152, checksum 0x2ed9 >>>> fragment id 0x0000 >>>> 10:54:17:240304: ipsec-if-input >>>> IPSec: spi 3181023528 seq 29 >>>> 10:54:17:240311: dpdk-esp-decrypt >>>> cipher aes-cbc-128 auth sha1-96 >>>> ESP: spi 3181023528, seq 29 >>>> 10:54:17:240323: dpdk-crypto-input >>>> status: success >>>> 10:54:17:240335: dpdk-esp-decrypt-post >>>> cipher aes-cbc-128 auth sha1-96 >>>> ICMP: 10.1.3.235 -> 10.0.1.253 >>>> tos 0x00, ttl 63, length 84, checksum 0x79df >>>> fragment id 0xa7e1, flags DONT_FRAGMENT >>>> ICMP echo_request checksum 0x5047 >>>> 10:54:17:240338: ip4-input-no-checksum >>>> ICMP: 10.1.3.235 -> 10.0.1.253 >>>> tos 0x00, ttl 63, length 84, checksum 0x79df >>>> fragment id 0xa7e1, flags DONT_FRAGMENT >>>> ICMP echo_request checksum 0x5047 >>>> 10:54:17:240339: ip4-lookup >>>> fib 0 dpo-idx 4 flow hash: 0x00000000 >>>> ICMP: 10.1.3.235 -> 10.0.1.253 >>>> tos 0x00, ttl 63, length 84, checksum 0x79df >>>> fragment id 0xa7e1, flags DONT_FRAGMENT >>>> ICMP echo_request checksum 0x5047 >>>> 10:54:17:240341: ip4-rewrite >>>> tx_sw_if_index 1 dpo-idx 4 : ipv4 via 10.0.1.253 >>>> VirtualFunctionEthernet0/6/0: 0efe85797fa60e0481cb96080800 flow hash: >>>> 0x00000000 >>>> 00000000: >>>> 0efe85797fa60e0481cb9608080045000054a7e140003e017adf0a0103eb0a00 >>>> 00000020: 01fd08005047549c00013de0335a00000000210e0200000000001011 >>>> 10:54:17:240344: VirtualFunctionEthernet0/6/0-output >>>> VirtualFunctionEthernet0/6/0 >>>> IP4: 0e:04:81:cb:96:08 -> 0e:fe:85:79:7f:a6 >>>> ICMP: 10.1.3.235 -> 10.0.1.253 >>>> tos 0x00, ttl 62, length 84, checksum 0x7adf >>>> fragment id 0xa7e1, flags DONT_FRAGMENT >>>> ICMP echo_request checksum 0x5047 >>>> 10:54:17:240353: VirtualFunctionEthernet0/6/0-tx >>>> VirtualFunctionEthernet0/6/0 tx queue 0 >>>> buffer 0x2925: current data 44, length 98, free-list 0, clone-count 0, >>>> totlen- >>>> nifb 0, trace 0x0 >>>> l4-cksum-computed l4-cksum-correct l2-hdr-offset 0 >>>> l3-hdr-offset 14 >>>> PKT MBUF: port 1, nb_segs 1, pkt_len 98 >>>> buf_len 2176, data_len 98, ol_flags 0x0, data_off 172, phys_addr >>>> 0x2c4a49c0 >>>> packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 >>>> Packet Offload Flags >>>> IP4: 0e:04:81:cb:96:08 -> 0e:fe:85:79:7f:a6 >>>> ICMP: 10.1.3.235 -> 10.0.1.253 >>>> tos 0x00, ttl 62, length 84, checksum 0x7adf >>>> fragment id 0xa7e1, flags DONT_FRAGMENT >>>> ICMP echo_request checksum 0x5047 >>>> >>>> Packet 2 >>>> >>>> 10:54:22:261159: dpdk-input >>>> VirtualFunctionEthernet0/7/0 rx queue 0 >>>> buffer 0xd8d4: current data 14, length 184, free-list 0, clone-count 0, >>>> totlen-nifb 0, trace 0x1 >>>> l4-cksum-computed l4-cksum-correct l2-hdr-offset 0 >>>> l3-hdr-offset 14 >>>> PKT MBUF: port 1, nb_segs 1, pkt_len 198 >>>> buf_len 2176, data_len 198, ol_flags 0x0, data_off 128, phys_addr >>>> 0x24763580 >>>> packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 >>>> Packet Offload Flags >>>> IP4: 1a:85:68:80:00:01 -> 0e:d2:b9:22:74:fe >>>> IPSEC_ESP: 10.1.2.79 -> 10.0.0.213 >>>> tos 0x00, ttl 252, length 184, checksum 0x2eb9 >>>> fragment id 0x0000 >>>> 10:54:22:261177: ip4-input >>>> IPSEC_ESP: 10.1.2.79 -> 10.0.0.213 >>>> tos 0x00, ttl 252, length 184, checksum 0x2eb9 >>>> fragment id 0x0000 >>>> 10:54:22:261183: ip4-lookup >>>> fib 0 dpo-idx 7 flow hash: 0x00000000 >>>> IPSEC_ESP: 10.1.2.79 -> 10.0.0.213 >>>> tos 0x00, ttl 252, length 184, checksum 0x2eb9 >>>> fragment id 0x0000 >>>> 10:54:22:261188: ip4-local >>>> IPSEC_ESP: 10.1.2.79 -> 10.0.0.213 >>>> tos 0x00, ttl 252, length 184, checksum 0x2eb9 >>>> fragment id 0x0000 >>>> 10:54:22:261192: ipsec-if-input >>>> IPSec: spi 3181023528 seq 44738 >>>> 10:54:22:261198: dpdk-esp-decrypt >>>> cipher aes-cbc-128 auth sha1-96 >>>> ESP: spi 3181023528, seq 44738 >>>> 10:54:22:261205: dpdk-crypto-input >>>> status: auth failed >>>> >>>> >>>> -Matt >>>> >>>> >>>>> On Dec 15, 2017, at 4:46 AM, Gonzalez Monroy, Sergio >>>>> <[email protected]> wrote: >>>>> >>>>> Hi, >>>>> >>>>> Could you show a full verbose trace of both packets from both endpoints? >>>>> >>>>> Thanks, >>>>> Sergio >>>>> >>>>> On 14/12/2017 21:26, Matthew Smith wrote: >>>>>> Hi, >>>>>> >>>>>> I’m experimenting with IPsec using the AES-NI MB cryptodev and seeing >>>>>> some weird behavior. >>>>>> >>>>>> I have a tunnel up. Both ends of the tunnel are VPP instances, both >>>>>> configured to use the AES-NI MB cryptodev. The SAs are established with >>>>>> AES-CBC-128 and SHA1. >>>>>> >>>>>> I can ping from a host on one side of the tunnel to a host on the other >>>>>> side. The packets successfully make it across the tunnel, as do the >>>>>> replies. When I try to send TCP or UDP packets in either direction, the >>>>>> VPP instance on the receiving side fails to decrypt the packet. The >>>>>> trace shows that the auth failed. Of note in the packet traces, when TCP >>>>>> or UDP packets arrive at the receiving side, the sequence numbers >>>>>> printed in the trace are wrong - i.e. they are not what the sending side >>>>>> lists as the current sequence number. >>>>>> >>>>>> If I ping, then try to send a DNS request, then ping again, I see the >>>>>> following: >>>>>> >>>>>> successful ping: >>>>>> >>>>>> 02:21:37:455916: dpdk-esp-decrypt >>>>>> cipher aes-cbc-128 auth sha1-96 >>>>>> ESP: spi 1333702992, seq 42 >>>>>> 02:21:37:455921: dpdk-crypto-input >>>>>> status: success >>>>>> >>>>>> not successful DNS packet: >>>>>> >>>>>> 02:21:40:710125: dpdk-esp-decrypt >>>>>> cipher aes-cbc-128 auth sha1-96 >>>>>> ESP: spi 1333702992, seq 4641 >>>>>> 02:21:40:710129: dpdk-crypto-input >>>>>> status: auth failed >>>>>> >>>>>> successful ping: >>>>>> >>>>>> 02:21:43:857313: dpdk-esp-decrypt >>>>>> cipher aes-cbc-128 auth sha1-96 >>>>>> ESP: spi 1333702992, seq 44 >>>>>> 02:21:43:857318: dpdk-crypto-input >>>>>> status: success >>>>>> >>>>>> Before I spend several hours figuring out what’s going on, I thought I’d >>>>>> check whether anyone else has seen this issue before or has any theories >>>>>> on what might be causing it. >>>>>> >>>>>> Thanks! >>>>>> -Matt >>>>>> >>>>>> _______________________________________________ >>>>>> vpp-dev mailing list >>>>>> [email protected] >>>>>> https://lists.fd.io/mailman/listinfo/vpp-dev >
_______________________________________________ vpp-dev mailing list [email protected] https://lists.fd.io/mailman/listinfo/vpp-dev
