Hi Sergio,
I think I identified the problem.
When the UDP packet arrives, there are packet checksum offload flags set on the
buffer (PKT_TX_TCP_CKSUM, PKT_TX_SCTP_CKSUM). Those can be seen in the previous
trace I sent. I’m not sure why a UDP packet is resulting in those 2 particular
flags being set. Maybe the trace function is showing the wrong thing, or maybe
the PMD is doing something weird. I think after encapsulation & encryption the
buffer still carries those original offload flags when it gets transmitted so
the PMD tells the hardware to calculate the checksum. The UDP checksum is at
the same offset in an IP packet as the lower 16 bits of an ESP sequence number
is so those 2 bytes are being overwritten by the hardware.
The ICMP packet buffers don’t have any offload flags set, which is why those
packets are being correctly delivered.
I only saw this behavior with the DPDK cryptodev and not the default openssl
encryption, because the openssl encrypt function esp_encrypt_node_fn()
allocates a new buffer and initializes the flags to
VLIB_BUFFER_TOTAL_LENGTH_VALID. When I applied the following patch to reset the
flags in the same way the default crypto encrypt function does, I stopped
seeing the ‘auth failed’ on UDP packets and saw those packets arrive at the
host on the other side of the tunnel.
diff --git a/src/plugins/dpdk/ipsec/esp_encrypt.c
b/src/plugins/dpdk/ipsec/esp_encrypt.c
index b4873d4..7bf24a4 100644
--- a/src/plugins/dpdk/ipsec/esp_encrypt.c
+++ b/src/plugins/dpdk/ipsec/esp_encrypt.c
@@ -265,6 +265,8 @@ dpdk_esp_encrypt_node_fn (vlib_main_t * vm,
iv_size = cipher_alg->iv_len;
trunc_size = auth_alg->trunc_size;
+ b0->flags = VLIB_BUFFER_TOTAL_LENGTH_VALID;
+
if (sa0->is_tunnel)
{
if (!is_ipv6 && !sa0->is_tunnel_ip6) /* ip4inip4 */
I’m not sure if that’s all that needs to change. VPP also crashed a short while
after I tested with that patch and I haven’t looked into the cause of the crash
yet, so there may be more to properly fixing the issue than adding that
statement.
-Matt
> On Dec 18, 2017, at 4:01 AM, Gonzalez Monroy, Sergio
> <[email protected]> wrote:
>
> Hi Matt,
>
> Could you add verbose to the trace? ie. 'trace add dpdk-input 10 verbose'
>
> Thanks,
> Sergio
>
> On 15/12/2017 15:11, Matthew Smith wrote:
>> Hi Sergio,
>>
>> Here is the sending side trace:
>>
>> Packet 1
>>
>> 10:54:40:291456: dpdk-input
>> VirtualFunctionEthernet0/6/0 rx queue 0
>> buffer 0x6e4f: current data 14, length 84, free-list 0, clone-count 0,
>> totlen-nifb 0, trace 0x0
>> l4-cksum-computed l4-cksum-correct l2-hdr-offset 0
>> l3-hdr-offset 14
>> PKT MBUF: port 0, nb_segs 1, pkt_len 98
>> buf_len 2176, data_len 98, ol_flags 0x0, data_off 128, phys_addr
>> 0x2c5b9440
>> packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
>> Packet Offload Flags
>> IP4: 0e:c6:15:62:1f:9e -> 0e:7e:f2:5d:0d:b6
>> ICMP: 10.1.3.235 -> 10.0.1.253
>> tos 0x00, ttl 64, length 84, checksum 0x78df
>> fragment id 0xa7e1, flags DONT_FRAGMENT
>> ICMP echo_request checksum 0x5047
>> 10:54:40:291473: ip4-input
>> ICMP: 10.1.3.235 -> 10.0.1.253
>> tos 0x00, ttl 64, length 84, checksum 0x78df
>> fragment id 0xa7e1, flags DONT_FRAGMENT
>> ICMP echo_request checksum 0x5047
>> 10:54:40:291485: ip4-lookup
>> fib 0 dpo-idx 2 flow hash: 0x00000000
>> ICMP: 10.1.3.235 -> 10.0.1.253
>> tos 0x00, ttl 64, length 84, checksum 0x78df
>> fragment id 0xa7e1, flags DONT_FRAGMENT
>> ICMP echo_request checksum 0x5047
>> 10:54:40:291491: ip4-rewrite
>> tx_sw_if_index 3 dpo-idx 2 : ipv4 via 0.0.0.0 ipsec0: flow hash:
>> 0x00000000
>> 00000000: 45000054a7e140003f0179df0a0103eb0a0001fd08005047549c00013de0335a
>> 00000020: 00000000210e020000000000101112131415161718191a1b1c1d1e1f
>> 10:54:40:291495: ipsec-if-output
>> IPSec: spi 3181023528 seq 28
>> 10:54:40:291500: dpdk-esp-encrypt
>> cipher aes-cbc-128 auth sha1-96
>> IPSEC_ESP: 10.1.2.79 -> 10.0.0.213
>> tos 0x00, ttl 254, length 152, checksum 0xc0d3
>> fragment id 0x0000
>> ESP: spi 3181023528, seq 29
>> 10:54:40:291514: dpdk-crypto-input
>> status: success
>>
>> Packet 2
>>
>> 10:54:45:312374: dpdk-input
>> VirtualFunctionEthernet0/6/0 rx queue 0
>> buffer 0x6e76: current data 14, length 112, free-list 0, clone-count 0,
>> totlen-nifb 0, trace 0x1
>> l4-cksum-computed l4-cksum-correct l2-hdr-offset 0
>> l3-hdr-offset 14
>> PKT MBUF: port 0, nb_segs 1, pkt_len 126
>> buf_len 2176, data_len 126, ol_flags 0x0, data_off 128, phys_addr
>> 0x2c5b9e00
>> packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
>> Packet Offload Flags
>> PKT_TX_TCP_CKSUM (0x0000) TCP cksum of TX pkt. computed by NIC
>> PKT_TX_SCTP_CKSUM (0x0000) SCTP cksum of TX pkt. computed by NIC
>> IP4: 0e:c6:15:62:1f:9e -> 0e:7e:f2:5d:0d:b6
>> UDP: 10.1.3.235 -> 10.0.1.253
>> tos 0x00, ttl 64, length 112, checksum 0x417c
>> fragment id 0x1f19
>> UDP: 55980 -> 53
>> length 92, checksum 0x6c45
>> 10:54:45:312393: ip4-input
>> UDP: 10.1.3.235 -> 10.0.1.253
>> tos 0x00, ttl 64, length 112, checksum 0x417c
>> fragment id 0x1f19
>> UDP: 55980 -> 53
>> length 92, checksum 0x6c45
>> 10:54:45:312400: ip4-lookup
>> fib 0 dpo-idx 2 flow hash: 0x00000000
>> UDP: 10.1.3.235 -> 10.0.1.253
>> tos 0x00, ttl 64, length 112, checksum 0x417c
>> fragment id 0x1f19
>> UDP: 55980 -> 53
>> length 92, checksum 0x6c45
>> 10:54:45:312406: ip4-rewrite
>> tx_sw_if_index 3 dpo-idx 2 : ipv4 via 0.0.0.0 ipsec0: flow hash:
>> 0x00000000
>> 00000000: 450000701f1900003f11427c0a0103eb0a0001fddaac0035005c6c45c0210120
>> 00000020: 0001000000000001337361666b6a687361666b6a6873666b6a686173
>> 10:54:45:312409: ipsec-if-output
>> IPSec: spi 3181023528 seq 29
>> 10:54:45:312414: dpdk-esp-encrypt
>> cipher aes-cbc-128 auth sha1-96
>> IPSEC_ESP: 10.1.2.79 -> 10.0.0.213
>> tos 0x00, ttl 254, length 184, checksum 0xc0b3
>> fragment id 0x0000
>> ESP: spi 3181023528, seq 30
>> 10:54:45:312425: dpdk-crypto-input
>> status: success
>>
>>
>> And the receiving side:
>>
>> Packet 1
>>
>> 10:54:17:240273: dpdk-input
>> VirtualFunctionEthernet0/7/0 rx queue 0
>> buffer 0x2925: current data 14, length 152, free-list 0, clone-count 0,
>> totlen-nifb 0, trace 0x0
>> l4-cksum-computed l4-cksum-correct l2-hdr-offset 0
>> l3-hdr-offset 14
>> PKT MBUF: port 1, nb_segs 1, pkt_len 166
>> buf_len 2176, data_len 166, ol_flags 0x0, data_off 128, phys_addr
>> 0x2c4a49c0
>> packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
>> Packet Offload Flags
>> IP4: 1a:85:68:80:00:01 -> 0e:d2:b9:22:74:fe
>> IPSEC_ESP: 10.1.2.79 -> 10.0.0.213
>> tos 0x00, ttl 252, length 152, checksum 0x2ed9
>> fragment id 0x0000
>> 10:54:17:240290: ip4-input
>> IPSEC_ESP: 10.1.2.79 -> 10.0.0.213
>> tos 0x00, ttl 252, length 152, checksum 0x2ed9
>> fragment id 0x0000
>> 10:54:17:240295: ip4-lookup
>> fib 0 dpo-idx 7 flow hash: 0x00000000
>> IPSEC_ESP: 10.1.2.79 -> 10.0.0.213
>> tos 0x00, ttl 252, length 152, checksum 0x2ed9
>> fragment id 0x0000
>> 10:54:17:240301: ip4-local
>> IPSEC_ESP: 10.1.2.79 -> 10.0.0.213
>> tos 0x00, ttl 252, length 152, checksum 0x2ed9
>> fragment id 0x0000
>> 10:54:17:240304: ipsec-if-input
>> IPSec: spi 3181023528 seq 29
>> 10:54:17:240311: dpdk-esp-decrypt
>> cipher aes-cbc-128 auth sha1-96
>> ESP: spi 3181023528, seq 29
>> 10:54:17:240323: dpdk-crypto-input
>> status: success
>> 10:54:17:240335: dpdk-esp-decrypt-post
>> cipher aes-cbc-128 auth sha1-96
>> ICMP: 10.1.3.235 -> 10.0.1.253
>> tos 0x00, ttl 63, length 84, checksum 0x79df
>> fragment id 0xa7e1, flags DONT_FRAGMENT
>> ICMP echo_request checksum 0x5047
>> 10:54:17:240338: ip4-input-no-checksum
>> ICMP: 10.1.3.235 -> 10.0.1.253
>> tos 0x00, ttl 63, length 84, checksum 0x79df
>> fragment id 0xa7e1, flags DONT_FRAGMENT
>> ICMP echo_request checksum 0x5047
>> 10:54:17:240339: ip4-lookup
>> fib 0 dpo-idx 4 flow hash: 0x00000000
>> ICMP: 10.1.3.235 -> 10.0.1.253
>> tos 0x00, ttl 63, length 84, checksum 0x79df
>> fragment id 0xa7e1, flags DONT_FRAGMENT
>> ICMP echo_request checksum 0x5047
>> 10:54:17:240341: ip4-rewrite
>> tx_sw_if_index 1 dpo-idx 4 : ipv4 via 10.0.1.253
>> VirtualFunctionEthernet0/6/0: 0efe85797fa60e0481cb96080800 flow hash:
>> 0x00000000
>> 00000000: 0efe85797fa60e0481cb9608080045000054a7e140003e017adf0a0103eb0a00
>> 00000020: 01fd08005047549c00013de0335a00000000210e0200000000001011
>> 10:54:17:240344: VirtualFunctionEthernet0/6/0-output
>> VirtualFunctionEthernet0/6/0
>> IP4: 0e:04:81:cb:96:08 -> 0e:fe:85:79:7f:a6
>> ICMP: 10.1.3.235 -> 10.0.1.253
>> tos 0x00, ttl 62, length 84, checksum 0x7adf
>> fragment id 0xa7e1, flags DONT_FRAGMENT
>> ICMP echo_request checksum 0x5047
>> 10:54:17:240353: VirtualFunctionEthernet0/6/0-tx
>> VirtualFunctionEthernet0/6/0 tx queue 0
>> buffer 0x2925: current data 44, length 98, free-list 0, clone-count 0,
>> totlen-
>> nifb 0, trace 0x0
>> l4-cksum-computed l4-cksum-correct l2-hdr-offset 0
>> l3-hdr-offset 14
>> PKT MBUF: port 1, nb_segs 1, pkt_len 98
>> buf_len 2176, data_len 98, ol_flags 0x0, data_off 172, phys_addr
>> 0x2c4a49c0
>> packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
>> Packet Offload Flags
>> IP4: 0e:04:81:cb:96:08 -> 0e:fe:85:79:7f:a6
>> ICMP: 10.1.3.235 -> 10.0.1.253
>> tos 0x00, ttl 62, length 84, checksum 0x7adf
>> fragment id 0xa7e1, flags DONT_FRAGMENT
>> ICMP echo_request checksum 0x5047
>>
>> Packet 2
>>
>> 10:54:22:261159: dpdk-input
>> VirtualFunctionEthernet0/7/0 rx queue 0
>> buffer 0xd8d4: current data 14, length 184, free-list 0, clone-count 0,
>> totlen-nifb 0, trace 0x1
>> l4-cksum-computed l4-cksum-correct l2-hdr-offset 0
>> l3-hdr-offset 14
>> PKT MBUF: port 1, nb_segs 1, pkt_len 198
>> buf_len 2176, data_len 198, ol_flags 0x0, data_off 128, phys_addr
>> 0x24763580
>> packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
>> Packet Offload Flags
>> IP4: 1a:85:68:80:00:01 -> 0e:d2:b9:22:74:fe
>> IPSEC_ESP: 10.1.2.79 -> 10.0.0.213
>> tos 0x00, ttl 252, length 184, checksum 0x2eb9
>> fragment id 0x0000
>> 10:54:22:261177: ip4-input
>> IPSEC_ESP: 10.1.2.79 -> 10.0.0.213
>> tos 0x00, ttl 252, length 184, checksum 0x2eb9
>> fragment id 0x0000
>> 10:54:22:261183: ip4-lookup
>> fib 0 dpo-idx 7 flow hash: 0x00000000
>> IPSEC_ESP: 10.1.2.79 -> 10.0.0.213
>> tos 0x00, ttl 252, length 184, checksum 0x2eb9
>> fragment id 0x0000
>> 10:54:22:261188: ip4-local
>> IPSEC_ESP: 10.1.2.79 -> 10.0.0.213
>> tos 0x00, ttl 252, length 184, checksum 0x2eb9
>> fragment id 0x0000
>> 10:54:22:261192: ipsec-if-input
>> IPSec: spi 3181023528 seq 44738
>> 10:54:22:261198: dpdk-esp-decrypt
>> cipher aes-cbc-128 auth sha1-96
>> ESP: spi 3181023528, seq 44738
>> 10:54:22:261205: dpdk-crypto-input
>> status: auth failed
>>
>>
>> -Matt
>>
>>
>>> On Dec 15, 2017, at 4:46 AM, Gonzalez Monroy, Sergio
>>> <[email protected]> wrote:
>>>
>>> Hi,
>>>
>>> Could you show a full verbose trace of both packets from both endpoints?
>>>
>>> Thanks,
>>> Sergio
>>>
>>> On 14/12/2017 21:26, Matthew Smith wrote:
>>>> Hi,
>>>>
>>>> I’m experimenting with IPsec using the AES-NI MB cryptodev and seeing some
>>>> weird behavior.
>>>>
>>>> I have a tunnel up. Both ends of the tunnel are VPP instances, both
>>>> configured to use the AES-NI MB cryptodev. The SAs are established with
>>>> AES-CBC-128 and SHA1.
>>>>
>>>> I can ping from a host on one side of the tunnel to a host on the other
>>>> side. The packets successfully make it across the tunnel, as do the
>>>> replies. When I try to send TCP or UDP packets in either direction, the
>>>> VPP instance on the receiving side fails to decrypt the packet. The trace
>>>> shows that the auth failed. Of note in the packet traces, when TCP or UDP
>>>> packets arrive at the receiving side, the sequence numbers printed in the
>>>> trace are wrong - i.e. they are not what the sending side lists as the
>>>> current sequence number.
>>>>
>>>> If I ping, then try to send a DNS request, then ping again, I see the
>>>> following:
>>>>
>>>> successful ping:
>>>>
>>>> 02:21:37:455916: dpdk-esp-decrypt
>>>> cipher aes-cbc-128 auth sha1-96
>>>> ESP: spi 1333702992, seq 42
>>>> 02:21:37:455921: dpdk-crypto-input
>>>> status: success
>>>>
>>>> not successful DNS packet:
>>>>
>>>> 02:21:40:710125: dpdk-esp-decrypt
>>>> cipher aes-cbc-128 auth sha1-96
>>>> ESP: spi 1333702992, seq 4641
>>>> 02:21:40:710129: dpdk-crypto-input
>>>> status: auth failed
>>>>
>>>> successful ping:
>>>>
>>>> 02:21:43:857313: dpdk-esp-decrypt
>>>> cipher aes-cbc-128 auth sha1-96
>>>> ESP: spi 1333702992, seq 44
>>>> 02:21:43:857318: dpdk-crypto-input
>>>> status: success
>>>>
>>>> Before I spend several hours figuring out what’s going on, I thought I’d
>>>> check whether anyone else has seen this issue before or has any theories
>>>> on what might be causing it.
>>>>
>>>> Thanks!
>>>> -Matt
>>>>
>>>> _______________________________________________
>>>> vpp-dev mailing list
>>>> [email protected]
>>>> https://lists.fd.io/mailman/listinfo/vpp-dev
>
_______________________________________________
vpp-dev mailing list
[email protected]
https://lists.fd.io/mailman/listinfo/vpp-dev
