[vpp-dev] a few bugfixes
Hi folks, I've finally got around to submitting a few bugfixes that have been sitting in our downstream repo for a while. Please review this CRs, if possible. Some of these patches replace older ones that got merge conflicts. Here they are: misc: pass NULL instead of 0 for pointer in variadic functions https://gerrit.fd.io/r/c/vpp/+/36588 vppinfra: fix memory leak in sparse_vec_free() https://gerrit.fd.io/r/c/vpp/+/36589 nat: fix handling checksum offload in nat44-ed https://gerrit.fd.io/r/c/vpp/+/36590 gre: fix returning the flags in the API https://gerrit.fd.io/r/c/vpp/+/36591 stats: handle interface renames properly https://gerrit.fd.io/r/c/vpp/+/36592 -- Ivan Shvedunov ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 2807 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#21614): https://lists.fd.io/g/vpp-dev/message/21614 Mute This Topic: https://lists.fd.io/mt/92198326/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] renaming tables
Hi folks, I have a question regarding "ip/ip6 table add" CLI command and the corresponding ip_table_add_del API call. There's a possibility to pass the "name" argument to the CLI command (even though it's omitted from its help): https://github.com/FDio/vpp/blob/0891b6aa449cca525b61d0cc23759b2efcd158dc/src/vnet/ip/lookup.c#L412 and also to the call https://github.com/FDio/vpp/blob/0891b6aa449cca525b61d0cc23759b2efcd158dc/src/vnet/ip/ip.api#L44 When "ip/ip6 table add" command is called repeatedly for the same table, fib_table_find_or_create_and_lock_i() is called each time. But this function doesn't replace the table name if it's already set: https://github.com/FDio/vpp/blob/0891b6aa449cca525b61d0cc23759b2efcd158dc/src/vnet/fib/fib_table.c#L1152-L1164 My question is: is this the intended behavior so that the table name (description) can't be changed once set? If it isn't, I'd gladly provide a patch. Or, should I add another API call (and perhaps a CLI command too) to rename tables? (Or maybe there is one already that I missed?) Thanks, Ivan -- Ivan Shvedunov ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 2807 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#21406): https://lists.fd.io/g/vpp-dev/message/21406 Mute This Topic: https://lists.fd.io/mt/91058465/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] patch: fix cxq_vring field in virtio (affects taps)
Hi Mohsin, thanks for the patches. I've abandoned mine in favor of these. On Thu, Sep 23, 2021 at 12:23 AM Mohsin Kazmi (sykazmi) wrote: > Hi Ivan, > > > > Thank you so much for reporting the issue. cxq_vring field is specific to > virtio pci. It should not be accessed or set in tap driver. > > Please find the proper fix here: > >1. https://gerrit.fd.io/r/c/vpp/+/33798/1 >2. https://gerrit.fd.io/r/c/vpp/+/33796/3 > > > > -br > > Mohsin > > > > *From: * on behalf of Ivan Shvedunov < > ivan...@gmail.com> > *Date: *Wednesday, September 22, 2021 at 2:10 PM > *To: *vpp-dev > *Subject: *[vpp-dev] patch: fix cxq_vring field in virtio (affects taps) > > > > Hi, > > I've noticed that VPP returns bad host IPv6 addresses from > sw_interface_tap_v2_dump() API call. As it turned out, the problem is due > to this line: > > > https://github.com/FDio/vpp/blob/ef356f57b54b948d990b293514f062aebf86da72/src/vnet/devices/tap/tap.c#L743 > > with cxq_vring field belonging to the part of a union inside virtio_if_t > that's only used for virtio_pci, but also coincides with host_ip6_addr in > the other part of the union. As there are more code paths that use this > field without checking virtio type, for example in virtio_show(), I thought > it's probably safer to move it out of the union: > > https://gerrit.fd.io/r/c/vpp/+/33791 > > This patch fixes the issue with TAP details. > > > -- > > Ivan Shvedunov > ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 > 2807 > -- Ivan Shvedunov ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 2807 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20195): https://lists.fd.io/g/vpp-dev/message/20195 Mute This Topic: https://lists.fd.io/mt/85789362/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] patch: fix cxq_vring field in virtio (affects taps)
Hi, I've noticed that VPP returns bad host IPv6 addresses from sw_interface_tap_v2_dump() API call. As it turned out, the problem is due to this line: https://github.com/FDio/vpp/blob/ef356f57b54b948d990b293514f062aebf86da72/src/vnet/devices/tap/tap.c#L743 with cxq_vring field belonging to the part of a union inside virtio_if_t that's only used for virtio_pci, but also coincides with host_ip6_addr in the other part of the union. As there are more code paths that use this field without checking virtio type, for example in virtio_show(), I thought it's probably safer to move it out of the union: https://gerrit.fd.io/r/c/vpp/+/33791 This patch fixes the issue with TAP details. -- Ivan Shvedunov ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 2807 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20179): https://lists.fd.io/g/vpp-dev/message/20179 Mute This Topic: https://lists.fd.io/mt/85789362/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] problem with updating incomplete adj nbr entries upon MAC address change
Hi Neale, inline On Wed, Aug 11, 2021 at 6:36 PM Neale Ranns wrote: > Dear Ivan, > > > > That’s the best bug report I’ve seen in ages, so thank you! > You're welcome :) > I’ll take your patch as a starting point, but I’ll think I move the > responsibility for walking the incomplete adjacencies into the adj code; it > seems a little unfair for the ip-neighbour module to be responsible for > updating state it does not own. > > Can I add a ‘signed-off’ line with this email in the commit so you get the > deserved credit? > Sure, that would be nice, thanks! > > /neale > > > > > > *From: *vpp-dev@lists.fd.io on behalf of Ivan > Shvedunov via lists.fd.io > *Date: *Wednesday, 11 August 2021 at 15:06 > *To: *vpp-dev > *Subject: *[vpp-dev] problem with updating incomplete adj nbr entries > upon MAC address change > > Hi, > > I've stumbled upon a problem in the neighbor adj code that handles > incomplete entries. These entries are pre-created under some circumstances > such as creation of a VXLAN tunnel and used to make Ethernet headers for > outbound ARP requests. Problem is, when the MAC address of the interface > changes, these entries are never updated. > > I've posted the detailed description and the patch as a GitHub gist [1]. > > Sadly, my fd.io Gerrit account is completely broken after what turned out > to be an unsuccessful login fix by the LF infra (IT-22698, IT-22257) so I > can't post the patch myself, sorry. > > > > [1] https://gist.github.com/ivan4th/d72d08be0fc53228cd4e7c6f19926c09 > > > > -- > > Ivan Shvedunov > ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 > 2807 > -- Ivan Shvedunov ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 2807 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19958): https://lists.fd.io/g/vpp-dev/message/19958 Mute This Topic: https://lists.fd.io/mt/84815615/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] problem with updating incomplete adj nbr entries upon MAC address change
Hi, I've stumbled upon a problem in the neighbor adj code that handles incomplete entries. These entries are pre-created under some circumstances such as creation of a VXLAN tunnel and used to make Ethernet headers for outbound ARP requests. Problem is, when the MAC address of the interface changes, these entries are never updated. I've posted the detailed description and the patch as a GitHub gist [1]. Sadly, my fd.io Gerrit account is completely broken after what turned out to be an unsuccessful login fix by the LF infra (IT-22698, IT-22257) so I can't post the patch myself, sorry. [1] https://gist.github.com/ivan4th/d72d08be0fc53228cd4e7c6f19926c09 -- Ivan Shvedunov ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 2807 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19955): https://lists.fd.io/g/vpp-dev/message/19955 Mute This Topic: https://lists.fd.io/mt/84815615/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] ip4 length > l2 length with memif + TRex
Hi, I have debugged the issue mentioned in the original message further and found a couple of bugs in the DPDK memif driver. There's a bug in mbuf chaining code [1] and also another bug in memif_stats_get() function [2] for which I made downstream PRs for TRex [3][4]. This is not in any way a VPP problem, but I hope if someone trying to use TRex or another DPDK app with VPP and memifs stumbles upon this message it might be of some help (as far as I know, TRex is used in VPP CSIT to some degree). [1] https://bugs.dpdk.org/show_bug.cgi?id=729 [2] https://bugs.dpdk.org/show_bug.cgi?id=734 [3] https://github.com/cisco-system-traffic-generator/trex-core/pull/723 [4] https://github.com/cisco-system-traffic-generator/trex-core/pull/728 On Wed, Apr 21, 2021 at 5:08 PM Ivan Shvedunov via lists.fd.io wrote: > Hello, > > I've been struggling with the following issue (observed with VPP 21.01): > when trying any of ASTF TRex profiles with VPP, I get multiple errors: > "ip4 length > l2 length". > These don't happen with each packet but rather with some of them. It looks > like the TCP payload is truncated down to 600 bytes for some reason, while > looking at the TRex simulation, it should be 1448 bytes. > I have collected a snippet from dispatch trace dissected by Wireshark > and other info here: > https://gist.github.com/ivan4th/e0130dac619f91a6bb8e7987550c2466 > I'm not really sure this is VPP and not a TRex issue, especially given > that it's only seen in ASTF mode of TRex, but I'm asking it here b/c > (1) TRex is used for CSIT > (2) There's an old open ticket for a similar CSIT problem: > https://jira.fd.io/browse/VPP-1676 > Not sure if it was resolved somehow already. > > Passing --tso-disable --lro-disable --checksum-offload-disable flags to > TRex doesn't change anything. > > If there's no known solution for this issue, I'll dig into it myself, > but I'd like to ask if maybe it was already solved by someone. Thanks! > > -- > Ivan Shvedunov > ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 > 2807 > > > > -- Ivan Shvedunov ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 2807 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19566): https://lists.fd.io/g/vpp-dev/message/19566 Mute This Topic: https://lists.fd.io/mt/83533776/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] ip4 length > l2 length with memif + TRex
Hello, I've been struggling with the following issue (observed with VPP 21.01): when trying any of ASTF TRex profiles with VPP, I get multiple errors: "ip4 length > l2 length". These don't happen with each packet but rather with some of them. It looks like the TCP payload is truncated down to 600 bytes for some reason, while looking at the TRex simulation, it should be 1448 bytes. I have collected a snippet from dispatch trace dissected by Wireshark and other info here: https://gist.github.com/ivan4th/e0130dac619f91a6bb8e7987550c2466 I'm not really sure this is VPP and not a TRex issue, especially given that it's only seen in ASTF mode of TRex, but I'm asking it here b/c (1) TRex is used for CSIT (2) There's an old open ticket for a similar CSIT problem: https://jira.fd.io/browse/VPP-1676 Not sure if it was resolved somehow already. Passing --tso-disable --lro-disable --checksum-offload-disable flags to TRex doesn't change anything. If there's no known solution for this issue, I'll dig into it myself, but I'd like to ask if maybe it was already solved by someone. Thanks! -- Ivan Shvedunov ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 2807 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19267): https://lists.fd.io/g/vpp-dev/message/19267 Mute This Topic: https://lists.fd.io/mt/82260824/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] UDP data corruption and MTU issues
Great, thanks!
On Wed, Mar 17, 2021 at 6:03 PM Florin Coras wrote:
> Infra woes solved so finally merged!
>
> Florin
>
> On Mar 17, 2021, at 6:13 AM, Ivan Shvedunov wrote:
>
> I see CI keeps failing for my patch despite rebase... I have run the tests
> that are failing locally with my patch applied (test_map, test_flowprobe,
> test_memif) and they do pass there. So indeed must be something related to
> the test environment
>
> On Tue, Mar 16, 2021 at 7:02 PM Florin Coras
> wrote:
>
>> Hi Ivan,
>>
>>
>> On Mar 16, 2021, at 8:51 AM, Ivan Shvedunov wrote:
>>
>> Thanks for clearing things up :)
>>
>>
>> Thanks for bringing this up ;-)
>>
>> Concerning our use case: it's about PFCP protocol [1] that's used in the
>> 5G / LTE CUPS setting. High throughput is usually not important there as
>> that's a control protocol but sometimes it happens to produce rather large
>> UDP packets. We use PFCP in our VPP-based project [2], it's implemented on
>> top of the VPP session layer there.
>>
>>
>> Got it and cool!
>>
>> BTW we've been hitting the hard-coded IP reassembly limit (3 fragments)
>> with it too, mind if I post a patch that makes it configurable?
>>
>>
>> CC’ed Klement for his opinion on this.
>>
>> Cheers,
>> Florin
>>
>>
>> [1] https://en.wikipedia.org/wiki/PFCP
>> [2] https://github.com/travelping/upg-vpp
>>
>>
>> On Tue, Mar 16, 2021 at 6:19 PM Florin Coras
>> wrote:
>>
>>> Hi Ivan,
>>>
>>>
>>> On Mar 16, 2021, at 6:40 AM, Ivan Shvedunov wrote:
>>>
>>> Hi Florin,
>>>
>>> (inline)
>>>
>>> On Mon, Mar 15, 2021 at 10:27 PM Florin Coras
>>> wrote:
>>>
>>>> On Mar 15, 2021, at 9:08 AM, Ivan Shvedunov wrote:
>>>>
>>>> Hello,
>>>>
>>>> I've noticed a problem in the VPP session layer and posted a patch to
>>>> fix it [1].
>>>> Namely, UDP datagrams with length > 1908 that require buffer chaining
>>>> get corrupted, because there's a bug in session_tx_fifo_chain_tail() that's
>>>> responsible for the buffer chaining (it doesn't account for
>>>> SESSION_CONN_HDR_LEN). The bug only affects datagram transports and doesn't
>>>> break TCP.
>>>>
>>>>
>>>> Looks good. Thanks! More lower.
>>>>
>>>
>>> There seems to be an issue with CI, but I'm not sure if it's related to
>>> the patch in any way.
>>>
>>>
>>> Most probably not. Rebased.
>>>
>>>
>>>
>>>>
>>>>
>>>> There's another problem with the UDP code: "udp mtu". As far as I
>>>> understand, the plan there [2] is to use the MTU of the output interface
>>>> for each datagram being sent, and "udp { mtu ... }" setting is only used as
>>>> a temporary measure. Yet, it seems to me that what is happening when a
>>>> datagram exceeds that MTU value is not exactly correct: instead of
>>>> undergoing IP fragmentation as one would expect, the datagrams get split
>>>> into multiple UDP datagrams. This is not handled correctly by apps that use
>>>> UDP, most of the time, and did cause me some hours spent debugging strange
>>>> app behavior. Wouldn't failing to send such datagrams be more correct?
>>>>
>>>>
>>>> The thing to be aware of here is that session layer offers as service
>>>> the sending of datagrams written by apps to the network. If the datagrams
>>>> are larger than the mtu (mss of sorts actually) configured for udp, session
>>>> layer chops the datagrams to mtu size. The network layer can then fragment
>>>> the resulting udp datagrams in accordance to the output interface’s mtu.
>>>>
>>>
>>>> It’s not recommended to use this feature because original datagram
>>>> borders are lost. Moreover, losses will lead to even more problems. The
>>>> option is there for historical reasons so we could consider removing it at
>>>> one point, once we’re fully convinced it’s not useful.
>>>>
>>>
>>> Problem is, the UDP fragmentation feature is on by default, with default
>>> size limit being 1500 [1]. So, if one ever needs to send UDP datagrams
>>> bigger than that they will be split, which may be rather surprising if one
>>> is unaware of this UDP splitting feature. Maybe it's worth changing t
Re: [vpp-dev] UDP data corruption and MTU issues
I see CI keeps failing for my patch despite rebase... I have run the tests
that are failing locally with my patch applied (test_map, test_flowprobe,
test_memif) and they do pass there. So indeed must be something related to
the test environment
On Tue, Mar 16, 2021 at 7:02 PM Florin Coras wrote:
> Hi Ivan,
>
>
> On Mar 16, 2021, at 8:51 AM, Ivan Shvedunov wrote:
>
> Thanks for clearing things up :)
>
>
> Thanks for bringing this up ;-)
>
> Concerning our use case: it's about PFCP protocol [1] that's used in the
> 5G / LTE CUPS setting. High throughput is usually not important there as
> that's a control protocol but sometimes it happens to produce rather large
> UDP packets. We use PFCP in our VPP-based project [2], it's implemented on
> top of the VPP session layer there.
>
>
> Got it and cool!
>
> BTW we've been hitting the hard-coded IP reassembly limit (3 fragments)
> with it too, mind if I post a patch that makes it configurable?
>
>
> CC’ed Klement for his opinion on this.
>
> Cheers,
> Florin
>
>
> [1] https://en.wikipedia.org/wiki/PFCP
> [2] https://github.com/travelping/upg-vpp
>
>
> On Tue, Mar 16, 2021 at 6:19 PM Florin Coras
> wrote:
>
>> Hi Ivan,
>>
>>
>> On Mar 16, 2021, at 6:40 AM, Ivan Shvedunov wrote:
>>
>> Hi Florin,
>>
>> (inline)
>>
>> On Mon, Mar 15, 2021 at 10:27 PM Florin Coras
>> wrote:
>>
>>> On Mar 15, 2021, at 9:08 AM, Ivan Shvedunov wrote:
>>>
>>> Hello,
>>>
>>> I've noticed a problem in the VPP session layer and posted a patch to
>>> fix it [1].
>>> Namely, UDP datagrams with length > 1908 that require buffer chaining
>>> get corrupted, because there's a bug in session_tx_fifo_chain_tail() that's
>>> responsible for the buffer chaining (it doesn't account for
>>> SESSION_CONN_HDR_LEN). The bug only affects datagram transports and doesn't
>>> break TCP.
>>>
>>>
>>> Looks good. Thanks! More lower.
>>>
>>
>> There seems to be an issue with CI, but I'm not sure if it's related to
>> the patch in any way.
>>
>>
>> Most probably not. Rebased.
>>
>>
>>
>>>
>>>
>>> There's another problem with the UDP code: "udp mtu". As far as I
>>> understand, the plan there [2] is to use the MTU of the output interface
>>> for each datagram being sent, and "udp { mtu ... }" setting is only used as
>>> a temporary measure. Yet, it seems to me that what is happening when a
>>> datagram exceeds that MTU value is not exactly correct: instead of
>>> undergoing IP fragmentation as one would expect, the datagrams get split
>>> into multiple UDP datagrams. This is not handled correctly by apps that use
>>> UDP, most of the time, and did cause me some hours spent debugging strange
>>> app behavior. Wouldn't failing to send such datagrams be more correct?
>>>
>>>
>>> The thing to be aware of here is that session layer offers as service
>>> the sending of datagrams written by apps to the network. If the datagrams
>>> are larger than the mtu (mss of sorts actually) configured for udp, session
>>> layer chops the datagrams to mtu size. The network layer can then fragment
>>> the resulting udp datagrams in accordance to the output interface’s mtu.
>>>
>>
>>> It’s not recommended to use this feature because original datagram
>>> borders are lost. Moreover, losses will lead to even more problems. The
>>> option is there for historical reasons so we could consider removing it at
>>> one point, once we’re fully convinced it’s not useful.
>>>
>>
>> Problem is, the UDP fragmentation feature is on by default, with default
>> size limit being 1500 [1]. So, if one ever needs to send UDP datagrams
>> bigger than that they will be split, which may be rather surprising if one
>> is unaware of this UDP splitting feature. Maybe it's worth changing the
>> default value to some large number so that the splitting does not happen
>> unexpectedly?
>>
>>
>> The goal there is to avoid exercising the ip fragmentation code for the
>> reasons you discovered lower, i.e., default max of 3 fragments. Splitting
>> udp datagrams will always lead to some sort of problems because of loss so
>> for now the value used is 1.5k just because it’s the common interface mtu.
>> Those interested in using larger datagrams (8-9kB) will hopefully find this
>> or older threads instead of going through the pain you had to go through!
>>
Re: [vpp-dev] UDP data corruption and MTU issues
Thanks for clearing things up :)
Concerning our use case: it's about PFCP protocol [1] that's used in the 5G
/ LTE CUPS setting. High throughput is usually not important there as
that's a control protocol but sometimes it happens to produce rather large
UDP packets. We use PFCP in our VPP-based project [2], it's implemented on
top of the VPP session layer there. BTW we've been hitting the hard-coded
IP reassembly limit (3 fragments) with it too, mind if I post a patch that
makes it configurable?
[1] https://en.wikipedia.org/wiki/PFCP
[2] https://github.com/travelping/upg-vpp
On Tue, Mar 16, 2021 at 6:19 PM Florin Coras wrote:
> Hi Ivan,
>
>
> On Mar 16, 2021, at 6:40 AM, Ivan Shvedunov wrote:
>
> Hi Florin,
>
> (inline)
>
> On Mon, Mar 15, 2021 at 10:27 PM Florin Coras
> wrote:
>
>> On Mar 15, 2021, at 9:08 AM, Ivan Shvedunov wrote:
>>
>> Hello,
>>
>> I've noticed a problem in the VPP session layer and posted a patch to
>> fix it [1].
>> Namely, UDP datagrams with length > 1908 that require buffer chaining get
>> corrupted, because there's a bug in session_tx_fifo_chain_tail() that's
>> responsible for the buffer chaining (it doesn't account for
>> SESSION_CONN_HDR_LEN). The bug only affects datagram transports and doesn't
>> break TCP.
>>
>>
>> Looks good. Thanks! More lower.
>>
>
> There seems to be an issue with CI, but I'm not sure if it's related to
> the patch in any way.
>
>
> Most probably not. Rebased.
>
>
>
>>
>>
>> There's another problem with the UDP code: "udp mtu". As far as I
>> understand, the plan there [2] is to use the MTU of the output interface
>> for each datagram being sent, and "udp { mtu ... }" setting is only used as
>> a temporary measure. Yet, it seems to me that what is happening when a
>> datagram exceeds that MTU value is not exactly correct: instead of
>> undergoing IP fragmentation as one would expect, the datagrams get split
>> into multiple UDP datagrams. This is not handled correctly by apps that use
>> UDP, most of the time, and did cause me some hours spent debugging strange
>> app behavior. Wouldn't failing to send such datagrams be more correct?
>>
>>
>> The thing to be aware of here is that session layer offers as service the
>> sending of datagrams written by apps to the network. If the datagrams are
>> larger than the mtu (mss of sorts actually) configured for udp, session
>> layer chops the datagrams to mtu size. The network layer can then fragment
>> the resulting udp datagrams in accordance to the output interface’s mtu.
>>
>
>> It’s not recommended to use this feature because original datagram
>> borders are lost. Moreover, losses will lead to even more problems. The
>> option is there for historical reasons so we could consider removing it at
>> one point, once we’re fully convinced it’s not useful.
>>
>
> Problem is, the UDP fragmentation feature is on by default, with default
> size limit being 1500 [1]. So, if one ever needs to send UDP datagrams
> bigger than that they will be split, which may be rather surprising if one
> is unaware of this UDP splitting feature. Maybe it's worth changing the
> default value to some large number so that the splitting does not happen
> unexpectedly?
>
>
> The goal there is to avoid exercising the ip fragmentation code for the
> reasons you discovered lower, i.e., default max of 3 fragments. Splitting
> udp datagrams will always lead to some sort of problems because of loss so
> for now the value used is 1.5k just because it’s the common interface mtu.
> Those interested in using larger datagrams (8-9kB) will hopefully find this
> or older threads instead of going through the pain you had to go through!
>
>
>
>> Another thing I've noticed is that if UDP MTU is high enough sometimes
>> the datagrams being sent still somehow are split into smaller IP fragments
>> than necessary. E.g. I have MTU 9000 on my interfaces and UDP MTU also is
>> 9000, and 8000-byte UDP datagram is sent in one piece, while the IP packets
>> carrying 11000-byte UDP datagram are split into ~2kb IP fragments. Any
>> ideas why this could happen?
>>
>>
>> Hm, have you tried running an iperf3 test like the one part of make test
>> [1] to see if the issue is still reproducible? Also, I’d avoid sending from
>> the app dgrams larger than udp’s mtu.
>>
>
> In this case the datagrams were smaller than the "udp mtu" value, but
> larger than the interface's MTU.
> Looking at ip_frag code [2], it seems like it just can't create fragments
> bigger than the buffer size (2048 bytes
Re: [vpp-dev] UDP data corruption and MTU issues
Hi Florin,
(inline)
On Mon, Mar 15, 2021 at 10:27 PM Florin Coras
wrote:
> On Mar 15, 2021, at 9:08 AM, Ivan Shvedunov wrote:
>
> Hello,
>
> I've noticed a problem in the VPP session layer and posted a patch to
> fix it [1].
> Namely, UDP datagrams with length > 1908 that require buffer chaining get
> corrupted, because there's a bug in session_tx_fifo_chain_tail() that's
> responsible for the buffer chaining (it doesn't account for
> SESSION_CONN_HDR_LEN). The bug only affects datagram transports and doesn't
> break TCP.
>
>
> Looks good. Thanks! More lower.
>
There seems to be an issue with CI, but I'm not sure if it's related to the
patch in any way.
>
>
> There's another problem with the UDP code: "udp mtu". As far as I
> understand, the plan there [2] is to use the MTU of the output interface
> for each datagram being sent, and "udp { mtu ... }" setting is only used as
> a temporary measure. Yet, it seems to me that what is happening when a
> datagram exceeds that MTU value is not exactly correct: instead of
> undergoing IP fragmentation as one would expect, the datagrams get split
> into multiple UDP datagrams. This is not handled correctly by apps that use
> UDP, most of the time, and did cause me some hours spent debugging strange
> app behavior. Wouldn't failing to send such datagrams be more correct?
>
>
> The thing to be aware of here is that session layer offers as service the
> sending of datagrams written by apps to the network. If the datagrams are
> larger than the mtu (mss of sorts actually) configured for udp, session
> layer chops the datagrams to mtu size. The network layer can then fragment
> the resulting udp datagrams in accordance to the output interface’s mtu.
>
> It’s not recommended to use this feature because original datagram borders
> are lost. Moreover, losses will lead to even more problems. The option is
> there for historical reasons so we could consider removing it at one point,
> once we’re fully convinced it’s not useful.
>
Problem is, the UDP fragmentation feature is on by default, with default
size limit being 1500 [1]. So, if one ever needs to send UDP datagrams
bigger than that they will be split, which may be rather surprising if one
is unaware of this UDP splitting feature. Maybe it's worth changing the
default value to some large number so that the splitting does not happen
unexpectedly?
> Another thing I've noticed is that if UDP MTU is high enough sometimes
> the datagrams being sent still somehow are split into smaller IP fragments
> than necessary. E.g. I have MTU 9000 on my interfaces and UDP MTU also is
> 9000, and 8000-byte UDP datagram is sent in one piece, while the IP packets
> carrying 11000-byte UDP datagram are split into ~2kb IP fragments. Any
> ideas why this could happen?
>
>
> Hm, have you tried running an iperf3 test like the one part of make test
> [1] to see if the issue is still reproducible? Also, I’d avoid sending from
> the app dgrams larger than udp’s mtu.
>
In this case the datagrams were smaller than the "udp mtu" value, but
larger than the interface's MTU.
Looking at ip_frag code [2], it seems like it just can't create fragments
bigger than the buffer size (2048 bytes by default), that is, it can't
chain buffers for the fragments it creates, unless I'm somehow mistaken.
With IP reassembly limit e.g. in VPP itself being just 3 this may be indeed
an issue. E.g. one has MTU of 9000 and expects a packet of length 1 to
be split in 2 and not 5 fragments; these 5 fragments will not be
reassembled if received by another VPP's session layer further along the
line. But perhaps one is expected to increase buffer size when using higher
interface MTU values?
[1] https://github.com/FDio/vpp/blob/0ac5782/src/vnet/udp/udp.c#L522
[2]
https://github.com/FDio/vpp/blob/0ac5782e600097b66e6b06e0b9edc79651f3a4bd/src/vnet/ip/ip_frag.c#L107-L109
--
Ivan Shvedunov
;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 2807
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18941): https://lists.fd.io/g/vpp-dev/message/18941
Mute This Topic: https://lists.fd.io/mt/81353121/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] UDP data corruption and MTU issues
Hello,
I've noticed a problem in the VPP session layer and posted a patch to fix
it [1].
Namely, UDP datagrams with length > 1908 that require buffer chaining get
corrupted, because there's a bug in session_tx_fifo_chain_tail() that's
responsible for the buffer chaining (it doesn't account for
SESSION_CONN_HDR_LEN). The bug only affects datagram transports and doesn't
break TCP.
There's another problem with the UDP code: "udp mtu". As far as I
understand, the plan there [2] is to use the MTU of the output interface
for each datagram being sent, and "udp { mtu ... }" setting is only used as
a temporary measure. Yet, it seems to me that what is happening when a
datagram exceeds that MTU value is not exactly correct: instead of
undergoing IP fragmentation as one would expect, the datagrams get split
into multiple UDP datagrams. This is not handled correctly by apps that use
UDP, most of the time, and did cause me some hours spent debugging strange
app behavior. Wouldn't failing to send such datagrams be more correct?
Another thing I've noticed is that if UDP MTU is high enough sometimes
the datagrams being sent still somehow are split into smaller IP fragments
than necessary. E.g. I have MTU 9000 on my interfaces and UDP MTU also is
9000, and 8000-byte UDP datagram is sent in one piece, while the IP packets
carrying 11000-byte UDP datagram are split into ~2kb IP fragments. Any
ideas why this could happen?
I've also published some of the scripts I used for testing as a github
gist [3]. Not runnable out of the box but maybe they can be of some help.
[1] https://gerrit.fd.io/r/c/vpp/+/31647
[2] https://github.com/FDio/vpp/blob/0ac5782/src/vnet/udp/udp.c#L316-L317
[3] https://gist.github.com/ivan4th/af7ca91857ea05aa35c7ab4773b089f8
--
Ivan Shvedunov
;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 2807
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18921): https://lists.fd.io/g/vpp-dev/message/18921
Mute This Topic: https://lists.fd.io/mt/81353121/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] configuring ip-neighbor via the debug CLI
Hi, here's the patch: https://gerrit.fd.io/r/c/vpp/+/31363 On Wed, Feb 17, 2021 at 11:11 AM Neale Ranns wrote: > Hi Ivan, > > > > Providing config options via the CLI is not a priority, hence I’s not > often done. There’s also no testing for it, so even if it does work once, > it may not continue to do so. > > However, if you wish to contribute such a CLI, then it would be welcome. > > > > Regards, > > Neale > > > > > > *From: *vpp-dev@lists.fd.io on behalf of Ivan > Shvedunov via lists.fd.io > *Date: *Tuesday, 16 February 2021 at 12:02 > *To: *vpp-dev > *Subject: *[vpp-dev] configuring ip-neighbor via the debug CLI > > Hello, > > > please correct me if I'm wrong, but it looks like while there's "show ip > neighbor-config" command [1] it appears that there's no way to configure ip > neighbor table except for using binary API. While this may be a design > choice, it's at times handy to be able to set limit/age/recycle (especially > age in my case) through the debug cli. I'd suggest we add a new command, > e.g. > > > > set ip neighbor-config [limit NN] [age NN] [recycle NN] > > > > If this works for you, I can make a CR to add this functionality. > > Or please let me know if I missed something and this is supposed to be > done in some other way. > > Thanks! > > > > [1] > https://github.com/FDio/vpp/blob/b2c31b6/src/vnet/ip-neighbor/ip_neighbor.c#L1701-L1726 > > > > -- > > Ivan Shvedunov > ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 > 2807 > -- Ivan Shvedunov ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 2807 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18773): https://lists.fd.io/g/vpp-dev/message/18773 Mute This Topic: https://lists.fd.io/mt/80675556/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] configuring ip-neighbor via the debug CLI
Hello, please correct me if I'm wrong, but it looks like while there's "show ip neighbor-config" command [1] it appears that there's no way to configure ip neighbor table except for using binary API. While this may be a design choice, it's at times handy to be able to set limit/age/recycle (especially age in my case) through the debug cli. I'd suggest we add a new command, e.g. set ip neighbor-config [limit NN] [age NN] [recycle NN] If this works for you, I can make a CR to add this functionality. Or please let me know if I missed something and this is supposed to be done in some other way. Thanks! [1] https://github.com/FDio/vpp/blob/b2c31b6/src/vnet/ip-neighbor/ip_neighbor.c#L1701-L1726 -- Ivan Shvedunov ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 2807 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18754): https://lists.fd.io/g/vpp-dev/message/18754 Mute This Topic: https://lists.fd.io/mt/80675556/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] fix: ping: don't abort on non-keypress events
Hi folks, right now it's rather hard to use the "ping" command over e.g. govpp as it aborts immediately most of the time, usually upon receiving SOCKET_READ_EVENT. I've made a patch that makes ping stop only on UNIX_CLI_PROCESS_EVENT_READ_READY / UNIX_CLI_PROCESS_EVENT_QUIT event, so it does stop upon keypress, but not upon unrelated events: https://gerrit.fd.io/r/c/vpp/+/31035 -- Ivan Shvedunov ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 2807 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18635): https://lists.fd.io/g/vpp-dev/message/18635 Mute This Topic: https://lists.fd.io/mt/80282262/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Assertion failure on half-open connection pending cleanup
Looks like there are some CI failures in unrelated test cases ... https://jenkins.fd.io/job/vpp-debug-verify-master-ubuntu2004-x86_64/696/console On Sat, Jan 16, 2021 at 3:58 AM Florin Coras wrote: > Thanks! > > Cheers, > Florin > > On Jan 15, 2021, at 4:49 PM, Ivan Shvedunov wrote: > > https://gerrit.fd.io/r/c/vpp/+/30791 > > On Sat, Jan 16, 2021 at 3:22 AM Florin Coras > wrote: > >> Hi Ivan, >> >> You’re right that the assert wrongly assumes the connection must’ve been >> established. I’m guessing you’re only sporadically hitting it because >> typically the peer should not retransmit the syn after our reset (but >> probably the packet is lost). Want to push a patch to remove it or should >> I do it? >> >> Regards, >> Florin >> >> On Jan 15, 2021, at 3:57 PM, Ivan Shvedunov wrote: >> >> Hi, >> in our UPG project[1] I'm sometimes hitting the following assertion in >> tcp[46]-syn-sent when trying to use at least 1 worker thread: >> >> https://github.com/FDio/vpp/blob/35ef865678d82b5a6fd3936716de8afb2fd49e60/src/vnet/tcp/tcp_input.c#L1820-L1821 >> We're using the host stack in a rather non-standard way, so chances are >> that this assertion is valid, but so far I didn't manage to understand why. >> Problem is that half-open connection cleanup may happen after an >> unsuccessful connect notify callback: >> >> https://github.com/FDio/vpp/blob/35ef865678d82b5a6fd3936716de8afb2fd49e60/src/vnet/tcp/tcp_input.c#L1955-L1962 >> >> https://github.com/FDio/vpp/blob/35ef865678d82b5a6fd3936716de8afb2fd49e60/src/vnet/tcp/tcp_input.c#L1978-L1986 >> which leads us here: >> >> https://github.com/FDio/vpp/blob/35ef865678d82b5a6fd3936716de8afb2fd49e60/src/vnet/tcp/tcp_input.c#L2016-L2021 >> In this case, if the half-open connection belongs to another thread (and >> it usually belongs to the main thread), the actual cleanup is delayed >> and TCP_CONN_HALF_OPEN_DONE flag is set for the connection, but, due to the >> notify call failure, the connection establishment is not completed so the >> assertion mentioned above can't be satisfied. >> Am I missing a reason why the buffer should actually never reach >> tcp[46]-syn-sent node after the corresponding half-open connection is >> undergoing this delayed cleanup, so perhaps UPG TCP proxy code needs to be >> corrected there, or is this ASSERT indeed invalid? >> >> [1] https://github.com/travelping/upg-vpp >> >> >> >> > > -- > Ivan Shvedunov > ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 > 2807 > > > -- Ivan Shvedunov ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 2807 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18526): https://lists.fd.io/g/vpp-dev/message/18526 Mute This Topic: https://lists.fd.io/mt/79717445/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Assertion failure on half-open connection pending cleanup
https://gerrit.fd.io/r/c/vpp/+/30791 On Sat, Jan 16, 2021 at 3:22 AM Florin Coras wrote: > Hi Ivan, > > You’re right that the assert wrongly assumes the connection must’ve been > established. I’m guessing you’re only sporadically hitting it because > typically the peer should not retransmit the syn after our reset (but > probably the packet is lost). Want to push a patch to remove it or should > I do it? > > Regards, > Florin > > On Jan 15, 2021, at 3:57 PM, Ivan Shvedunov wrote: > > Hi, > in our UPG project[1] I'm sometimes hitting the following assertion in > tcp[46]-syn-sent when trying to use at least 1 worker thread: > > https://github.com/FDio/vpp/blob/35ef865678d82b5a6fd3936716de8afb2fd49e60/src/vnet/tcp/tcp_input.c#L1820-L1821 > We're using the host stack in a rather non-standard way, so chances are > that this assertion is valid, but so far I didn't manage to understand why. > Problem is that half-open connection cleanup may happen after an > unsuccessful connect notify callback: > > https://github.com/FDio/vpp/blob/35ef865678d82b5a6fd3936716de8afb2fd49e60/src/vnet/tcp/tcp_input.c#L1955-L1962 > > https://github.com/FDio/vpp/blob/35ef865678d82b5a6fd3936716de8afb2fd49e60/src/vnet/tcp/tcp_input.c#L1978-L1986 > which leads us here: > > https://github.com/FDio/vpp/blob/35ef865678d82b5a6fd3936716de8afb2fd49e60/src/vnet/tcp/tcp_input.c#L2016-L2021 > In this case, if the half-open connection belongs to another thread (and > it usually belongs to the main thread), the actual cleanup is delayed > and TCP_CONN_HALF_OPEN_DONE flag is set for the connection, but, due to the > notify call failure, the connection establishment is not completed so the > assertion mentioned above can't be satisfied. > Am I missing a reason why the buffer should actually never reach > tcp[46]-syn-sent node after the corresponding half-open connection is > undergoing this delayed cleanup, so perhaps UPG TCP proxy code needs to be > corrected there, or is this ASSERT indeed invalid? > > [1] https://github.com/travelping/upg-vpp > > > > -- Ivan Shvedunov ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 2807 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18523): https://lists.fd.io/g/vpp-dev/message/18523 Mute This Topic: https://lists.fd.io/mt/79717445/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] Assertion failure on half-open connection pending cleanup
Hi, in our UPG project[1] I'm sometimes hitting the following assertion in tcp[46]-syn-sent when trying to use at least 1 worker thread: https://github.com/FDio/vpp/blob/35ef865678d82b5a6fd3936716de8afb2fd49e60/src/vnet/tcp/tcp_input.c#L1820-L1821 We're using the host stack in a rather non-standard way, so chances are that this assertion is valid, but so far I didn't manage to understand why. Problem is that half-open connection cleanup may happen after an unsuccessful connect notify callback: https://github.com/FDio/vpp/blob/35ef865678d82b5a6fd3936716de8afb2fd49e60/src/vnet/tcp/tcp_input.c#L1955-L1962 https://github.com/FDio/vpp/blob/35ef865678d82b5a6fd3936716de8afb2fd49e60/src/vnet/tcp/tcp_input.c#L1978-L1986 which leads us here: https://github.com/FDio/vpp/blob/35ef865678d82b5a6fd3936716de8afb2fd49e60/src/vnet/tcp/tcp_input.c#L2016-L2021 In this case, if the half-open connection belongs to another thread (and it usually belongs to the main thread), the actual cleanup is delayed and TCP_CONN_HALF_OPEN_DONE flag is set for the connection, but, due to the notify call failure, the connection establishment is not completed so the assertion mentioned above can't be satisfied. Am I missing a reason why the buffer should actually never reach tcp[46]-syn-sent node after the corresponding half-open connection is undergoing this delayed cleanup, so perhaps UPG TCP proxy code needs to be corrected there, or is this ASSERT indeed invalid? [1] https://github.com/travelping/upg-vpp -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18520): https://lists.fd.io/g/vpp-dev/message/18520 Mute This Topic: https://lists.fd.io/mt/79717445/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] TCP timer race and another possible TCP issue
Hi Florin, while trying to fix the proxy cleanup issue, I've spotted another problem in the TCP stack, namely RSTs being ignored in SYN_SENT (half-open) connection state: https://gerrit.fd.io/r/c/vpp/+/28103 The following fix for handling failed active connections in the proxy has worked for me, but please confirm that it's correct: https://gerrit.fd.io/r/c/vpp/+/28104 After https://gerrit.fd.io/r/c/vpp/+/28096 , I no longer see any "disp error" diagnostics for CLOSE_WAIT but I still do see "disp errors" for SYNs in TIME_WAIT and FIN_WAIT_2 states. I didn't enable reorder in netem, the config for it that I now use is this: ip netns exec client tc qdisc add dev client root netem \ loss 15% delay 100ms 300ms duplicate 10% I tried to abandon my older proxy patch that just removes ASSERT (0), but somehow I've lost the ability to log into fd.io Gerrit with my Linux Foundation login/password, although I can still submit patches and also log into identity.linuxfoundation.org just fine. Cc'ing Vanessa again here, could you please help with this? On Tue, Jul 28, 2020 at 9:27 PM Florin Coras wrote: > Hi Ivan, > > Inline. > > On Jul 28, 2020, at 8:45 AM, Ivan Shvedunov wrote: > > Hi Florin, > thanks, the fix has worked and http_static no longer crashes. > > > Perfect, thanks for confirming! > > > I still get a number of messages like this when using release build: > /usr/bin/vpp[39]: state_sent_ok:954: BUG: couldn't send response header! > Not sure if it's actually a bug or the queue being actually full because > of the packet loss being caused by netem. > > > Not sure. Either that or there’s some issue with the send logic in some > corner case. If you have hsm->debug_level > 0 you should get a warning when > the enqueue fails because there’s no space left for sending. Note that we > probably need some logic to retry sending if enqueue fails (fill in usual > disclaimer that these are example test apps, unfortunately). I can try to > provide some guidance if anybody feels like tackling that specific problem. > > > There are also numerous messages like > /usr/bin/vpp[39]: tcp_input_dispatch_buffer:2817: tcp conn 13836 disp > error state CLOSE_WAIT flags 0x02 SYN > but I guess that's caused by a large number of connections from the same > client => port reuse and, again, the packet loss. > > > Are you also inducing reorder? > > It seems the client closes the connection cleanly (fin) and before that is > completed, which as a first step would require a close from the app and a > move to last-ack, it sends a syn. Alternatively, reordering makes it look > that way. Those warnings are not harmful per se, because tcp will just drop > the packets, but we could try processing the syns to maybe force resets. If > you have the time to do it, try this [1] > > > I've spotted another crash in the proxy example for which I've submitted a > patch: > https://gerrit.fd.io/r/c/vpp/+/28085 > > One problem with this fix is that the proxy example is actually > incomplete, as the client connection should be dropped in case if the proxy > can't establish the corresponding outgoing connection (e.g. timeout or RST > b/c of "connection refused"), whereas currently the client-side connection > just hangs instead. I'll take a stab at fixing that as our proxy in UPF > also has problems with handling similar situations and it would be easier > for me to fix it in this simpler example first. But, in any case, crashing > VPP is probably much worse than just letting the connection hang, so maybe > we should remove that ASSERT after all. > > > Exactly. The assert there is just a reminder to eventually do the proper > cleanup. Thanks for looking into it! > > Regards, > Florin > > [1] https://gerrit.fd.io/r/c/vpp/+/28096 > > > > On Mon, Jul 27, 2020 at 10:59 PM Florin Coras > wrote: > >> Hi Ivan, >> >> Took a look at the static http server and, as far as I can tell, it has >> the same type of issue the proxy had, i.e., premature session >> cleanup/reuse. Does this solve the problem for you [1]? >> >> Also, merged your elog fix patch. Thanks! >> >> Regards, >> Florin >> >> [1] https://gerrit.fd.io/r/c/vpp/+/28076 >> >> On Jul 27, 2020, at 10:22 AM, Ivan Shvedunov wrote: >> >> Hi. >> I've debugged http server issue a bit more and here are my observations: >> if I add an ASSERT(0) in the place of "No http session for thread 0 >> session_index 54", >> I get stack trace along the lines of >> Program received signal SIGABRT, Aborted. >> 0x7470bf47 in raise () from /lib/x86_64-linux-gnu/libc.so.6 >> #0 0x7470bf47 in raise () from /l
Re: [vpp-dev] TCP timer race and another possible TCP issue
Hi Florin, thanks, the fix has worked and http_static no longer crashes. I still get a number of messages like this when using release build: /usr/bin/vpp[39]: state_sent_ok:954: BUG: couldn't send response header! Not sure if it's actually a bug or the queue being actually full because of the packet loss being caused by netem. There are also numerous messages like /usr/bin/vpp[39]: tcp_input_dispatch_buffer:2817: tcp conn 13836 disp error state CLOSE_WAIT flags 0x02 SYN but I guess that's caused by a large number of connections from the same client => port reuse and, again, the packet loss. I've spotted another crash in the proxy example for which I've submitted a patch: https://gerrit.fd.io/r/c/vpp/+/28085 One problem with this fix is that the proxy example is actually incomplete, as the client connection should be dropped in case if the proxy can't establish the corresponding outgoing connection (e.g. timeout or RST b/c of "connection refused"), whereas currently the client-side connection just hangs instead. I'll take a stab at fixing that as our proxy in UPF also has problems with handling similar situations and it would be easier for me to fix it in this simpler example first. But, in any case, crashing VPP is probably much worse than just letting the connection hang, so maybe we should remove that ASSERT after all. On Mon, Jul 27, 2020 at 10:59 PM Florin Coras wrote: > Hi Ivan, > > Took a look at the static http server and, as far as I can tell, it has > the same type of issue the proxy had, i.e., premature session > cleanup/reuse. Does this solve the problem for you [1]? > > Also, merged your elog fix patch. Thanks! > > Regards, > Florin > > [1] https://gerrit.fd.io/r/c/vpp/+/28076 > > On Jul 27, 2020, at 10:22 AM, Ivan Shvedunov wrote: > > Hi. > I've debugged http server issue a bit more and here are my observations: > if I add an ASSERT(0) in the place of "No http session for thread 0 > session_index 54", > I get stack trace along the lines of > Program received signal SIGABRT, Aborted. > 0x7470bf47 in raise () from /lib/x86_64-linux-gnu/libc.so.6 > #0 0x7470bf47 in raise () from /lib/x86_64-linux-gnu/libc.so.6 > #1 0x7470d8b1 in abort () from /lib/x86_64-linux-gnu/libc.so.6 > #2 0x00407193 in os_panic () at /src/vpp/src/vpp/vnet/main.c:371 > #3 0x755ec619 in debugger () at /src/vpp/src/vppinfra/error.c:84 > #4 0x755ec397 in _clib_error (how_to_die=2, function_name=0x0, > line_number=0, > fmt=0x7fffb0e1324b "%s:%d (%s) assertion `%s' fails") > at /src/vpp/src/vppinfra/error.c:143 > #5 0x7fffb0e0cc3d in http_static_server_rx_tx_callback > (s=0x7fffb66447c0, > cf=CALLED_FROM_RX) at > /src/vpp/src/plugins/http_static/static_server.c:1030 > #6 0x7fffb0e0b437 in http_static_server_rx_callback (s=0x7fffb66447c0) > at /src/vpp/src/plugins/http_static/static_server.c:1056 > #7 0x7745a94e in app_worker_builtin_rx (app_wrk=0x7fffb59c7e00, > s=0x7fffb66447c0) at /src/vpp/src/vnet/session/application_worker.c:485 > #8 0x7745c00e in app_send_io_evt_rx (app_wrk=0x7fffb59c7e00, > s=0x7fffb66447c0) > at /src/vpp/src/vnet/session/application_worker.c:691 > #9 0x7745b34e in app_worker_lock_and_send_event > (app=0x7fffb59c7e00, > s=0x7fffb66447c0, evt_type=0 '\000') > at /src/vpp/src/vnet/session/application_worker.c:779 > #10 0x773eda5e in session_enqueue_notify_inline (s=0x7fffb66447c0) > at /src/vpp/src/vnet/session/session.c:632 > #11 0x773eddbf in session_main_flush_enqueue_events > (transport_proto=0 '\000', > thread_index=0) at /src/vpp/src/vnet/session/session.c:736 > #12 0x7680c2d4 in tcp46_rcv_process_inline ( > vm=0x76477b80 , node=0x7fffb522a680, > from_frame=0x7fffb5a05f80, is_ip4=1) at > /src/vpp/src/vnet/tcp/tcp_input.c:2478 > #13 0x7680b5ea in tcp4_rcv_process_node_fn_hsw > > (line numbers maybe slightly off due to debug print). > If I further dump the connection elog for such sessions I get > /usr/bin/vpp[39]: http_static_server_rx_tx_callback:1028: Z: > connection elog: >92.494240039: close: cidx 52 >92.494241379: fin-tx: snd_nxt 1573 rcv_nxt 235 >92.494241431: state: fin-wait-1 >92.495088424: out: flags 11, bytes 32 >92.922939929: timer-pop: retransmit cidx 52 sidx 54 >92.922978305: cc_stat: cwnd 2234 flight 1 space 1448 ssthresh 3961 > snd_wnd 64128 >92.922978440: rcv_stat: rto 344 srtt 80 mrtt-us 39371 rttvar 66 >92.922978491: snd_stat: cc_space 2233 sacked 0 lost 0 out 0 rxt 262 >92.922978711: cc: rxt-timeout snd_space 2233 snd_una 1573 out 0 > flight 1 >92.922980365: fin-tx: snd_nxt
Re: [vpp-dev] TCP timer race and another possible TCP issue
.181436102: snd_stat: cc_space 2233 sacked 0 lost 0 out 0 rxt 55986 104.181436602: ack-rx: 1 snd_una 1574 snd_wnd 64128 cwnd 2234 inflight 0 104.181437352: state: fin-wait-2 104.181438642: in: order len 39 written 39 rcv_nxt 274 rcv_wnd(o) 8192 104.181439438: fin-rx: snd_nxt 1574 rcv_nxt 274 104.181439499: state: time-wait which, along with some added debug print, indicates that http_static_server_rx_callback() on a connection that was already closed and for which http_static_server_session_disconnect_callback() was already called. Looks like some of the enqueued data is pushed via session_main_flush_enqueue_events() after the disconnect callback. At this point, I'm not sure how that should be fixed, but probably it should be as invoking rx callback after disconnect one doesn't seem to be the expected behaviour. I also noticed another problem with uninitialized elog track with TCP debugging enabled for which I submitted a fix (Gerrit worked for me this time), basically that TCP_EVT (TCP_EVT_SYN_RCVD, child, 1); that was moved in one of previous fixes must be moved even further up, as in case of failed session_stream_accept() another elog crash will happen in tcp_connection_cleanup() if the elog track is not initialized. The patch is here: https://gerrit.fd.io/r/c/vpp/+/28074 As of the timer crash in http_static in case of the release build, I'm a bit unsure but it looks like http_static_server_session_timer_stop() is not idempotent, as it doesn't reset the timer handle. If you think this is a correct guess, I can re-verify this patch and submit it. diff --git a/src/plugins/http_static/static_server.c b/src/plugins/http_static/static_server.c index 45d8731af..7772a9ec0 100644 --- a/src/plugins/http_static/static_server.c +++ b/src/plugins/http_static/static_server.c @@ -124,6 +124,7 @@ http_static_server_session_timer_stop (http_session_t * hs) return; clib_spinlock_lock (_static_server_main.tw_lock); tw_timer_stop_2t_1w_2048sl (_static_server_main.tw, hs->timer_handle); + hs->timer_handle = ~0; clib_spinlock_unlock (_static_server_main.tw_lock); } On Sat, Jul 25, 2020 at 12:13 AM Florin Coras wrote: > Great! Thanks for confirming! > > Let me know how it goes with the static http server. > > Cheers, > Florin > > On Jul 24, 2020, at 2:00 PM, Ivan Shvedunov wrote: > > Hi Florin, > I re-verified the patches and the modified patch doesn't crash either, so > I think it's safe to merge it. > Thanks! > > I will try to see what is the remaining problem with http_static > > On Fri, Jul 24, 2020 at 8:15 PM Florin Coras > wrote: > >> Hi Ivan, >> >> Adding Vanessa to see if she can help with the account issues. >> >> Thanks a lot for the patches! Pushed them here [1] and [2]. I took the >> liberty of slightly changing [2], so if you get a chance, do try it out >> again. >> >> Finally, the static http server still needs fixes. Most probably it >> mishandles cleanups and corrupts the sessions/connections under heavy load. >> >> Regards, >> Florin >> >> [1] https://gerrit.fd.io/r/c/vpp/+/28063 >> [2] https://gerrit.fd.io/r/c/vpp/+/28064 >> >> >> On Jul 24, 2020, at 9:27 AM, Ivan Shvedunov wrote: >> >> I did a bit more debugging and found an issue that was causing >> invalid TCP connection lookups. >> Basically, if session_connected_callback was failing for an app (in case >> of proxy, e.g. b/c the other corresponding connection got closed), it was >> leaving an invalid entry in the session lookup table. >> Another less serious issue that I found while debugging this issue is >> that when TCP elog is enabled in tcp_debug.h, the TCP stack crashes upon >> accepting connections because of misplaced TCP_EVT. >> After fixing these issues, my test no longer crashes with the proxy >> example. That being said, http_static plugin still crashes with the timer >> error. >> >> I tried to submit both patches to fd.io Gerrit, but as I didn't submit >> any CRs to this Gerrit before, I ran into a problem with my old email in >> Linux Foundation account settings. I've fixed that and then updated the >> email address in fd.io Gerrit too, but still getting the same error: >> remote: ERROR: commit e98791a: email address ivan...@gmail.com is not >> registered in your account, and you lack 'forge committer' permission. >> >> Because of these issues I'm attaching both patches here for now. >> >> On Thu, Jul 23, 2020 at 6:47 PM Florin Coras >> wrote: >> >>> Ah, I didn’t try running test.sh 80. The only difference in how I’m >>> running the test is that I start vpp outside of start.sh straight from >>> binaries. >>> >>> Rega
Re: [vpp-dev] TCP timer race and another possible TCP issue
Hi Florin, I re-verified the patches and the modified patch doesn't crash either, so I think it's safe to merge it. Thanks! I will try to see what is the remaining problem with http_static On Fri, Jul 24, 2020 at 8:15 PM Florin Coras wrote: > Hi Ivan, > > Adding Vanessa to see if she can help with the account issues. > > Thanks a lot for the patches! Pushed them here [1] and [2]. I took the > liberty of slightly changing [2], so if you get a chance, do try it out > again. > > Finally, the static http server still needs fixes. Most probably it > mishandles cleanups and corrupts the sessions/connections under heavy load. > > Regards, > Florin > > [1] https://gerrit.fd.io/r/c/vpp/+/28063 > [2] https://gerrit.fd.io/r/c/vpp/+/28064 > > > On Jul 24, 2020, at 9:27 AM, Ivan Shvedunov wrote: > > I did a bit more debugging and found an issue that was causing invalid TCP > connection lookups. > Basically, if session_connected_callback was failing for an app (in case > of proxy, e.g. b/c the other corresponding connection got closed), it was > leaving an invalid entry in the session lookup table. > Another less serious issue that I found while debugging this issue is that > when TCP elog is enabled in tcp_debug.h, the TCP stack crashes upon > accepting connections because of misplaced TCP_EVT. > After fixing these issues, my test no longer crashes with the proxy > example. That being said, http_static plugin still crashes with the timer > error. > > I tried to submit both patches to fd.io Gerrit, but as I didn't submit > any CRs to this Gerrit before, I ran into a problem with my old email in > Linux Foundation account settings. I've fixed that and then updated the > email address in fd.io Gerrit too, but still getting the same error: > remote: ERROR: commit e98791a: email address ivan...@gmail.com is not > registered in your account, and you lack 'forge committer' permission. > > Because of these issues I'm attaching both patches here for now. > > On Thu, Jul 23, 2020 at 6:47 PM Florin Coras > wrote: > >> Ah, I didn’t try running test.sh 80. The only difference in how I’m >> running the test is that I start vpp outside of start.sh straight from >> binaries. >> >> Regards, >> Florin >> >> On Jul 23, 2020, at 8:22 AM, Ivan Shvedunov wrote: >> >> Well, I always run the same test, the difference being only >> "test.sh 80" for http_static (it's configured to be listening on that >> port) >> or just "test.sh" for the proxy. As far as I understand, you run the >> tests without using the containers, does that include setting up netem like >> this [1] ? >> >> [1] https://github.com/ivan4th/vpp-tcp-test/blob/a3b02ec/start.sh#L34-L35 >> >> On Thu, Jul 23, 2020 at 5:10 PM Florin Coras >> wrote: >> >>> Hi Ivan, >>> >>> Updated [1] but I’m not seeing [3] after several test iterations. >>> >>> Probably the static server needs the same treatment as the proxy. Are >>> you running a slightly different test? All of the builtin apps have the >>> potential to crash vpp or leave the host stack in an unwanted state since >>> they run inline. >>> >>> Either way, to solve this, first step would be to get rid of error like, >>> “no http session for thread 0 session_index x”. Will eventually try to look >>> into it if nobody beats me to it. >>> >>> Regards, >>> Florin >>> >>> On Jul 23, 2020, at 4:59 AM, Ivan Shvedunov wrote: >>> >>> http_static produces some errors: >>> /usr/bin/vpp[40]: http_static_server_rx_tx_callback:1010: No http >>> session for thread 0 session_index 4124 >>> /usr/bin/vpp[40]: http_static_server_rx_tx_callback:1010: No http >>> session for thread 0 session_index 4124 >>> /usr/bin/vpp[40]: tcp_input_dispatch_buffer:2812: tcp conn 13658 disp >>> error state CLOSE_WAIT flags 0x02 SYN >>> /usr/bin/vpp[40]: tcp_input_dispatch_buffer:2812: tcp conn 13658 disp >>> error state CLOSE_WAIT flags 0x02 SYN >>> /usr/bin/vpp[40]: tcp_input_dispatch_buffer:2812: tcp conn 13350 disp >>> error state CLOSE_WAIT flags 0x02 SYN >>> >>> also with multiple different CP connection states related to connections >>> being closed and receiving SYN / SYN+ACK. >>> The release build crashes (did already happen before, so it's unrelated >>> to any of the fixes [1]): >>> >>> /usr/bin/vpp[39]: state_sent_ok:973: BUG: couldn't send response header! >>> /usr/bin/vpp[39]: state_sent_ok:973: BUG: couldn't send response header!
Re: [vpp-dev] TCP timer race and another possible TCP issue
I did a bit more debugging and found an issue that was causing invalid TCP connection lookups. Basically, if session_connected_callback was failing for an app (in case of proxy, e.g. b/c the other corresponding connection got closed), it was leaving an invalid entry in the session lookup table. Another less serious issue that I found while debugging this issue is that when TCP elog is enabled in tcp_debug.h, the TCP stack crashes upon accepting connections because of misplaced TCP_EVT. After fixing these issues, my test no longer crashes with the proxy example. That being said, http_static plugin still crashes with the timer error. I tried to submit both patches to fd.io Gerrit, but as I didn't submit any CRs to this Gerrit before, I ran into a problem with my old email in Linux Foundation account settings. I've fixed that and then updated the email address in fd.io Gerrit too, but still getting the same error: remote: ERROR: commit e98791a: email address ivan...@gmail.com is not registered in your account, and you lack 'forge committer' permission. Because of these issues I'm attaching both patches here for now. On Thu, Jul 23, 2020 at 6:47 PM Florin Coras wrote: > Ah, I didn’t try running test.sh 80. The only difference in how I’m > running the test is that I start vpp outside of start.sh straight from > binaries. > > Regards, > Florin > > On Jul 23, 2020, at 8:22 AM, Ivan Shvedunov wrote: > > Well, I always run the same test, the difference being only > "test.sh 80" for http_static (it's configured to be listening on that port) > or just "test.sh" for the proxy. As far as I understand, you run the tests > without using the containers, does that include setting up netem like this > [1] ? > > [1] https://github.com/ivan4th/vpp-tcp-test/blob/a3b02ec/start.sh#L34-L35 > > On Thu, Jul 23, 2020 at 5:10 PM Florin Coras > wrote: > >> Hi Ivan, >> >> Updated [1] but I’m not seeing [3] after several test iterations. >> >> Probably the static server needs the same treatment as the proxy. Are you >> running a slightly different test? All of the builtin apps have the >> potential to crash vpp or leave the host stack in an unwanted state since >> they run inline. >> >> Either way, to solve this, first step would be to get rid of error like, >> “no http session for thread 0 session_index x”. Will eventually try to look >> into it if nobody beats me to it. >> >> Regards, >> Florin >> >> On Jul 23, 2020, at 4:59 AM, Ivan Shvedunov wrote: >> >> http_static produces some errors: >> /usr/bin/vpp[40]: http_static_server_rx_tx_callback:1010: No http session >> for thread 0 session_index 4124 >> /usr/bin/vpp[40]: http_static_server_rx_tx_callback:1010: No http session >> for thread 0 session_index 4124 >> /usr/bin/vpp[40]: tcp_input_dispatch_buffer:2812: tcp conn 13658 disp >> error state CLOSE_WAIT flags 0x02 SYN >> /usr/bin/vpp[40]: tcp_input_dispatch_buffer:2812: tcp conn 13658 disp >> error state CLOSE_WAIT flags 0x02 SYN >> /usr/bin/vpp[40]: tcp_input_dispatch_buffer:2812: tcp conn 13350 disp >> error state CLOSE_WAIT flags 0x02 SYN >> >> also with multiple different CP connection states related to connections >> being closed and receiving SYN / SYN+ACK. >> The release build crashes (did already happen before, so it's unrelated >> to any of the fixes [1]): >> >> /usr/bin/vpp[39]: state_sent_ok:973: BUG: couldn't send response header! >> /usr/bin/vpp[39]: state_sent_ok:973: BUG: couldn't send response header! >> >> Program received signal SIGSEGV, Segmentation fault. >> 0x74fdcfb9 in timer_remove (pool=0x7fffb56b6828, elt=> out>) >> at /src/vpp/src/vppinfra/tw_timer_template.c:154 >> 154 /src/vpp/src/vppinfra/tw_timer_template.c: No such file or >> directory. >> #0 0x74fdcfb9 in timer_remove (pool=0x7fffb56b6828, >> elt=) at /src/vpp/src/vppinfra/tw_timer_template.c:154 >> #1 tw_timer_stop_2t_1w_2048sl ( >> tw=0x7fffb0967728 , handle=7306) >> at /src/vpp/src/vppinfra/tw_timer_template.c:374 >> ---Type to continue, or q to quit--- >> #2 0x7fffb076146f in http_static_server_session_timer_stop >> (hs=) >> at /src/vpp/src/plugins/http_static/static_server.c:126 >> #3 http_static_server_rx_tx_callback (s=0x7fffb5e13a40, >> cf=CALLED_FROM_RX) >> at /src/vpp/src/plugins/http_static/static_server.c:1026 >> #4 0x7fffb0760eb8 in http_static_server_rx_callback ( >> s=0x7fffb0967728 ) >> at /src/vpp/src/plugins/http_static/static_server.c:1037 >> #5 0x7774a9de in app_worker_builtin_rx (app_wrk=, >> s=0x7fffb5e13
Re: [vpp-dev] TCP timer race and another possible TCP issue
Well, I always run the same test, the difference being only "test.sh 80" for http_static (it's configured to be listening on that port) or just "test.sh" for the proxy. As far as I understand, you run the tests without using the containers, does that include setting up netem like this [1] ? [1] https://github.com/ivan4th/vpp-tcp-test/blob/a3b02ec/start.sh#L34-L35 On Thu, Jul 23, 2020 at 5:10 PM Florin Coras wrote: > Hi Ivan, > > Updated [1] but I’m not seeing [3] after several test iterations. > > Probably the static server needs the same treatment as the proxy. Are you > running a slightly different test? All of the builtin apps have the > potential to crash vpp or leave the host stack in an unwanted state since > they run inline. > > Either way, to solve this, first step would be to get rid of error like, > “no http session for thread 0 session_index x”. Will eventually try to look > into it if nobody beats me to it. > > Regards, > Florin > > On Jul 23, 2020, at 4:59 AM, Ivan Shvedunov wrote: > > http_static produces some errors: > /usr/bin/vpp[40]: http_static_server_rx_tx_callback:1010: No http session > for thread 0 session_index 4124 > /usr/bin/vpp[40]: http_static_server_rx_tx_callback:1010: No http session > for thread 0 session_index 4124 > /usr/bin/vpp[40]: tcp_input_dispatch_buffer:2812: tcp conn 13658 disp > error state CLOSE_WAIT flags 0x02 SYN > /usr/bin/vpp[40]: tcp_input_dispatch_buffer:2812: tcp conn 13658 disp > error state CLOSE_WAIT flags 0x02 SYN > /usr/bin/vpp[40]: tcp_input_dispatch_buffer:2812: tcp conn 13350 disp > error state CLOSE_WAIT flags 0x02 SYN > > also with multiple different CP connection states related to connections > being closed and receiving SYN / SYN+ACK. > The release build crashes (did already happen before, so it's unrelated to > any of the fixes [1]): > > /usr/bin/vpp[39]: state_sent_ok:973: BUG: couldn't send response header! > /usr/bin/vpp[39]: state_sent_ok:973: BUG: couldn't send response header! > > Program received signal SIGSEGV, Segmentation fault. > 0x74fdcfb9 in timer_remove (pool=0x7fffb56b6828, elt= out>) > at /src/vpp/src/vppinfra/tw_timer_template.c:154 > 154 /src/vpp/src/vppinfra/tw_timer_template.c: No such file or > directory. > #0 0x74fdcfb9 in timer_remove (pool=0x7fffb56b6828, > elt=) at /src/vpp/src/vppinfra/tw_timer_template.c:154 > #1 tw_timer_stop_2t_1w_2048sl ( > tw=0x7fffb0967728 , handle=7306) > at /src/vpp/src/vppinfra/tw_timer_template.c:374 > ---Type to continue, or q to quit--- > #2 0x7fffb076146f in http_static_server_session_timer_stop > (hs=) > at /src/vpp/src/plugins/http_static/static_server.c:126 > #3 http_static_server_rx_tx_callback (s=0x7fffb5e13a40, cf=CALLED_FROM_RX) > at /src/vpp/src/plugins/http_static/static_server.c:1026 > #4 0x7fffb0760eb8 in http_static_server_rx_callback ( > s=0x7fffb0967728 ) > at /src/vpp/src/plugins/http_static/static_server.c:1037 > #5 0x7774a9de in app_worker_builtin_rx (app_wrk=, > s=0x7fffb5e13a40) > at /src/vpp/src/vnet/session/application_worker.c:485 > #6 app_send_io_evt_rx (app_wrk=, s=0x7fffb5e13a40) > at /src/vpp/src/vnet/session/application_worker.c:691 > #7 0x77713d9a in session_enqueue_notify_inline (s=0x7fffb5e13a40) > at /src/vpp/src/vnet/session/session.c:632 > #8 0x77713fd1 in session_main_flush_enqueue_events > (transport_proto=, > thread_index=0) at /src/vpp/src/vnet/session/session.c:736 > #9 0x763960e9 in tcp46_established_inline (vm=0x75ddc6c0 > , > node=, frame=, is_ip4=1) at > /src/vpp/src/vnet/tcp/tcp_input.c:1558 > #10 tcp4_established_node_fn_hsw (vm=0x75ddc6c0 , > node=, > from_frame=0x7fffb5458480) at /src/vpp/src/vnet/tcp/tcp_input.c:1573 > #11 0x75b5f509 in dispatch_node (vm=0x75ddc6c0 > , node=0x7fffb4baf400, > type=VLIB_NODE_TYPE_INTERNAL, dispatch_state=VLIB_NODE_STATE_POLLING, > frame=, > last_time_stamp=) at /src/vpp/src/vlib/main.c:1194 > #12 dispatch_pending_node (vm=0x75ddc6c0 , > pending_frame_index=, > last_time_stamp=) at /src/vpp/src/vlib/main.c:1353 > #13 vlib_main_or_worker_loop (vm=, is_main=1) at > /src/vpp/src/vlib/main.c:1848 > #14 vlib_main_loop (vm=) at /src/vpp/src/vlib/main.c:1976 > #15 0x75b5daf0 in vlib_main (vm=0x75ddc6c0 , > input=0x7fffb4762fb0) > at /src/vpp/src/vlib/main.c: > #16 0x75bc2816 in thread0 (arg=140737318340288) at > /src/vpp/src/vlib/unix/main.c:660 > #17 0x74fa9ec4 in clib_calljmp () from > /usr/lib/x86_64-linux-gnu/libvppinfra.so.20.09 > #18 0x7fffd8b0 in ?? () > #19 0x75bc27c8 in vlib_unix_
Re: [vpp-dev] TCP timer race and another possible TCP issue
http_static produces some errors: /usr/bin/vpp[40]: http_static_server_rx_tx_callback:1010: No http session for thread 0 session_index 4124 /usr/bin/vpp[40]: http_static_server_rx_tx_callback:1010: No http session for thread 0 session_index 4124 /usr/bin/vpp[40]: tcp_input_dispatch_buffer:2812: tcp conn 13658 disp error state CLOSE_WAIT flags 0x02 SYN /usr/bin/vpp[40]: tcp_input_dispatch_buffer:2812: tcp conn 13658 disp error state CLOSE_WAIT flags 0x02 SYN /usr/bin/vpp[40]: tcp_input_dispatch_buffer:2812: tcp conn 13350 disp error state CLOSE_WAIT flags 0x02 SYN also with multiple different CP connection states related to connections being closed and receiving SYN / SYN+ACK. The release build crashes (did already happen before, so it's unrelated to any of the fixes [1]): /usr/bin/vpp[39]: state_sent_ok:973: BUG: couldn't send response header! /usr/bin/vpp[39]: state_sent_ok:973: BUG: couldn't send response header! Program received signal SIGSEGV, Segmentation fault. 0x74fdcfb9 in timer_remove (pool=0x7fffb56b6828, elt=) at /src/vpp/src/vppinfra/tw_timer_template.c:154 154 /src/vpp/src/vppinfra/tw_timer_template.c: No such file or directory. #0 0x74fdcfb9 in timer_remove (pool=0x7fffb56b6828, elt=) at /src/vpp/src/vppinfra/tw_timer_template.c:154 #1 tw_timer_stop_2t_1w_2048sl ( tw=0x7fffb0967728 , handle=7306) at /src/vpp/src/vppinfra/tw_timer_template.c:374 ---Type to continue, or q to quit--- #2 0x7fffb076146f in http_static_server_session_timer_stop (hs=) at /src/vpp/src/plugins/http_static/static_server.c:126 #3 http_static_server_rx_tx_callback (s=0x7fffb5e13a40, cf=CALLED_FROM_RX) at /src/vpp/src/plugins/http_static/static_server.c:1026 #4 0x7fffb0760eb8 in http_static_server_rx_callback ( s=0x7fffb0967728 ) at /src/vpp/src/plugins/http_static/static_server.c:1037 #5 0x7774a9de in app_worker_builtin_rx (app_wrk=, s=0x7fffb5e13a40) at /src/vpp/src/vnet/session/application_worker.c:485 #6 app_send_io_evt_rx (app_wrk=, s=0x7fffb5e13a40) at /src/vpp/src/vnet/session/application_worker.c:691 #7 0x77713d9a in session_enqueue_notify_inline (s=0x7fffb5e13a40) at /src/vpp/src/vnet/session/session.c:632 #8 0x77713fd1 in session_main_flush_enqueue_events (transport_proto=, thread_index=0) at /src/vpp/src/vnet/session/session.c:736 #9 0x763960e9 in tcp46_established_inline (vm=0x75ddc6c0 , node=, frame=, is_ip4=1) at /src/vpp/src/vnet/tcp/tcp_input.c:1558 #10 tcp4_established_node_fn_hsw (vm=0x75ddc6c0 , node=, from_frame=0x7fffb5458480) at /src/vpp/src/vnet/tcp/tcp_input.c:1573 #11 0x75b5f509 in dispatch_node (vm=0x75ddc6c0 , node=0x7fffb4baf400, type=VLIB_NODE_TYPE_INTERNAL, dispatch_state=VLIB_NODE_STATE_POLLING, frame=, last_time_stamp=) at /src/vpp/src/vlib/main.c:1194 #12 dispatch_pending_node (vm=0x75ddc6c0 , pending_frame_index=, last_time_stamp=) at /src/vpp/src/vlib/main.c:1353 #13 vlib_main_or_worker_loop (vm=, is_main=1) at /src/vpp/src/vlib/main.c:1848 #14 vlib_main_loop (vm=) at /src/vpp/src/vlib/main.c:1976 #15 0x75b5daf0 in vlib_main (vm=0x75ddc6c0 , input=0x7fffb4762fb0) at /src/vpp/src/vlib/main.c: #16 0x75bc2816 in thread0 (arg=140737318340288) at /src/vpp/src/vlib/unix/main.c:660 #17 0x74fa9ec4 in clib_calljmp () from /usr/lib/x86_64-linux-gnu/libvppinfra.so.20.09 #18 0x7fffd8b0 in ?? () #19 0x75bc27c8 in vlib_unix_main (argc=, argv=) at /src/vpp/src/vlib/unix/main.c:733 [1] https://github.com/ivan4th/vpp-tcp-test/blob/master/logs/crash-release-http_static-timer_remove.log On Thu, Jul 23, 2020 at 2:47 PM Ivan Shvedunov via lists.fd.io wrote: > Hi, > I've found a problem with the timer fix and commented in Gerrit [1] > accordingly. > Basically this change [2] makes the tcp_prepare_retransmit_segment() issue > go away for me. > > Concerning the proxy example, I can no longer see the SVM FIFO crashes, > but when using debug build, VPP crashes with this error (full log [3]) > during my test: > /usr/bin/vpp[39]: /src/vpp/src/vnet/tcp/tcp_input.c:2857 > (tcp46_input_inline) assertion `tcp_lookup_is_valid (tc1, b[1], > tcp_buffer_hdr (b[1]))' fails > > When using release build, it produces a lot of messages like this instead: > /usr/bin/vpp[39]: tcp_input_dispatch_buffer:2812: tcp conn 15168 disp > error state CLOSE_WAIT flags 0x02 SYN > /usr/bin/vpp[39]: tcp_input_dispatch_buffer:2812: tcp conn 9417 disp error > state FIN_WAIT_2 flags 0x12 SYN ACK > /usr/bin/vpp[39]: tcp_input_dispatch_buffer:2812: tcp conn 10703 disp > error state TIME_WAIT flags 0x12 SYN ACK > > and also > > /usr/bin/vpp[39]: active_open_connected_callback:439: connection 85557 > failed! > > [1] https://gerrit.fd.io/r/c/vpp/+/27952/4/src/vnet/tcp/tcp_timer.h#39 > [2] > https://github.com/travelping/vpp/commit/045123
Re: [vpp-dev] TCP timer race and another possible TCP issue
Hi, I've found a problem with the timer fix and commented in Gerrit [1] accordingly. Basically this change [2] makes the tcp_prepare_retransmit_segment() issue go away for me. Concerning the proxy example, I can no longer see the SVM FIFO crashes, but when using debug build, VPP crashes with this error (full log [3]) during my test: /usr/bin/vpp[39]: /src/vpp/src/vnet/tcp/tcp_input.c:2857 (tcp46_input_inline) assertion `tcp_lookup_is_valid (tc1, b[1], tcp_buffer_hdr (b[1]))' fails When using release build, it produces a lot of messages like this instead: /usr/bin/vpp[39]: tcp_input_dispatch_buffer:2812: tcp conn 15168 disp error state CLOSE_WAIT flags 0x02 SYN /usr/bin/vpp[39]: tcp_input_dispatch_buffer:2812: tcp conn 9417 disp error state FIN_WAIT_2 flags 0x12 SYN ACK /usr/bin/vpp[39]: tcp_input_dispatch_buffer:2812: tcp conn 10703 disp error state TIME_WAIT flags 0x12 SYN ACK and also /usr/bin/vpp[39]: active_open_connected_callback:439: connection 85557 failed! [1] https://gerrit.fd.io/r/c/vpp/+/27952/4/src/vnet/tcp/tcp_timer.h#39 [2] https://github.com/travelping/vpp/commit/04512323f311ceebfda351672372033b567d37ca [3] https://github.com/ivan4th/vpp-tcp-test/blob/master/logs/crash-debug-proxy-tcp_lookup_is_valid.log#L71 I will look into src/vcl/test/test_vcl.py to see if I can reproduce something like my test there, thanks! And waiting for Dave's input concerning the CSIT part, too, of course. On Thu, Jul 23, 2020 at 5:22 AM Florin Coras wrote: > Hi Ivan, > > Thanks for the test. After modifying it a bit to run straight from > binaries, I managed to repro the issue. As expected, the proxy is not > cleaning up the sessions correctly (example apps do run out of sync ..). > Here’s a quick patch that solves some of the obvious issues [1] (note that > it’s chained with gerrit 27952). I didn’t do too much testing, so let me > know if you hit some other problems. As far as I can tell, 27952 is needed. > > As for the CI, I guess there are two types of tests we might want (cc-ing > Dave since he has experience with this): > - functional test that could live as part of “make test” infra. The host > stack already has some functional integration tests, i.e., the vcl tests in > src/vcl/test/test_vcl.py (quic, tls, tcp also have some). We could do > something similar for the proxy app, but the tests need to be lightweight > as they’re run as part of the verify jobs > - CSIT scale/performance tests. We could use something like your scripts > to test the proxy but also ld_preload + nginx and other applications. Dave > should have more opinions here :-) > > Regards, > Florin > > [1] https://gerrit.fd.io/r/c/vpp/+/28041 > > On Jul 22, 2020, at 1:18 PM, Ivan Shvedunov wrote: > > Concerning the CI: I'd be glad to add that test to "make test", but not > sure how to approach it. The test is not about containers but more about > using network namespaces and some tools like wrk to create a lot of TCP > connections to do some "stress testing" of VPP host stack (and as it was > noted, it fails not only on the proxy example, but also on http_static > plugin). It's probably doable w/o any external tooling at all, and even > without the network namespaces either, using only VPP's own TCP stack, but > that is probably rather hard. Could you suggest some ideas how it could be > added to "make test"? Should I add a `test_py` under `tests/` that > creates host interfaces in VPP and uses these via OS networking instead of > the packet generator? As far as I can see there's something like that in > srv6-mobile plugin [1]. > > [1] > https://github.com/travelping/vpp/blob/feature/2005/upf/src/plugins/srv6-mobile/extra/runner.py#L125 > > On Wed, Jul 22, 2020 at 8:25 PM Florin Coras > wrote: > >> I missed the point about the CI in my other reply. If we can somehow >> integrate some container based tests into the “make test” infra, I wouldn’t >> mind at all! :-) >> >> Regards, >> Florin >> >> On Jul 22, 2020, at 4:17 AM, Ivan Shvedunov wrote: >> >> Hi, >> sadly the patch apparently didn't work. It should have worked but for >> some reason it didn't ... >> >> On the bright side, I've made a test case [1] using fresh upstream VPP >> code with no UPF that reproduces the issues I mentioned, including both >> timer and TCP retransmit one along with some other possible problems using >> http_static plugin and the proxy example, along with nginx (with proxy) and >> wrk. >> >> It is docker-based, but the main scripts (start.sh and test.sh) can be >> used without Docker, too. >> I've used our own Dockerfiles to build the images, but I'm not sure if >> that makes any difference. >> I've added some log files resulting from th
Re: [vpp-dev] TCP timer race and another possible TCP issue
Concerning the CI: I'd be glad to add that test to "make test", but not sure how to approach it. The test is not about containers but more about using network namespaces and some tools like wrk to create a lot of TCP connections to do some "stress testing" of VPP host stack (and as it was noted, it fails not only on the proxy example, but also on http_static plugin). It's probably doable w/o any external tooling at all, and even without the network namespaces either, using only VPP's own TCP stack, but that is probably rather hard. Could you suggest some ideas how it could be added to "make test"? Should I add a `test_py` under `tests/` that creates host interfaces in VPP and uses these via OS networking instead of the packet generator? As far as I can see there's something like that in srv6-mobile plugin [1]. [1] https://github.com/travelping/vpp/blob/feature/2005/upf/src/plugins/srv6-mobile/extra/runner.py#L125 On Wed, Jul 22, 2020 at 8:25 PM Florin Coras wrote: > I missed the point about the CI in my other reply. If we can somehow > integrate some container based tests into the “make test” infra, I wouldn’t > mind at all! :-) > > Regards, > Florin > > On Jul 22, 2020, at 4:17 AM, Ivan Shvedunov wrote: > > Hi, > sadly the patch apparently didn't work. It should have worked but for some > reason it didn't ... > > On the bright side, I've made a test case [1] using fresh upstream VPP > code with no UPF that reproduces the issues I mentioned, including both > timer and TCP retransmit one along with some other possible problems using > http_static plugin and the proxy example, along with nginx (with proxy) and > wrk. > > It is docker-based, but the main scripts (start.sh and test.sh) can be > used without Docker, too. > I've used our own Dockerfiles to build the images, but I'm not sure if > that makes any difference. > I've added some log files resulting from the runs that crashed in > different places. For me, the tests crash on each run, but in different > places. > > The TCP retransmit problem happens with http_static when using the debug > build. When using release build, some unrelated crash in timer_remove() > happens instead. > The SVM FIFO crash happens when using the proxy. It can happen with both > release and debug builds. > > Please see the repo [1] for details and crash logs. > > [1] https://github.com/ivan4th/vpp-tcp-test > > P.S. As the tests do expose some problems with VPP host stack and some of > the VPP plugins/examples, maybe we should consider adding them to the VPP > CI, too? > > On Thu, Jul 16, 2020 at 8:33 PM Florin Coras > wrote: > >> Hi Ivan, >> >> Thanks for the detailed report! >> >> I assume this is a situation where most of the connections time out and >> the rate limiting we apply on the pending timer queue delays handling for >> long enough to be in a situation like the one you described. Here’s a draft >> patch that starts tracking pending timers [1]. Let me know if it solves the >> first problem. >> >> Regarding the second, it looks like the first chunk in the fifo is not >> properly initialized/corrupted. It’s hard to tell what leads to that given >> that I haven’t seen this sort of issues even with larger number of >> connections. You could maybe try calling svm_fifo_is_sane() in the >> enqueue/dequeue functions, or after the proxy allocates/shares the fifos to >> catch the issue as early as possible. >> >> Regards, >> Florin >> >> [1] https://gerrit.fd.io/r/c/vpp/+/27952 >> >> On Jul 16, 2020, at 2:03 AM, ivan...@gmail.com wrote: >> >> Hi, >> I'm working on the Travelping UPF project >> https://github.com/travelping/vpp <https://github.com/travelping/vpp>For >> variety of reasons, it's presently maintained as a fork of UPF that's >> rebased on top of upstream master from time to time, but really it's just a >> plugin. During 40K TCP connection test with netem, I found an issue with >> TCP timer race (timers firing after tcp_timer_reset() was called for them) >> which I tried to work around only to stumble into another crash, which I'm >> presently debugging (an SVM FIFO bug, possibly) but maybe some of you folks >> have some ideas what it could be. >> I've described my findings in this JIRA ticket: >> https://jira.fd.io/browse/VPP-1923 >> Although the last upstream commit UPF is presently based on >> (afc233aa93c3f23b30b756cb4ae2967f9681) >> was some time ago, I believe the problems are still relevant as there >> were no changes in these parts of code in master since that commit. >> >> >> >> > > -- > Iv
Re: [vpp-dev] TCP timer race and another possible TCP issue
Some preliminary observations concerning the crashes in the proxy example: * !rb_tree_is_init(...) assertion failures are likely caused by multiple active_open_connected_callback() invocations for the same connection * f_update_ooo_deq() SIGSEGV crash is possibly caused for late callbacks for connections that are already gone On Wed, Jul 22, 2020 at 2:18 PM Ivan Shvedunov via lists.fd.io wrote: > Hi, > sadly the patch apparently didn't work. It should have worked but for some > reason it didn't ... > > On the bright side, I've made a test case [1] using fresh upstream VPP > code with no UPF that reproduces the issues I mentioned, including both > timer and TCP retransmit one along with some other possible problems using > http_static plugin and the proxy example, along with nginx (with proxy) and > wrk. > > It is docker-based, but the main scripts (start.sh and test.sh) can be > used without Docker, too. > I've used our own Dockerfiles to build the images, but I'm not sure if > that makes any difference. > I've added some log files resulting from the runs that crashed in > different places. For me, the tests crash on each run, but in different > places. > > The TCP retransmit problem happens with http_static when using the debug > build. When using release build, some unrelated crash in timer_remove() > happens instead. > The SVM FIFO crash happens when using the proxy. It can happen with both > release and debug builds. > > Please see the repo [1] for details and crash logs. > > [1] https://github.com/ivan4th/vpp-tcp-test > > P.S. As the tests do expose some problems with VPP host stack and some of > the VPP plugins/examples, maybe we should consider adding them to the VPP > CI, too? > > On Thu, Jul 16, 2020 at 8:33 PM Florin Coras > wrote: > >> Hi Ivan, >> >> Thanks for the detailed report! >> >> I assume this is a situation where most of the connections time out and >> the rate limiting we apply on the pending timer queue delays handling for >> long enough to be in a situation like the one you described. Here’s a draft >> patch that starts tracking pending timers [1]. Let me know if it solves the >> first problem. >> >> Regarding the second, it looks like the first chunk in the fifo is not >> properly initialized/corrupted. It’s hard to tell what leads to that given >> that I haven’t seen this sort of issues even with larger number of >> connections. You could maybe try calling svm_fifo_is_sane() in the >> enqueue/dequeue functions, or after the proxy allocates/shares the fifos to >> catch the issue as early as possible. >> >> Regards, >> Florin >> >> [1] https://gerrit.fd.io/r/c/vpp/+/27952 >> >> On Jul 16, 2020, at 2:03 AM, ivan...@gmail.com wrote: >> >> Hi, >> I'm working on the Travelping UPF project >> https://github.com/travelping/vpp <https://github.com/travelping/vpp>For >> variety of reasons, it's presently maintained as a fork of UPF that's >> rebased on top of upstream master from time to time, but really it's just a >> plugin. During 40K TCP connection test with netem, I found an issue with >> TCP timer race (timers firing after tcp_timer_reset() was called for them) >> which I tried to work around only to stumble into another crash, which I'm >> presently debugging (an SVM FIFO bug, possibly) but maybe some of you folks >> have some ideas what it could be. >> I've described my findings in this JIRA ticket: >> https://jira.fd.io/browse/VPP-1923 >> Although the last upstream commit UPF is presently based on >> (afc233aa93c3f23b30b756cb4ae2967f9681) >> was some time ago, I believe the problems are still relevant as there >> were no changes in these parts of code in master since that commit. >> >> >> > > -- > Ivan Shvedunov > ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 > 2807 > > -- Ivan Shvedunov ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 2807 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#17037): https://lists.fd.io/g/vpp-dev/message/17037 Mute This Topic: https://lists.fd.io/mt/75537746/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] VPP vs Kernel proxy performance
Most likely, this particular error (!rb_tree_is_init) may stem from the fact that proxy's active_open_connected_callback() is invoked multiple times for the same connection. I'm not sure it's supposed to happen this way. Also, there seem to be other SVM FIFO issues, too On Wed, Jul 22, 2020 at 4:27 PM Ivan Shvedunov via lists.fd.io wrote: > Hi, > this SVM FIFO error looks like a crash that is mentioned in the ticket > related to a TCP timer bug [1]. > I do sometimes get this exact error, too, it just happens less frequently > than the other kinds of the crash. > It can probably be reproduced using my test repo [2] that I have mentioned > in another topic [3]. > Looks like it is present in the proxy example and our UPF TCP proxy has > inherited it. > > [1] https://jira.fd.io/browse/VPP-1923 > [2] https://github.com/ivan4th/vpp-tcp-test > [3] https://lists.fd.io/g/vpp-dev/message/17032 > > On Wed, Jul 22, 2020 at 12:23 PM Sebastiano Miano < > mianosebasti...@gmail.com> wrote: > >> Hi Florin, >> thanks for your reply. >> Unfortunately, changing the "fifo size" to "4m" has not changed the >> performance that much. I've only got 2Gbps instead of 1.5Gbps. >> Moreover, I have checked both the "show errors" output and it looks like >> no errors are shown [1]. >> The "show run" output looks fine, which makes me think that VPP is >> actually loaded and running. >> >> What is weird is that, sometimes, after a couple of runs, the application >> crashes with the following error: >> "vpp/src/svm/svm_fifo.c:410 (svm_fifo_init_ooo_lookup) assertion >> `!rb_tree_is_init (>ooo_deq_lookup)' fails" >> >> What is causing this error? >> >> Thanks again, >> Sebastiano >> >> [1] https://gist.github.com/sebymiano/8bc582bc6491cc88f5a608d4a83b25e9 >> >> Il giorno mar 21 lug 2020 alle ore 19:22 Florin Coras < >> fcoras.li...@gmail.com> ha scritto: >> >>> Hi Sebastiano, >>> >>> The test proxy application is just an example, so it’s far from >>> optimized. Nonetheless, last time I tested it was capable of saturating a >>> 10Gbps nic. So some things to consider while debugging: >>> - fifo size configuration. The wiki page does not set a fifo size and as >>> a result a small default is used. Try adding “fifo size 4096” to the “test >>> proxy” cli to force fifos of 4MB. >>> - check error counters with “show error” to see if tcp or the interfaces >>> exhibit other errors. >>> - see if vpp is actually loaded with “show run” (do a clear run to check >>> only recent data). >>> >>> Regards, >>> Florin >>> >>> On Jul 21, 2020, at 8:44 AM, Sebastiano Miano >>> wrote: >>> >>> Dear all, >>> I was trying to test the performance of the VPP Host Stack, compared to >>> the one of the Linux kernel TCP/IP stack. >>> In particular, I was testing the TestProxy application [1] and compare >>> it with the simpleproxy application available at this URL [2]. >>> >>> My setup is composed of a server1, which runs the VPP test proxy >>> application attached to two physical interfaces (dual-port Intel >>> XL710 40GbE NIC). The application listens to TCP traffic on a given >>> IP1:port1 (interface 1) and forwards it to a "backend" server listening on >>> another IP2:port2 (interface 2). >>> Another server2 (same NIC) is running both an iperf3 client, which sends >>> traffic to the proxy at IP1:port1 and an iperf3 server, which receives >>> traffic on IP2:port2. Both iperf3 client and servers are running in two >>> different netns on server2 [3]. >>> The VPP startup configuration that I used is the following [4]. >>> >>> The configuration works fine, however, these are the results that I got: >>> VPP Test Proxy (VPP) - 1.50 Gbits/sec >>> Simpleproxy (Linux kernel) - 9.39 Gbits/sec >>> >>> I am wondering if there is something wrong with my setup/configuration, >>> I am quite new to VPP. >>> >>> Thank you in advance. >>> Sebastiano >>> >>> [1] https://wiki.fd.io/view/VPP/HostStack/TestProxy >>> [2] https://github.com/vzaliva/simpleproxy >>> [3] https://gist.github.com/sebymiano/0ff74ed0ec2805591fca7cd688b805d9 >>> [4] https://gist.github.com/sebymiano/f75c0f4d506fbf722cb2358b1deaa250 >>> >>> >>> >> > > -- > Ivan Shvedunov > ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 > 2807 > > -- Ivan Shvedunov ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 2807 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#17035): https://lists.fd.io/g/vpp-dev/message/17035 Mute This Topic: https://lists.fd.io/mt/75706142/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] VPP vs Kernel proxy performance
Hi, this SVM FIFO error looks like a crash that is mentioned in the ticket related to a TCP timer bug [1]. I do sometimes get this exact error, too, it just happens less frequently than the other kinds of the crash. It can probably be reproduced using my test repo [2] that I have mentioned in another topic [3]. Looks like it is present in the proxy example and our UPF TCP proxy has inherited it. [1] https://jira.fd.io/browse/VPP-1923 [2] https://github.com/ivan4th/vpp-tcp-test [3] https://lists.fd.io/g/vpp-dev/message/17032 On Wed, Jul 22, 2020 at 12:23 PM Sebastiano Miano wrote: > Hi Florin, > thanks for your reply. > Unfortunately, changing the "fifo size" to "4m" has not changed the > performance that much. I've only got 2Gbps instead of 1.5Gbps. > Moreover, I have checked both the "show errors" output and it looks like > no errors are shown [1]. > The "show run" output looks fine, which makes me think that VPP is > actually loaded and running. > > What is weird is that, sometimes, after a couple of runs, the application > crashes with the following error: > "vpp/src/svm/svm_fifo.c:410 (svm_fifo_init_ooo_lookup) assertion > `!rb_tree_is_init (>ooo_deq_lookup)' fails" > > What is causing this error? > > Thanks again, > Sebastiano > > [1] https://gist.github.com/sebymiano/8bc582bc6491cc88f5a608d4a83b25e9 > > Il giorno mar 21 lug 2020 alle ore 19:22 Florin Coras < > fcoras.li...@gmail.com> ha scritto: > >> Hi Sebastiano, >> >> The test proxy application is just an example, so it’s far from >> optimized. Nonetheless, last time I tested it was capable of saturating a >> 10Gbps nic. So some things to consider while debugging: >> - fifo size configuration. The wiki page does not set a fifo size and as >> a result a small default is used. Try adding “fifo size 4096” to the “test >> proxy” cli to force fifos of 4MB. >> - check error counters with “show error” to see if tcp or the interfaces >> exhibit other errors. >> - see if vpp is actually loaded with “show run” (do a clear run to check >> only recent data). >> >> Regards, >> Florin >> >> On Jul 21, 2020, at 8:44 AM, Sebastiano Miano >> wrote: >> >> Dear all, >> I was trying to test the performance of the VPP Host Stack, compared to >> the one of the Linux kernel TCP/IP stack. >> In particular, I was testing the TestProxy application [1] and compare it >> with the simpleproxy application available at this URL [2]. >> >> My setup is composed of a server1, which runs the VPP test proxy >> application attached to two physical interfaces (dual-port Intel >> XL710 40GbE NIC). The application listens to TCP traffic on a given >> IP1:port1 (interface 1) and forwards it to a "backend" server listening on >> another IP2:port2 (interface 2). >> Another server2 (same NIC) is running both an iperf3 client, which sends >> traffic to the proxy at IP1:port1 and an iperf3 server, which receives >> traffic on IP2:port2. Both iperf3 client and servers are running in two >> different netns on server2 [3]. >> The VPP startup configuration that I used is the following [4]. >> >> The configuration works fine, however, these are the results that I got: >> VPP Test Proxy (VPP) - 1.50 Gbits/sec >> Simpleproxy (Linux kernel) - 9.39 Gbits/sec >> >> I am wondering if there is something wrong with my setup/configuration, I >> am quite new to VPP. >> >> Thank you in advance. >> Sebastiano >> >> [1] https://wiki.fd.io/view/VPP/HostStack/TestProxy >> [2] https://github.com/vzaliva/simpleproxy >> [3] https://gist.github.com/sebymiano/0ff74ed0ec2805591fca7cd688b805d9 >> [4] https://gist.github.com/sebymiano/f75c0f4d506fbf722cb2358b1deaa250 >> >> >> > -- Ivan Shvedunov ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 2807 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#17034): https://lists.fd.io/g/vpp-dev/message/17034 Mute This Topic: https://lists.fd.io/mt/75706142/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] TCP timer race and another possible TCP issue
Hi, sadly the patch apparently didn't work. It should have worked but for some reason it didn't ... On the bright side, I've made a test case [1] using fresh upstream VPP code with no UPF that reproduces the issues I mentioned, including both timer and TCP retransmit one along with some other possible problems using http_static plugin and the proxy example, along with nginx (with proxy) and wrk. It is docker-based, but the main scripts (start.sh and test.sh) can be used without Docker, too. I've used our own Dockerfiles to build the images, but I'm not sure if that makes any difference. I've added some log files resulting from the runs that crashed in different places. For me, the tests crash on each run, but in different places. The TCP retransmit problem happens with http_static when using the debug build. When using release build, some unrelated crash in timer_remove() happens instead. The SVM FIFO crash happens when using the proxy. It can happen with both release and debug builds. Please see the repo [1] for details and crash logs. [1] https://github.com/ivan4th/vpp-tcp-test P.S. As the tests do expose some problems with VPP host stack and some of the VPP plugins/examples, maybe we should consider adding them to the VPP CI, too? On Thu, Jul 16, 2020 at 8:33 PM Florin Coras wrote: > Hi Ivan, > > Thanks for the detailed report! > > I assume this is a situation where most of the connections time out and > the rate limiting we apply on the pending timer queue delays handling for > long enough to be in a situation like the one you described. Here’s a draft > patch that starts tracking pending timers [1]. Let me know if it solves the > first problem. > > Regarding the second, it looks like the first chunk in the fifo is not > properly initialized/corrupted. It’s hard to tell what leads to that given > that I haven’t seen this sort of issues even with larger number of > connections. You could maybe try calling svm_fifo_is_sane() in the > enqueue/dequeue functions, or after the proxy allocates/shares the fifos to > catch the issue as early as possible. > > Regards, > Florin > > [1] https://gerrit.fd.io/r/c/vpp/+/27952 > > On Jul 16, 2020, at 2:03 AM, ivan...@gmail.com wrote: > > Hi, > I'm working on the Travelping UPF project > https://github.com/travelping/vpp <https://github.com/travelping/vpp>For > variety of reasons, it's presently maintained as a fork of UPF that's > rebased on top of upstream master from time to time, but really it's just a > plugin. During 40K TCP connection test with netem, I found an issue with > TCP timer race (timers firing after tcp_timer_reset() was called for them) > which I tried to work around only to stumble into another crash, which I'm > presently debugging (an SVM FIFO bug, possibly) but maybe some of you folks > have some ideas what it could be. > I've described my findings in this JIRA ticket: > https://jira.fd.io/browse/VPP-1923 > Although the last upstream commit UPF is presently based on > (afc233aa93c3f23b30b756cb4ae2967f9681) > was some time ago, I believe the problems are still relevant as there > were no changes in these parts of code in master since that commit. > > > > -- Ivan Shvedunov ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 2807 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#17032): https://lists.fd.io/g/vpp-dev/message/17032 Mute This Topic: https://lists.fd.io/mt/75537746/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
