Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64

[Dave Wallace]

Ben,

Excellent debugging work!
I will rebase my gerrit change onto yours, retest in the Jenkins 
Sandbox, and report back.


Thanks,
-daw-

On 2/11/21 2:18 PM, Benoit Ganne (bganne) wrote:

Hi Dave,


There are still 6 ikev2 test failures from this test run that need to be
resolved.

I identified the issue: undefined behaviour + aggressive gcc optimization.
The tentative fix is here: https://gerrit.fd.io/r/c/vpp/+/31240

Note that for this to happen, you need both:
  - compile with gcc
  - use gcc memcpy builtin
This is the case for ARM64 but not x86, as we do not use gcc memcpy builtin on 
x86.

It looks like there are other cases where we call memcpy with NULL pointers: 
https://gerrit.fd.io/r/c/vpp/+/31241
I am going to chase them down.

Best
ben




-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18734): https://lists.fd.io/g/vpp-dev/message/18734
Mute This Topic: https://lists.fd.io/mt/80495347/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64

[Andrew Yourtchenko]

Wow, this is a great catch, Ben! Kudos! 

--a

> On 11 Feb 2021, at 20:18, Benoit Ganne (bganne) via lists.fd.io 
>  wrote:
> 
> Hi Dave,
> 
>> There are still 6 ikev2 test failures from this test run that need to be
>> resolved.
> 
> I identified the issue: undefined behaviour + aggressive gcc optimization.
> The tentative fix is here: https://gerrit.fd.io/r/c/vpp/+/31240
> 
> Note that for this to happen, you need both:
> - compile with gcc
> - use gcc memcpy builtin
> This is the case for ARM64 but not x86, as we do not use gcc memcpy builtin 
> on x86.
> 
> It looks like there are other cases where we call memcpy with NULL pointers: 
> https://gerrit.fd.io/r/c/vpp/+/31241
> I am going to chase them down.
> 
> Best
> ben
> 
> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18733): https://lists.fd.io/g/vpp-dev/message/18733
Mute This Topic: https://lists.fd.io/mt/80495347/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64

[Benoit Ganne (bganne) via lists.fd.io]

Hi Dave,

> There are still 6 ikev2 test failures from this test run that need to be
> resolved.

I identified the issue: undefined behaviour + aggressive gcc optimization.
The tentative fix is here: https://gerrit.fd.io/r/c/vpp/+/31240

Note that for this to happen, you need both:
 - compile with gcc
 - use gcc memcpy builtin
This is the case for ARM64 but not x86, as we do not use gcc memcpy builtin on 
x86.

It looks like there are other cases where we call memcpy with NULL pointers: 
https://gerrit.fd.io/r/c/vpp/+/31241
I am going to chase them down.

Best
ben


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18732): https://lists.fd.io/g/vpp-dev/message/18732
Mute This Topic: https://lists.fd.io/mt/80495347/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] RFC: Interface Mirroring for Linux Network Stackintegration

[Andrew Yourtchenko]

Hi Neale, Ben,

> On 11 Feb 2021, at 17:50, Neale Ranns  wrote:
> 
> 
>
> Hi Ben,
>
> Not a silly idea  but given the shared memory’s mutex based read/write 
> mechanism, one can’t connect to VPP form within VPP.

Once upon a time I had a very cursed throwaway prototype called “VPP api 
freezer”, where I did just that, more over it also maintained its own separate 
shared memory endpoint pretending to be an earlier version of VPP

it was an extremely nasty trick of hijacking the queue signal callback but 
maybe it can be reincarnated into something less cursed :-)

https://github.com/ayourtch/vpp-api-freezer/blob/142c25b71d2edd27117946e4cbc290243aa4b468/src/api1908compat.c#L535

Though I remember my biggest problem with that hack was not calling the VPP api 
(that seemed to work fine) but rather monitoring two “server” sides of shared 
memory at once ...

(Also, as I finished typing, I also remembered about the “binary-api” command 
that seemingly does just this - issuing calls to binary api to its own running 
instance ?)

Not sure if it is if any use, and sorry if I misunderstood anything from the 
problem ! :-)

—a


> Maybe socket transport for the API is an option, otherwise we considered a 
> new ‘direct’ transport for the API, which as you say would be a bit like RPC. 
> The plugin could then use the VAPI function wrappers over this direct 
> transport.
>
> /neale
>
>
>
> From: Benoit Ganne (bganne) 
> Date: Thursday, 11 February 2021 at 17:17
> To: Neale Ranns , vpp-dev@lists.fd.io 
> 
> Subject: RE: [vpp-dev] RFC: Interface Mirroring for Linux Network 
> Stackintegration
> 
> Hi Neale,
> 
> > this adds another plugin that acts as a netlink listener and invokes VPP
> > internal APIs (i.e. directly calls e.g. FIB and ip-neighbour) to add state
> > to VPP.
> [...]
> > Disadvantages:
> > 1.Not using VPP's binary API means you do not get the tracing
> > function, which in turn means you do not get the replay function. These
> > are both exceptionally useful field debugging tools.
> 
> Maybe a silly idea, but I guess you could always call the APIs from the 
> plugin instead of direct function calls, a bit like RPC to main thread? That 
> still means a single process to operate but you can use API replay.
> 
> ben
> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18731): https://lists.fd.io/g/vpp-dev/message/18731
Mute This Topic: https://lists.fd.io/mt/80033164/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] RFC: Interface Mirroring for Linux Network Stackintegration

[Florin Coras]

Hi Neale, 

You should be able to fake a binary api connection from within vpp with 
vl_api_memclnt_create_internal (there be dragons). The mutex is used only to 
synchronize producers and consumer (typically should be only one) so it 
shouldn’t block unless you run out of space in the queue. Similarly, with 
socket transport you can in theory run out of socket tx buffer, so you need to 
decide what to do in that case, i.e., probably drop. 

Cheers, 
Florin

> On Feb 11, 2021, at 8:49 AM, Neale Ranns  wrote:
> 
>
> Hi Ben,
>
> Not a silly idea  but given the shared memory’s mutex based read/write 
> mechanism, one can’t connect to VPP form within VPP. Maybe socket transport 
> for the API is an option, otherwise we considered a new ‘direct’ transport 
> for the API, which as you say would be a bit like RPC. The plugin could then 
> use the VAPI function wrappers over this direct transport.
>
> /neale
>
>
>
> From: Benoit Ganne (bganne) mailto:bga...@cisco.com>>
> Date: Thursday, 11 February 2021 at 17:17
> To: Neale Ranns mailto:ne...@graphiant.com>>, 
> vpp-dev@lists.fd.io   >
> Subject: RE: [vpp-dev] RFC: Interface Mirroring for Linux Network 
> Stackintegration
> 
> Hi Neale,
> 
> > this adds another plugin that acts as a netlink listener and invokes VPP
> > internal APIs (i.e. directly calls e.g. FIB and ip-neighbour) to add state
> > to VPP.
> [...]
> > Disadvantages:
> > 1.Not using VPP's binary API means you do not get the tracing
> > function, which in turn means you do not get the replay function. These
> > are both exceptionally useful field debugging tools.
> 
> Maybe a silly idea, but I guess you could always call the APIs from the 
> plugin instead of direct function calls, a bit like RPC to main thread? That 
> still means a single process to operate but you can use API replay.
> 
> ben
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18730): https://lists.fd.io/g/vpp-dev/message/18730
Mute This Topic: https://lists.fd.io/mt/80033164/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] RFC: Interface Mirroring for Linux Network Stackintegration

[Pim van Pelt]

Hoi,

Thanks for sharing, Neale. I think the simplicity of single-process
(advantage #1) is important to me, as keeping track of control plane / FIB
syncer jobs, etc, is tricky as restarts will desynchronize them from VPP
FIB and Kernel routing tables (or FRR, Bird, OpenBGPD etc), and those tend
to be costly and brittle. Making assurances on the synchronized state of
kernel and fib is a still tricky but easier if we can make some guarantees
on the syncer (plugin) and VPP itself being in the same process. Regarding
the second disadvantage, operators can opt to disable this (and other)
plugins on production environments, so that seems like a manageable risk. I
have no strong feelings on disadvantage #1 mostly because in my travels,
I've not needed API tracing for the netlink to interface/lcp/fib
interactions.

As with the previous patch, I think this plugin will open a world of
possibilities for dynamic routing environments such as ISPa wanting a
performant dataplane that VPP provides, but programmable by routing
controlplane daemons outside of the dataplane (eg. Bird, FRR,
OpenBGPd/OpenOSPF etc). I'd love to see it merged.

The patch itself - I think there are a few gotchas with Netlink, one of
which Matt has addressed with message queue that gets the messages from the
kernel/netlink socket as quickly as they become available,  and another
being inserting routes (of a BGP full table at ~840K IPv4 and 100K IPv6
currently) does take a while on multithreaded systems. Some stats counters
may be useful (FIB inserts/deletes, netlink messages consumed, etc); and
finally, I think making a decision on lcp_auto_intf() or not, would
simplify testing. I found that creating/deleting subinterfaces (including
qinq interfaces), changing interface params like MTU and link state, are
all quite difficult to do functional/regression testing for. Maybe it's
worth chosing one or the other, to reduce the total testing surface area.

groet,
Pim

On Thu, Feb 11, 2021 at 11:11 AM Neale Ranns  wrote:

>
>
> Dear All,
>
>
>
> Thank you for the positive comments, the patch was merged recently.
>
>
>
> Here is the next phase:
>
>   https://gerrit.fd.io/r/c/vpp/+/31122
>
>
>
> this adds another plugin that acts as a netlink listener and invokes VPP
> internal APIs (i.e. directly calls e.g. FIB and ip-neighbour) to add state
> to VPP. An alternative to using a plugin as the netlink listener is to have
> a separate process as the listener that invokes VPP’s official API; there
> are advantages and disadvantages to both.
>
> Advantages of a plugin:
>
>1. Simpler. The system has less processes to manage. There’s no need
>for state reconciliation (between kernel and VPP) if the separate process
>were to crash.
>
> Disadvantages:
>
>1. Not using VPP’s binary API means you do not get the tracing
>function, which in turn means you do not get the replay function. These are
>both exceptionally useful field debugging tools.
>2. More code in VPP process means more bugs that could crash VPP and
>hence stop packet forwarding.
>
>
>
> There are some general challenges with using netlink, but these challenges
> need to be overcome whichever choice one makes for integration, so I don’t
> discuss them here.
>
>
>
> Comments and suggestions please.
>
>
>
> Thanks,
>
> Neale
>
>
>
>
>
>
>
>
>
> *From: *Neale Ranns 
> *Date: *Friday, 22 January 2021 at 16:55
> *To: *vpp-dev@lists.fd.io 
> *Subject: *RFC: Interface Mirroring for Linux Network Stackintegration
>
>
>
> Dear All,
>
>
>
> I’d like to solicit comments for this proposed patch:
>
>   https://gerrit.fd.io/r/c/vpp/+/30759
>
>
>
> this is a scheme that aids with the integration of VPP with the Linux
> network stack by mirroring a [user defined] set of interfaces that VPP owns
> with tap/tun interfaces in the kernel. More details are in the .rst file.
>
>
>
> Thanks,
>
> /neale
>
>
>
> 
>
>

-- 
Pim van Pelt 
PBVP1-RIPE - http://www.ipng.nl/

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18729): https://lists.fd.io/g/vpp-dev/message/18729
Mute This Topic: https://lists.fd.io/mt/80033164/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] RFC: Interface Mirroring for Linux Network Stackintegration

[Neale Ranns]

Hi Ben,

Not a silly idea  but given the shared memory’s mutex based read/write 
mechanism, one can’t connect to VPP form within VPP. Maybe socket transport for 
the API is an option, otherwise we considered a new ‘direct’ transport for the 
API, which as you say would be a bit like RPC. The plugin could then use the 
VAPI function wrappers over this direct transport.

/neale



From: Benoit Ganne (bganne) 
Date: Thursday, 11 February 2021 at 17:17
To: Neale Ranns , vpp-dev@lists.fd.io 
Subject: RE: [vpp-dev] RFC: Interface Mirroring for Linux Network 
Stackintegration
Hi Neale,

> this adds another plugin that acts as a netlink listener and invokes VPP
> internal APIs (i.e. directly calls e.g. FIB and ip-neighbour) to add state
> to VPP.
[...]
> Disadvantages:
> 1.Not using VPP's binary API means you do not get the tracing
> function, which in turn means you do not get the replay function. These
> are both exceptionally useful field debugging tools.

Maybe a silly idea, but I guess you could always call the APIs from the plugin 
instead of direct function calls, a bit like RPC to main thread? That still 
means a single process to operate but you can use API replay.

ben

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18728): https://lists.fd.io/g/vpp-dev/message/18728
Mute This Topic: https://lists.fd.io/mt/80033164/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] RFC: Interface Mirroring for Linux Network Stackintegration

[Benoit Ganne (bganne) via lists.fd.io]

Hi Neale,

> this adds another plugin that acts as a netlink listener and invokes VPP
> internal APIs (i.e. directly calls e.g. FIB and ip-neighbour) to add state
> to VPP.
[...]
> Disadvantages:
> 1.Not using VPP's binary API means you do not get the tracing
> function, which in turn means you do not get the replay function. These
> are both exceptionally useful field debugging tools.

Maybe a silly idea, but I guess you could always call the APIs from the plugin 
instead of direct function calls, a bit like RPC to main thread? That still 
means a single process to operate but you can use API replay.

ben

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18727): https://lists.fd.io/g/vpp-dev/message/18727
Mute This Topic: https://lists.fd.io/mt/80033164/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64

[Dave Wallace]

Excellent!  Thanks Ben :)

-daw-


On 2/11/2021 11:06 AM, Benoit Ganne (bganne) wrote:

Hi Dave,

I am working on the ikev2 crashes, I think I identified the issue, stay tuned.

ben


-Original Message-
From: Dave Wallace 
Sent: mercredi 10 février 2021 22:49
To: v...@barachs.net; 'Paul Vinciguerra' 
Cc: Benoit Ganne (bganne) ; 'Ole Troan'
; 'vpp-dev' ; 'Juraj Linkeš'

Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64

Verified that the fix resolves the Bihash test failure (rebased 30734):

https://logs.fd.io/sandbox/vex-yul-rot-jenkins-2/daw_30773-vpp-verify-
master-ubuntu2004-aarch64/2/

There are still 6 ikev2 test failures from this test run that need to be
resolved.  These all have core file stack traces available (see
core.traceback.gz in the test directories.   All of these tests appear to
fail with this in the stacktrace:

#5  0x699683dc in ikev2_payload_add_data (data=data@entry=0x0,
c=, c=) at /w/workspace/daw_30773-vpp-
verify-master-ubuntu2004-aarch64/src/plugins/ikev2/ikev2_payload.c:136

Thanks,
-daw-



On 2/10/21 10:18 AM, Dave Wallace via lists.fd.io wrote:


Thanks Dave!
-daw-


On 2/10/2021 7:54 AM, v...@barachs.net 
wrote:


To make the pain go away, I quadrupled the vapi timeout for
the bihash test vectors. Turns out that a couple of tests run for almost 5
seconds (on aarch64, debug) which caused sporadic failures.



That’s probably about the right size hammer for this specific
problem.



FWIW... Dave



From: vpp-dev@lists.fd.io      On Behalf Of Paul
Vinciguerra
Sent: Tuesday, February 9, 2021 1:31 PM
To: Dave Barach  
Cc: Benoit Ganne (bganne) 
 ; Ole Troan 
 ; Dave Wallace 
 ; vpp-dev   ; Juraj Linkeš 

Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04
AARCH64



Hi Dave,



test_cli.py verifies the timeout checking code is working.



The groundwork to clean up the framework can be found here.
[0]



The test framework cannot distinguish between a timeout and a
more serious problem, so for now we assume vpp has died.



For test, I would have liked to do something similar to what
you do with growing your vectors.  If it is a true Timeout, rerun the test
with say 3/2 the failing value and output the timeout with the run.



We could have the testcases use different values for arm, but
when load or hw specs change, the tests will break again.



Paul



[0] https://gerrit.fd.io/r/c/vpp/+/24085







On Tue, Feb 9, 2021 at 8:37 AM Dave Barach mailto:v...@barachs.net> > wrote:

On aarch64, "make TEST=test_bihash test-debug" involves
a debug CLI timeout which is right on the hairy edge of being too short.



05:29:45,470 Calling cli_inband('cmd':'test bihash
threads 2 nbuckets 64000 careful 0 verbose
0\n','context':2,'_vl_msg_id':574)

05:29:45,471 TIMEOUT:: 5



This seems like the right incantation to fix it in
src/vppinfra/test/test_bihash.py, but apparently it is not:



error = self.vapi.cli("test bihash threads 2
nbuckets" +

  " 64000 careful 0 verbose
0", timeout=15)



Please advise... Thanks... Dave



-Original Message-
From: vpp-dev@lists.fd.io 
mailto:vpp-dev@lists.fd.io> > On Behalf Of Benoit
Ganne (bganne) via lists.fd.io 
Sent: Tuesday, February 9, 2021 8:06 AM
To: Dave Barach mailto:v...@barachs.net> >; 'Ole Troan' mailto:otr...@employees.org> >; 'Dave Wallace' mailto:dwallac...@gmail.com> >; vpp-dev@lists.fd.io 
Cc: 'Juraj Linkeš' mailto:juraj.lin...@pantheon.tech> >
Subject: Re: [vpp-dev] Make test failures on ubuntu-
20.04 AARCH64



I am going to have a look at IKE.



In case it is useful to others, thanks to QEMU you can
run a Ubuntu 20.04 ARM docker container on your x86 workstation:



# step 0: add support for multiarch (must be done once
after reboot) docker run --rm --privileged multiarch/qemu-user-static --
reset --persistent yes --credential yes



# step 1: create your chroot (must be done once - I am
sharing my homedir with my chroot and same UID/GID) 

Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64

[Benoit Ganne (bganne) via lists.fd.io]

Hi Dave,

I am working on the ikev2 crashes, I think I identified the issue, stay tuned.

ben

> -Original Message-
> From: Dave Wallace 
> Sent: mercredi 10 février 2021 22:49
> To: v...@barachs.net; 'Paul Vinciguerra' 
> Cc: Benoit Ganne (bganne) ; 'Ole Troan'
> ; 'vpp-dev' ; 'Juraj Linkeš'
> 
> Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
> 
> Verified that the fix resolves the Bihash test failure (rebased 30734):
> 
> https://logs.fd.io/sandbox/vex-yul-rot-jenkins-2/daw_30773-vpp-verify-
> master-ubuntu2004-aarch64/2/
> 
> There are still 6 ikev2 test failures from this test run that need to be
> resolved.  These all have core file stack traces available (see
> core.traceback.gz in the test directories.   All of these tests appear to
> fail with this in the stacktrace:
> 
> #5  0x699683dc in ikev2_payload_add_data (data=data@entry=0x0,
> c=, c=) at /w/workspace/daw_30773-vpp-
> verify-master-ubuntu2004-aarch64/src/plugins/ikev2/ikev2_payload.c:136
> 
> Thanks,
> -daw-
> 
> 
> 
> On 2/10/21 10:18 AM, Dave Wallace via lists.fd.io wrote:
> 
> 
>   Thanks Dave!
>   -daw-
> 
> 
>   On 2/10/2021 7:54 AM, v...@barachs.net 
> wrote:
> 
> 
>   To make the pain go away, I quadrupled the vapi timeout for
> the bihash test vectors. Turns out that a couple of tests run for almost 5
> seconds (on aarch64, debug) which caused sporadic failures.
> 
> 
> 
>   That’s probably about the right size hammer for this specific
> problem.
> 
> 
> 
>   FWIW... Dave
> 
> 
> 
>   From: vpp-dev@lists.fd.io    d...@lists.fd.io>   On Behalf Of Paul
> Vinciguerra
>   Sent: Tuesday, February 9, 2021 1:31 PM
>   To: Dave Barach  
>   Cc: Benoit Ganne (bganne) 
>  ; Ole Troan 
>  ; Dave Wallace 
>  ; vpp-dev   d...@lists.fd.io> ; Juraj Linkeš 
> 
>   Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04
> AARCH64
> 
> 
> 
>   Hi Dave,
> 
> 
> 
>   test_cli.py verifies the timeout checking code is working.
> 
> 
> 
>   The groundwork to clean up the framework can be found here.
> [0]
> 
> 
> 
>   The test framework cannot distinguish between a timeout and a
> more serious problem, so for now we assume vpp has died.
> 
> 
> 
>   For test, I would have liked to do something similar to what
> you do with growing your vectors.  If it is a true Timeout, rerun the test
> with say 3/2 the failing value and output the timeout with the run.
> 
> 
> 
>   We could have the testcases use different values for arm, but
> when load or hw specs change, the tests will break again.
> 
> 
> 
>   Paul
> 
> 
> 
>   [0] https://gerrit.fd.io/r/c/vpp/+/24085
> 
> 
> 
> 
> 
> 
> 
>   On Tue, Feb 9, 2021 at 8:37 AM Dave Barach   > wrote:
> 
>   On aarch64, "make TEST=test_bihash test-debug" involves
> a debug CLI timeout which is right on the hairy edge of being too short.
> 
> 
> 
>   05:29:45,470 Calling cli_inband('cmd':'test bihash
> threads 2 nbuckets 64000 careful 0 verbose
> 0\n','context':2,'_vl_msg_id':574)
> 
>   05:29:45,471 TIMEOUT:: 5
> 
> 
> 
>   This seems like the right incantation to fix it in
> src/vppinfra/test/test_bihash.py, but apparently it is not:
> 
> 
> 
>   error = self.vapi.cli("test bihash threads 2
> nbuckets" +
> 
> " 64000 careful 0 verbose
> 0", timeout=15)
> 
> 
> 
>   Please advise... Thanks... Dave
> 
> 
> 
>   -Original Message-
>   From: vpp-dev@lists.fd.io 
> mailto:vpp-dev@lists.fd.io> > On Behalf Of Benoit
> Ganne (bganne) via lists.fd.io 
>   Sent: Tuesday, February 9, 2021 8:06 AM
>   To: Dave Barach   >; 'Ole Troan'   >; 'Dave Wallace'   >; vpp-dev@lists.fd.io  d...@lists.fd.io>
>   Cc: 'Juraj Linkeš'   >
>   Subject: Re: [vpp-dev] Make test failures on ubuntu-
> 20.04 AARCH64
> 
> 
> 
>   I am going to have a look at IKE.
> 
> 
> 
>   In case it is useful to others, thanks to QEMU you can
> run a Ubuntu 20.04 ARM docker container on your x86 workstation:
> 
> 
> 
>   # step 0: add support for multiarch (must be done once
> after reboot) docker run --rm --privileged