To make the pain go away, I quadrupled the vapi timeout for the bihash test vectors. Turns out that a couple of tests run for almost 5 seconds (on aarch64, debug) which caused sporadic failures.
That’s probably about the right size hammer for this specific problem. FWIW... Dave From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Paul Vinciguerra Sent: Tuesday, February 9, 2021 1:31 PM To: Dave Barach <v...@barachs.net> Cc: Benoit Ganne (bganne) <bga...@cisco.com>; Ole Troan <otr...@employees.org>; Dave Wallace <dwallac...@gmail.com>; vpp-dev <vpp-dev@lists.fd.io>; Juraj Linkeš <juraj.lin...@pantheon.tech> Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64 Hi Dave, test_cli.py verifies the timeout checking code is working. The groundwork to clean up the framework can be found here. [0] The test framework cannot distinguish between a timeout and a more serious problem, so for now we assume vpp has died. For test, I would have liked to do something similar to what you do with growing your vectors. If it is a true Timeout, rerun the test with say 3/2 the failing value and output the timeout with the run. We could have the testcases use different values for arm, but when load or hw specs change, the tests will break again. Paul [0] https://gerrit.fd.io/r/c/vpp/+/24085 On Tue, Feb 9, 2021 at 8:37 AM Dave Barach <v...@barachs.net <mailto:v...@barachs.net> > wrote: On aarch64, "make TEST=test_bihash test-debug" involves a debug CLI timeout which is right on the hairy edge of being too short. 05:29:45,470 Calling cli_inband('cmd':'test bihash threads 2 nbuckets 64000 careful 0 verbose 0\n','context':2,'_vl_msg_id':574) 05:29:45,471 TIMEOUT:: 5 This seems like the right incantation to fix it in src/vppinfra/test/test_bihash.py, but apparently it is not: error = self.vapi.cli("test bihash threads 2 nbuckets" + " 64000 careful 0 verbose 0", timeout=15) Please advise... Thanks... Dave -----Original Message----- From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> <vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > On Behalf Of Benoit Ganne (bganne) via lists.fd.io <http://lists.fd.io> Sent: Tuesday, February 9, 2021 8:06 AM To: Dave Barach <v...@barachs.net <mailto:v...@barachs.net> >; 'Ole Troan' <otr...@employees.org <mailto:otr...@employees.org> >; 'Dave Wallace' <dwallac...@gmail.com <mailto:dwallac...@gmail.com> >; vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> Cc: 'Juraj Linkeš' <juraj.lin...@pantheon.tech <mailto:juraj.lin...@pantheon.tech> > Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64 I am going to have a look at IKE. In case it is useful to others, thanks to QEMU you can run a Ubuntu 20.04 ARM docker container on your x86 workstation: # step 0: add support for multiarch (must be done once after reboot) docker run --rm --privileged multiarch/qemu-user-static --reset --persistent yes --credential yes # step 1: create your chroot (must be done once - I am sharing my homedir with my chroot and same UID/GID) docker run --name aarch64_u2004 --privileged --net host -v $HOME:$HOME -v /dev:/dev -td arm64v8/ubuntu:20.04 /bin/bash # step 2: update and add user inside container docker container exec aarch64_u2004 sh -c "apt -qy update && apt dist-upgrade -qy && apt install -qy vim sudo make git && groupadd -g $(id -rg) $USER && useradd -u $(id -ru) -g $(id -rg) -M -d $HOME -s /bin/bash $USER && echo '$USER ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers && echo aarch64_u2004 > /etc/debian_chroot" # step 3: log into ARM chroot docker container exec -it aarch64_deb10 /bin/login -f $USER Best ben > -----Original Message----- > From: <mailto:vpp-dev@lists.fd.io> vpp-dev@lists.fd.io < > <mailto:vpp-dev@lists.fd.io> vpp-dev@lists.fd.io> On Behalf Of Dave > Barach > Sent: mardi 9 février 2021 12:47 > To: 'Ole Troan' < <mailto:otr...@employees.org> otr...@employees.org>; 'Dave > Wallace' > < <mailto:dwallac...@gmail.com> dwallac...@gmail.com>; > <mailto:vpp-dev@lists.fd.io> vpp-dev@lists.fd.io > Cc: 'Juraj Linkeš' < <mailto:juraj.lin...@pantheon.tech> > juraj.lin...@pantheon.tech> > Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64 > > Would be glad to take a look at the aarch64 bihash failure, modulo the > fact that the LF data center resources I've used before seem to have > had their credentials changed. > > Please unicast appropriate (ip, userid, password) tuples. > > Thanks... Dave > > ssh <mailto:vppdev@10.30.51.65> vppdev@10.30.51.65 > Warning: Permanently added '10.30.51.65' (ECDSA) to the list of known > hosts. > <mailto:vppdev@10.30.51.65's> vppdev@10.30.51.65's password: > Permission denied, please try again. > <mailto:vppdev@10.30.51.65's> vppdev@10.30.51.65's password: > Permission denied, please try again. > <mailto:vppdev@10.30.51.65's> vppdev@10.30.51.65's password: > Received disconnect from 10.30.51.65 port 22:2: Too many > authentication failures Disconnected from 10.30.51.65 port 22 $ ssh > <mailto:vppdev@10.30.51.68> vppdev@10.30.51.68 > Warning: Permanently added '10.30.51.68' (ECDSA) to the list of known > hosts. > <mailto:vppdev@10.30.51.68's> vppdev@10.30.51.68's password: > Permission denied, please try again. > <mailto:vppdev@10.30.51.68's> vppdev@10.30.51.68's password: > Permission denied, please try again. > <mailto:vppdev@10.30.51.68's> vppdev@10.30.51.68's password: > Received disconnect from 10.30.51.68 port 22:2: Too many > authentication failures Disconnected from 10.30.51.68 port 22 > > -----Original Message----- > From: <mailto:vpp-dev@lists.fd.io> vpp-dev@lists.fd.io < > <mailto:vpp-dev@lists.fd.io> vpp-dev@lists.fd.io> On Behalf Of Ole Troan > Sent: Tuesday, February 9, 2021 5:20 AM > To: Dave Wallace < <mailto:dwallac...@gmail.com> dwallac...@gmail.com>; > <mailto:vpp-dev@lists.fd.io> vpp-dev@lists.fd.io > Cc: Juraj Linkeš < <mailto:juraj.lin...@pantheon.tech> > juraj.lin...@pantheon.tech> > Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64 > > And just to be clear, the failing tests are in the QUIC, bihash and > IKEv2 features. > > > [4] > <https://logs.fd.io/sandbox/vex-yul-rot-jenkins-2/daw_30773_31019-vpp-> > https://logs.fd.io/sandbox/vex-yul-rot-jenkins-2/daw_30773_31019-vpp- > verify- > master-ubuntu2004-aarch64/4/ > > vpp-unittest-QUICEchoIntMStreamTestCase-f5ywwo0l-FAILED > > vpp-unittest-TestBihash-_iohsbtr-FAILED > > vpp-unittest-TestInitiatorDelSAFromResponder-dgqdg78o-FAILED > > vpp-unittest-TestInitiatorKeepaliveMsg-0vo66yj2-FAILED > > vpp-unittest-TestInitiatorNATT-edyneyju-FAILED > > vpp-unittest-TestInitiatorPsk-m3vrysyi-FAILED > > vpp-unittest-TestInitiatorRekey-7lrb0f8t-FAILED > > vpp-unittest-TestInitiatorRequestWindowSize-4y4stlww-FAILED > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18716): https://lists.fd.io/g/vpp-dev/message/18716 Mute This Topic: https://lists.fd.io/mt/80495347/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
