Hi Dave, I am working on the ikev2 crashes, I think I identified the issue, stay tuned.
ben > -----Original Message----- > From: Dave Wallace <dwallac...@gmail.com> > Sent: mercredi 10 février 2021 22:49 > To: v...@barachs.net; 'Paul Vinciguerra' <pvi...@vinciconsulting.com> > Cc: Benoit Ganne (bganne) <bga...@cisco.com>; 'Ole Troan' > <otr...@employees.org>; 'vpp-dev' <vpp-dev@lists.fd.io>; 'Juraj Linkeš' > <juraj.lin...@pantheon.tech> > Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64 > > Verified that the fix resolves the Bihash test failure (rebased 30734): > > https://logs.fd.io/sandbox/vex-yul-rot-jenkins-2/daw_30773-vpp-verify- > master-ubuntu2004-aarch64/2/ > > There are still 6 ikev2 test failures from this test run that need to be > resolved. These all have core file stack traces available (see > core.traceback.gz in the test directories. All of these tests appear to > fail with this in the stacktrace: > > #5 0x0000ffff699683dc in ikev2_payload_add_data (data=data@entry=0x0, > c=<optimized out>, c=<optimized out>) at /w/workspace/daw_30773-vpp- > verify-master-ubuntu2004-aarch64/src/plugins/ikev2/ikev2_payload.c:136 > > Thanks, > -daw- > > > > On 2/10/21 10:18 AM, Dave Wallace via lists.fd.io wrote: > > > Thanks Dave! > -daw- > > > On 2/10/2021 7:54 AM, v...@barachs.net <mailto:v...@barachs.net> > wrote: > > > To make the pain go away, I quadrupled the vapi timeout for > the bihash test vectors. Turns out that a couple of tests run for almost 5 > seconds (on aarch64, debug) which caused sporadic failures. > > > > That’s probably about the right size hammer for this specific > problem. > > > > FWIW... Dave > > > > From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> <vpp- > d...@lists.fd.io> <mailto:vpp-dev@lists.fd.io> On Behalf Of Paul > Vinciguerra > Sent: Tuesday, February 9, 2021 1:31 PM > To: Dave Barach <v...@barachs.net> <mailto:v...@barachs.net> > Cc: Benoit Ganne (bganne) <bga...@cisco.com> > <mailto:bga...@cisco.com> ; Ole Troan <otr...@employees.org> > <mailto:otr...@employees.org> ; Dave Wallace <dwallac...@gmail.com> > <mailto:dwallac...@gmail.com> ; vpp-dev <vpp-dev@lists.fd.io> <mailto:vpp- > d...@lists.fd.io> ; Juraj Linkeš <juraj.lin...@pantheon.tech> > <mailto:juraj.lin...@pantheon.tech> > Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 > AARCH64 > > > > Hi Dave, > > > > test_cli.py verifies the timeout checking code is working. > > > > The groundwork to clean up the framework can be found here. > [0] > > > > The test framework cannot distinguish between a timeout and a > more serious problem, so for now we assume vpp has died. > > > > For test, I would have liked to do something similar to what > you do with growing your vectors. If it is a true Timeout, rerun the test > with say 3/2 the failing value and output the timeout with the run. > > > > We could have the testcases use different values for arm, but > when load or hw specs change, the tests will break again. > > > > Paul > > > > [0] https://gerrit.fd.io/r/c/vpp/+/24085 > > > > > > > > On Tue, Feb 9, 2021 at 8:37 AM Dave Barach <v...@barachs.net > <mailto:v...@barachs.net> > wrote: > > On aarch64, "make TEST=test_bihash test-debug" involves > a debug CLI timeout which is right on the hairy edge of being too short. > > > > 05:29:45,470 Calling cli_inband('cmd':'test bihash > threads 2 nbuckets 64000 careful 0 verbose > 0\n','context':2,'_vl_msg_id':574) > > 05:29:45,471 TIMEOUT:: 5 > > > > This seems like the right incantation to fix it in > src/vppinfra/test/test_bihash.py, but apparently it is not: > > > > error = self.vapi.cli("test bihash threads 2 > nbuckets" + > > " 64000 careful 0 verbose > 0", timeout=15) > > > > Please advise... Thanks... Dave > > > > -----Original Message----- > From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > <vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > On Behalf Of Benoit > Ganne (bganne) via lists.fd.io <http://lists.fd.io> > Sent: Tuesday, February 9, 2021 8:06 AM > To: Dave Barach <v...@barachs.net > <mailto:v...@barachs.net> >; 'Ole Troan' <otr...@employees.org > <mailto:otr...@employees.org> >; 'Dave Wallace' <dwallac...@gmail.com > <mailto:dwallac...@gmail.com> >; vpp-dev@lists.fd.io <mailto:vpp- > d...@lists.fd.io> > Cc: 'Juraj Linkeš' <juraj.lin...@pantheon.tech > <mailto:juraj.lin...@pantheon.tech> > > Subject: Re: [vpp-dev] Make test failures on ubuntu- > 20.04 AARCH64 > > > > I am going to have a look at IKE. > > > > In case it is useful to others, thanks to QEMU you can > run a Ubuntu 20.04 ARM docker container on your x86 workstation: > > > > # step 0: add support for multiarch (must be done once > after reboot) docker run --rm --privileged multiarch/qemu-user-static -- > reset --persistent yes --credential yes > > > > # step 1: create your chroot (must be done once - I am > sharing my homedir with my chroot and same UID/GID) docker run --name > aarch64_u2004 --privileged --net host -v $HOME:$HOME -v /dev:/dev -td > arm64v8/ubuntu:20.04 /bin/bash > > > > # step 2: update and add user inside container docker > container exec aarch64_u2004 sh -c "apt -qy update && apt dist-upgrade -qy > && apt install -qy vim sudo make git && groupadd -g $(id -rg) $USER && > useradd -u $(id -ru) -g $(id -rg) -M -d $HOME -s /bin/bash $USER && echo > '$USER ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers && echo aarch64_u2004 > > /etc/debian_chroot" > > > > # step 3: log into ARM chroot > > docker container exec -it aarch64_deb10 /bin/login -f > $USER > > > > Best > > ben > > > > > -----Original Message----- > > > From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > <vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > On Behalf Of Dave > > > Barach > > > Sent: mardi 9 février 2021 12:47 > > > To: 'Ole Troan' <otr...@employees.org > <mailto:otr...@employees.org> >; 'Dave Wallace' > > > <dwallac...@gmail.com <mailto:dwallac...@gmail.com> >; > vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > > > Cc: 'Juraj Linkeš' <juraj.lin...@pantheon.tech > <mailto:juraj.lin...@pantheon.tech> > > > > Subject: Re: [vpp-dev] Make test failures on ubuntu- > 20.04 AARCH64 > > > > > > Would be glad to take a look at the aarch64 bihash > failure, modulo the > > > fact that the LF data center resources I've used > before seem to have > > > had their credentials changed. > > > > > > Please unicast appropriate (ip, userid, password) > tuples. > > > > > > Thanks... Dave > > > > > > ssh vppdev@10.30.51.65 <mailto:vppdev@10.30.51.65> > > > Warning: Permanently added '10.30.51.65' (ECDSA) to > the list of known > > > hosts. > > > vppdev@10.30.51.65's <mailto:vppdev@10.30.51.65's> > password: > > > Permission denied, please try again. > > > vppdev@10.30.51.65's <mailto:vppdev@10.30.51.65's> > password: > > > Permission denied, please try again. > > > vppdev@10.30.51.65's <mailto:vppdev@10.30.51.65's> > password: > > > Received disconnect from 10.30.51.65 port 22:2: Too > many > > > authentication failures Disconnected from 10.30.51.65 > port 22 $ ssh > > > vppdev@10.30.51.68 <mailto:vppdev@10.30.51.68> > > > Warning: Permanently added '10.30.51.68' (ECDSA) to > the list of known > > > hosts. > > > vppdev@10.30.51.68's <mailto:vppdev@10.30.51.68's> > password: > > > Permission denied, please try again. > > > vppdev@10.30.51.68's <mailto:vppdev@10.30.51.68's> > password: > > > Permission denied, please try again. > > > vppdev@10.30.51.68's <mailto:vppdev@10.30.51.68's> > password: > > > Received disconnect from 10.30.51.68 port 22:2: Too > many > > > authentication failures Disconnected from 10.30.51.68 > port 22 > > > > > > -----Original Message----- > > > From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > <vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > On Behalf Of Ole Troan > > > Sent: Tuesday, February 9, 2021 5:20 AM > > > To: Dave Wallace <dwallac...@gmail.com > <mailto:dwallac...@gmail.com> >; vpp-dev@lists.fd.io <mailto:vpp- > d...@lists.fd.io> > > > Cc: Juraj Linkeš <juraj.lin...@pantheon.tech > <mailto:juraj.lin...@pantheon.tech> > > > > Subject: Re: [vpp-dev] Make test failures on ubuntu- > 20.04 AARCH64 > > > > > > And just to be clear, the failing tests are in the > QUIC, bihash and > > > IKEv2 features. > > > > > > > [4] > > > https://logs.fd.io/sandbox/vex-yul-rot-jenkins- > 2/daw_30773_31019-vpp- <https://logs.fd.io/sandbox/vex-yul-rot-jenkins- > 2/daw_30773_31019-vpp-> > > > verify- > > > master-ubuntu2004-aarch64/4/ > > > > vpp-unittest-QUICEchoIntMStreamTestCase- > f5ywwo0l-FAILED > > > > vpp-unittest-TestBihash-_iohsbtr-FAILED > > > > vpp-unittest-TestInitiatorDelSAFromResponder- > dgqdg78o-FAILED > > > > vpp-unittest-TestInitiatorKeepaliveMsg-0vo66yj2- > FAILED > > > > vpp-unittest-TestInitiatorNATT-edyneyju-FAILED > > > > vpp-unittest-TestInitiatorPsk-m3vrysyi-FAILED > > > > vpp-unittest-TestInitiatorRekey-7lrb0f8t-FAILED > > > > vpp-unittest-TestInitiatorRequestWindowSize- > 4y4stlww-FAILED > > > > > > > > > > > > > > > > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18725): https://lists.fd.io/g/vpp-dev/message/18725 Mute This Topic: https://lists.fd.io/mt/80495347/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
