I also noticed that although it shouldn’t be different from the vcl test apps which seem to be properly killed on test failure. Anyway, it’s on my todo list, if nobody beats me to it.
Florin > On Nov 16, 2018, at 5:27 AM, Klement Sekera <ksek...@cisco.com> wrote: > > I've also noticed that quite often a binary called sock_test_client is > left running after a vpp crash or test failure. What's worse, it eats > 100% CPU. > > Quoting Florin Coras (2018-11-16 00:40:06) >> Thanks, Dave! >> I’ll take a look at those as soon as I can. I’m running multiple >> connections between 2 vpp hosts without issue, so it’s either a >> cut-through session issue or it has to do with how we setup vpp for vrf >> leaking. >> Cheers, >> Florin >> >> On Nov 15, 2018, at 3:00 PM, Dave Wallace <[1]dwallac...@gmail.com> >> wrote: >> Same here. However, in the same workspace where all tests passed, I can >> get this test case to fail consistently: >> >> EXTENDED_TESTS=y TEST=vcl.VCLThruHostStackExtendedBTestCase.* make test >> EXTENDED_TESTS=y TEST=vcl.VCLIpv6ThruHostStackExtendedBTestCase.* make >> test >> >> In patch 13215, I discovered that making these test cases NOT run >> multiple sockets in parallel the test passes. My latest patch to that >> has the multiple sockets option commented out with "# ouch! Host Stack >> Bug?" so that all tests pass. >> >> Thanks, >> -daw- >> On 11/15/2018 4:16 PM, Florin Coras wrote: >> >> That’s an interesting failure. Is the test machine running out of memory? >> >> The extended tests are unstable on my server, so I do see quite a number of >> failures. However this: >> >> make test-ext TEST=vcl.VCLCutThruTestCase.test_vcl_cut_thru_uni_dir_nsock >> >> runs just fine. After the latest test framework changes, are we running >> multiple tests/vpps in parallel? I suspect that may be a source of issues. >> >> Florin >> >> >> On Nov 15, 2018, at 12:11 PM, Klement Sekera via Lists.Fd.Io >> [2]<ksekera=cisco....@lists.fd.io> wrote: >> >> I'm seeing timeouts and coredumps... >> >> e.g. >> >> #6 0x00007f9ba0404eb6 in svm_msg_q_try_lock (mq=0x204009440) >> at /home/ksekera/vpp/src/svm/message_queue.h:299 >> 299 return pthread_mutex_trylock (&mq->q->mutex); >> (gdb) p mq >> $1 = (svm_msg_q_t *) 0x204009440 >> (gdb) p mq->q >> $2 = (svm_queue_t *) 0x0 >> >> which is part of >> >> #4 <signal handler called> >> #5 __pthread_mutex_trylock (mutex=0x0) at ../nptl/pthread_mutex_trylock.c:39 >> #6 0x00007f9ba0404eb6 in svm_msg_q_try_lock (mq=0x204009440) >> at /home/ksekera/vpp/src/svm/message_queue.h:299 >> #7 0x00007f9ba04055d5 in svm_msg_q_lock_and_alloc_msg_w_ring >> (mq=0x204009440, >> ring_index=1, noblock=1 '\001', msg=0x7f9b5f7c2a80) >> at /home/ksekera/vpp/src/svm/message_queue.c:121 >> #8 0x00007f9ba14be449 in mq_try_lock_and_alloc_msg (app_mq=0x204009440, >> msg=0x7f9b5f7c2a80) at >> /home/ksekera/vpp/src/vnet/session/session_api.c:407 >> #9 0x00007f9ba14be509 in mq_send_session_accepted_cb (s=0x7f9b60351400) >> at /home/ksekera/vpp/src/vnet/session/session_api.c:432 >> #10 0x00007f9ba1496ba0 in application_local_session_connect ( >> client_wrk=0x7f9b60805800, server_wrk=0x7f9b60805780, ll=0x7f9b5f4c9e40, >> opaque=0) at /home/ksekera/vpp/src/vnet/session/application.c:1646 >> #11 0x00007f9ba14a5a62 in application_connect (a=0x7f9b5f7c2d30) >> at /home/ksekera/vpp/src/vnet/session/application_interface.c:327 >> ---Type <return> to continue, or q <return> to quit--- >> #12 0x00007f9ba14a69fd in vnet_connect (a=0x7f9b5f7c2d30) >> at /home/ksekera/vpp/src/vnet/session/application_interface.c:673 >> #13 0x00007f9ba14c0f27 in vl_api_connect_sock_t_handler (mp=0x1300a6218) >> at /home/ksekera/vpp/src/vnet/session/session_api.c:1305 >> #14 0x00007f9ba1b6cb25 in vl_msg_api_handler_with_vm_node ( >> am=0x7f9ba1d7dc60 <api_main>, the_msg=0x1300a6218, >> vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000) >> at /home/ksekera/vpp/src/vlibapi/api_shared.c:502 >> #15 0x00007f9ba1b39114 in void_mem_api_handle_msg_i ( >> am=0x7f9ba1d7dc60 <api_main>, vm=0x7f9ba08fc2c0 <vlib_global_main>, >> node=0x7f9b5f7ba000, q=0x13004c440) >> at /home/ksekera/vpp/src/vlibmemory/memory_api.c:700 >> #16 0x00007f9ba1b39183 in vl_mem_api_handle_msg_main ( >> vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000) >> at /home/ksekera/vpp/src/vlibmemory/memory_api.c:710 >> #17 0x00007f9ba1b572dd in vl_api_clnt_process ( >> vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000, f=0x0) >> at /home/ksekera/vpp/src/vlibmemory/vlib_api.c:350 >> #18 0x00007f9ba0674a11 in vlib_process_bootstrap (_a=140305300978672) >> at /home/ksekera/vpp/src/vlib/main.c:1276 >> #19 0x00007f9b9fef4e74 in clib_calljmp () >> from >> /home/ksekera/vpp/build-root/install-vpp_debug-native/vpp/lib/libvppinfra.so.19.01 >> >> could this be the result of a timeout and the killing of the child >> process? >> >> Thanks, >> Klement >> >> >> Quoting Dave Wallace (2018-11-15 20:27:55) >> >> Klement, >> >> I just pulled the top-of-tree on master and ran only VCL tests on my 18.04 >> box and they all passed (see below). Another strange thing about your >> failure is that the test that failed is NOT an extended test. >> >> I'm currently working on a patch ([1][3]https://gerrit.fd.io/r/#/c/13215/) >> to >> shorten the run time for the extended tests and convert them to regular >> tests. In the past, I have seen some unexplained failures of some of the >> extended tests. I'll let you know if I encounter any of them again. >> >> Thanks, >> -daw- >> >> ----- %< ----- >> TEST=vcl.* make test-ext >> . . . >> make[2]: Leaving directory '/scratch/dwallacelf/lf/vpp/test/ext' >> >> ============================================================================== >> Sanity test case - verify if VPP is able to start >> >> ============================================================================== >> Running tests using custom test runner >> Active filters: file=test_vcl.py, class=None, function=None >> Adding tests from directory tree /scratch/dwallacelf/lf/vpp/test >> 28 out of 858 tests match specified filters >> >> ============================================================================== >> VCL Thru Host Stack Tests >> >> ============================================================================== >> run LDP thru host stack echo test >> OK >> run VCL thru host stack echo test >> OK >> >> ============================================================================== >> VCL Thru Host Stack Extended B Tests >> >> ============================================================================== >> run LDP thru host stack bi-directional (multiple sockets) test >> OK >> >> ============================================================================== >> VCL Thru Host Stack Iperf Tests >> >> ============================================================================== >> run LDP thru host stack iperf3 test >> OK >> >> ============================================================================== >> VCL Thru Host Stack Extended D Tests >> >> ============================================================================== >> run VCL thru host stack uni-directional (multiple sockets) test >> OK >> >> ============================================================================== >> VCL IPv6 Thru Host Stack Tests >> >> ============================================================================== >> run LDP IPv6 thru host stack echo test >> OK >> run VCL IPv6 thru host stack echo test >> OK >> >> ============================================================================== >> VCL Cut Thru Tests >> >> ============================================================================== >> run LDP cut thru bi-directional (multiple sockets) test >> OK >> run LDP cut thru echo test >> OK >> run LDP cut thru iperf3 test >> OK >> run LDP cut thru uni-directional (multiple sockets) test >> OK >> run VCL cut thru bi-directional (multiple sockets) test >> OK >> run VCL cut thru echo test >> OK >> run VCL cut thru uni-directional (multiple sockets) test >> OK >> >> ============================================================================== >> VCL Thru Host Stack Extended C Tests >> >> ============================================================================== >> run LDP thru host stack uni-directional (multiple sockets) test >> OK >> >> ============================================================================== >> VCL Thru Host Stack NSession Bidir Tests >> >> ============================================================================== >> run VCL thru host stack bi-directional (multiple sockets) test >> OK >> >> ============================================================================== >> VCL IPv6 Thru Host Stack Extended A Tests >> >> ============================================================================== >> run VCL thru host stack bi-directional (multiple sockets) test >> OK >> >> ============================================================================== >> VCL IPv6 Cut Thru Tests >> >> ============================================================================== >> run LDP IPv6 cut thru bi-directional (multiple sockets) test >> OK >> run LDP IPv6 cut thru echo test >> OK >> run LDP IPv6 cut thru iperf3 test >> OK >> run LDP IPv6 cut thru uni-directional (multiple sockets) test >> OK >> run VCL IPv6 cut thru bi-directional (multiple sockets) test >> OK >> run VCL IPv6 cut thru echo test >> OK >> run VCL IPv6 cut thru uni-directional (multiple sockets) test >> OK >> >> ============================================================================== >> VCL IPv6 Thru Host Stack Extended C Tests >> >> ============================================================================== >> run LDP thru host stack uni-directional (multiple sockets) test >> OK >> >> ============================================================================== >> VCL IPv6 Thru Host Stack Extended B Tests >> >> ============================================================================== >> run LDP thru host stack bi-directional (multiple sockets) test >> OK >> >> ============================================================================== >> VCL IPv6 Thru Host Stack Iperf Tests >> >> ============================================================================== >> run LDP thru host stack iperf3 test >> OK >> >> ============================================================================== >> VCL IPv6 Thru Host Stack Extended D Tests >> >> ============================================================================== >> run VCL thru host stack uni-directional (multiple sockets) test >> OK >> >> Ran 28 tests in 250.800s >> >> OK >> >> >> ============================================================================== >> TEST RESULTS: >> Scheduled tests: 28 >> Executed tests: 28 >> Passed tests: 28 >> >> ============================================================================== >> >> Test run was successful >> ----- %< ----- >> On 11/15/2018 5:47 AM, Klement Sekera via Lists.Fd.Io wrote: >> >> Hi all, >> >> I'm seeing failures on master branch on ubuntu 18.04 when invoking `make >> test-ext` >> >> FAILURES AND ERRORS IN TESTS: >> Testcase name: VCL Cut Thru Tests >> FAILURE: run VCL cut thru uni-directional (multiple sockets) test >> Testcase name: L2BD Test Case >> ERROR: L2BD MAC learning dual-loop test >> ERROR: L2BD MAC learning dual-loop test >> ERROR: L2BD MAC learning single-loop test >> Testcase name: Classifier PBR Test Case >> ERROR: IP PBR test >> >> digging a bit further, L2BD failure also occurs in `make test-debug`, >> while it doesn't appear in `make test`. This is a core due to assert. >> >> I think we should run both `make test` (release build) and `make >> test-debug` (debug build) as part of verify process. If it was up to me, >> I would run all the tests which we have in the verify job. >> >> Thoughts? >> >> Regards, >> Klement >> >> >> -=-=-=-=-=-=-=-=-=-=-=- >> Links: You receive all messages sent to this group. >> >> View/Reply Online (#11257): [2][4]https://lists.fd.io/g/vpp-dev/message/11257 >> Mute This Topic: [3][5]https://lists.fd.io/mt/28144643/675079 >> Group Owner: [[6]4]vpp-dev+ow...@lists.fd.io >> Unsubscribe: [5][7]https://lists.fd.io/g/vpp-dev/unsub >> [[[8]6]dwallac...@gmail.com] >> -=-=-=-=-=-=-=-=-=-=-=- >> >> References >> >> Visible links >> 1. [9]https://gerrit.fd.io/r/#/c/13215/ >> 2. [10]https://lists.fd.io/g/vpp-dev/message/11257 >> 3. [11]https://lists.fd.io/mt/28144643/675079 >> 4. [12]mailto:vpp-dev+ow...@lists.fd.io >> 5. [13]https://lists.fd.io/g/vpp-dev/unsub >> 6. [14]mailto:dwallac...@gmail.com >> >> -=-=-=-=-=-=-=-=-=-=-=- >> Links: You receive all messages sent to this group. >> >> View/Reply Online (#11270): [15]https://lists.fd.io/g/vpp-dev/message/11270 >> Mute This Topic: [16]https://lists.fd.io/mt/28144643/675152 >> Group Owner: [17]vpp-dev+ow...@lists.fd.io >> Unsubscribe: [18]https://lists.fd.io/g/vpp-dev/unsub >> [[19]fcoras.li...@gmail.com] >> -=-=-=-=-=-=-=-=-=-=-=- >> >> References >> >> Visible links >> 1. mailto:dwallac...@gmail.com >> 2. mailto:ksekera=cisco....@lists.fd.io >> 3. https://gerrit.fd.io/r/#/c/13215/ >> 4. https://lists.fd.io/g/vpp-dev/message/11257 >> 5. https://lists.fd.io/mt/28144643/675079 >> 6. mailto:4]vpp-dev+ow...@lists.fd.io >> 7. https://lists.fd.io/g/vpp-dev/unsub >> 8. mailto:6]dwallac...@gmail.com >> 9. https://gerrit.fd.io/r/#/c/13215/ >> 10. https://lists.fd.io/g/vpp-dev/message/11257 >> 11. https://lists.fd.io/mt/28144643/675079 >> 12. mailto:vpp-dev+ow...@lists.fd.io >> 13. https://lists.fd.io/g/vpp-dev/unsub >> 14. mailto:dwallac...@gmail.com >> 15. https://lists.fd.io/g/vpp-dev/message/11270 >> 16. https://lists.fd.io/mt/28144643/675152 >> 17. mailto:vpp-dev+ow...@lists.fd.io >> 18. https://lists.fd.io/g/vpp-dev/unsub >> 19. mailto:fcoras.li...@gmail.com
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11291): https://lists.fd.io/g/vpp-dev/message/11291 Mute This Topic: https://lists.fd.io/mt/28144643/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
