That’s an interesting failure. Is the test machine running out of memory?
The extended tests are unstable on my server, so I do see quite a number of failures. However this: make test-ext TEST=vcl.VCLCutThruTestCase.test_vcl_cut_thru_uni_dir_nsock runs just fine. After the latest test framework changes, are we running multiple tests/vpps in parallel? I suspect that may be a source of issues. Florin > On Nov 15, 2018, at 12:11 PM, Klement Sekera via Lists.Fd.Io > <ksekera=cisco....@lists.fd.io> wrote: > > I'm seeing timeouts and coredumps... > > e.g. > > #6 0x00007f9ba0404eb6 in svm_msg_q_try_lock (mq=0x204009440) > at /home/ksekera/vpp/src/svm/message_queue.h:299 > 299 return pthread_mutex_trylock (&mq->q->mutex); > (gdb) p mq > $1 = (svm_msg_q_t *) 0x204009440 > (gdb) p mq->q > $2 = (svm_queue_t *) 0x0 > > which is part of > > #4 <signal handler called> > #5 __pthread_mutex_trylock (mutex=0x0) at ../nptl/pthread_mutex_trylock.c:39 > #6 0x00007f9ba0404eb6 in svm_msg_q_try_lock (mq=0x204009440) > at /home/ksekera/vpp/src/svm/message_queue.h:299 > #7 0x00007f9ba04055d5 in svm_msg_q_lock_and_alloc_msg_w_ring > (mq=0x204009440, > ring_index=1, noblock=1 '\001', msg=0x7f9b5f7c2a80) > at /home/ksekera/vpp/src/svm/message_queue.c:121 > #8 0x00007f9ba14be449 in mq_try_lock_and_alloc_msg (app_mq=0x204009440, > msg=0x7f9b5f7c2a80) at /home/ksekera/vpp/src/vnet/session/session_api.c:407 > #9 0x00007f9ba14be509 in mq_send_session_accepted_cb (s=0x7f9b60351400) > at /home/ksekera/vpp/src/vnet/session/session_api.c:432 > #10 0x00007f9ba1496ba0 in application_local_session_connect ( > client_wrk=0x7f9b60805800, server_wrk=0x7f9b60805780, ll=0x7f9b5f4c9e40, > opaque=0) at /home/ksekera/vpp/src/vnet/session/application.c:1646 > #11 0x00007f9ba14a5a62 in application_connect (a=0x7f9b5f7c2d30) > at /home/ksekera/vpp/src/vnet/session/application_interface.c:327 > ---Type <return> to continue, or q <return> to quit--- > #12 0x00007f9ba14a69fd in vnet_connect (a=0x7f9b5f7c2d30) > at /home/ksekera/vpp/src/vnet/session/application_interface.c:673 > #13 0x00007f9ba14c0f27 in vl_api_connect_sock_t_handler (mp=0x1300a6218) > at /home/ksekera/vpp/src/vnet/session/session_api.c:1305 > #14 0x00007f9ba1b6cb25 in vl_msg_api_handler_with_vm_node ( > am=0x7f9ba1d7dc60 <api_main>, the_msg=0x1300a6218, > vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000) > at /home/ksekera/vpp/src/vlibapi/api_shared.c:502 > #15 0x00007f9ba1b39114 in void_mem_api_handle_msg_i ( > am=0x7f9ba1d7dc60 <api_main>, vm=0x7f9ba08fc2c0 <vlib_global_main>, > node=0x7f9b5f7ba000, q=0x13004c440) > at /home/ksekera/vpp/src/vlibmemory/memory_api.c:700 > #16 0x00007f9ba1b39183 in vl_mem_api_handle_msg_main ( > vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000) > at /home/ksekera/vpp/src/vlibmemory/memory_api.c:710 > #17 0x00007f9ba1b572dd in vl_api_clnt_process ( > vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000, f=0x0) > at /home/ksekera/vpp/src/vlibmemory/vlib_api.c:350 > #18 0x00007f9ba0674a11 in vlib_process_bootstrap (_a=140305300978672) > at /home/ksekera/vpp/src/vlib/main.c:1276 > #19 0x00007f9b9fef4e74 in clib_calljmp () > from > /home/ksekera/vpp/build-root/install-vpp_debug-native/vpp/lib/libvppinfra.so.19.01 > > could this be the result of a timeout and the killing of the child > process? > > Thanks, > Klement > > > Quoting Dave Wallace (2018-11-15 20:27:55) >> Klement, >> >> I just pulled the top-of-tree on master and ran only VCL tests on my 18.04 >> box and they all passed (see below). Another strange thing about your >> failure is that the test that failed is NOT an extended test. >> >> I'm currently working on a patch ([1]https://gerrit.fd.io/r/#/c/13215/) to >> shorten the run time for the extended tests and convert them to regular >> tests. In the past, I have seen some unexplained failures of some of the >> extended tests. I'll let you know if I encounter any of them again. >> >> Thanks, >> -daw- >> >> ----- %< ----- >> TEST=vcl.* make test-ext >> . . . >> make[2]: Leaving directory '/scratch/dwallacelf/lf/vpp/test/ext' >> >> ============================================================================== >> Sanity test case - verify if VPP is able to start >> >> ============================================================================== >> Running tests using custom test runner >> Active filters: file=test_vcl.py, class=None, function=None >> Adding tests from directory tree /scratch/dwallacelf/lf/vpp/test >> 28 out of 858 tests match specified filters >> >> ============================================================================== >> VCL Thru Host Stack Tests >> >> ============================================================================== >> run LDP thru host stack echo test >> OK >> run VCL thru host stack echo test >> OK >> >> ============================================================================== >> VCL Thru Host Stack Extended B Tests >> >> ============================================================================== >> run LDP thru host stack bi-directional (multiple sockets) test >> OK >> >> ============================================================================== >> VCL Thru Host Stack Iperf Tests >> >> ============================================================================== >> run LDP thru host stack iperf3 test >> OK >> >> ============================================================================== >> VCL Thru Host Stack Extended D Tests >> >> ============================================================================== >> run VCL thru host stack uni-directional (multiple sockets) test >> OK >> >> ============================================================================== >> VCL IPv6 Thru Host Stack Tests >> >> ============================================================================== >> run LDP IPv6 thru host stack echo test >> OK >> run VCL IPv6 thru host stack echo test >> OK >> >> ============================================================================== >> VCL Cut Thru Tests >> >> ============================================================================== >> run LDP cut thru bi-directional (multiple sockets) test >> OK >> run LDP cut thru echo test >> OK >> run LDP cut thru iperf3 test >> OK >> run LDP cut thru uni-directional (multiple sockets) test >> OK >> run VCL cut thru bi-directional (multiple sockets) test >> OK >> run VCL cut thru echo test >> OK >> run VCL cut thru uni-directional (multiple sockets) test >> OK >> >> ============================================================================== >> VCL Thru Host Stack Extended C Tests >> >> ============================================================================== >> run LDP thru host stack uni-directional (multiple sockets) test >> OK >> >> ============================================================================== >> VCL Thru Host Stack NSession Bidir Tests >> >> ============================================================================== >> run VCL thru host stack bi-directional (multiple sockets) test >> OK >> >> ============================================================================== >> VCL IPv6 Thru Host Stack Extended A Tests >> >> ============================================================================== >> run VCL thru host stack bi-directional (multiple sockets) test >> OK >> >> ============================================================================== >> VCL IPv6 Cut Thru Tests >> >> ============================================================================== >> run LDP IPv6 cut thru bi-directional (multiple sockets) test >> OK >> run LDP IPv6 cut thru echo test >> OK >> run LDP IPv6 cut thru iperf3 test >> OK >> run LDP IPv6 cut thru uni-directional (multiple sockets) test >> OK >> run VCL IPv6 cut thru bi-directional (multiple sockets) test >> OK >> run VCL IPv6 cut thru echo test >> OK >> run VCL IPv6 cut thru uni-directional (multiple sockets) test >> OK >> >> ============================================================================== >> VCL IPv6 Thru Host Stack Extended C Tests >> >> ============================================================================== >> run LDP thru host stack uni-directional (multiple sockets) test >> OK >> >> ============================================================================== >> VCL IPv6 Thru Host Stack Extended B Tests >> >> ============================================================================== >> run LDP thru host stack bi-directional (multiple sockets) test >> OK >> >> ============================================================================== >> VCL IPv6 Thru Host Stack Iperf Tests >> >> ============================================================================== >> run LDP thru host stack iperf3 test >> OK >> >> ============================================================================== >> VCL IPv6 Thru Host Stack Extended D Tests >> >> ============================================================================== >> run VCL thru host stack uni-directional (multiple sockets) test >> OK >> >> Ran 28 tests in 250.800s >> >> OK >> >> >> ============================================================================== >> TEST RESULTS: >> Scheduled tests: 28 >> Executed tests: 28 >> Passed tests: 28 >> >> ============================================================================== >> >> Test run was successful >> ----- %< ----- >> On 11/15/2018 5:47 AM, Klement Sekera via Lists.Fd.Io wrote: >> >> Hi all, >> >> I'm seeing failures on master branch on ubuntu 18.04 when invoking `make >> test-ext` >> >> FAILURES AND ERRORS IN TESTS: >> Testcase name: VCL Cut Thru Tests >> FAILURE: run VCL cut thru uni-directional (multiple sockets) test >> Testcase name: L2BD Test Case >> ERROR: L2BD MAC learning dual-loop test >> ERROR: L2BD MAC learning dual-loop test >> ERROR: L2BD MAC learning single-loop test >> Testcase name: Classifier PBR Test Case >> ERROR: IP PBR test >> >> digging a bit further, L2BD failure also occurs in `make test-debug`, >> while it doesn't appear in `make test`. This is a core due to assert. >> >> I think we should run both `make test` (release build) and `make >> test-debug` (debug build) as part of verify process. If it was up to me, >> I would run all the tests which we have in the verify job. >> >> Thoughts? >> >> Regards, >> Klement >> >> >> -=-=-=-=-=-=-=-=-=-=-=- >> Links: You receive all messages sent to this group. >> >> View/Reply Online (#11257): [2]https://lists.fd.io/g/vpp-dev/message/11257 >> Mute This Topic: [3]https://lists.fd.io/mt/28144643/675079 >> Group Owner: [4]vpp-dev+ow...@lists.fd.io >> Unsubscribe: [5]https://lists.fd.io/g/vpp-dev/unsub >> [[6]dwallac...@gmail.com] >> -=-=-=-=-=-=-=-=-=-=-=- >> >> References >> >> Visible links >> 1. https://gerrit.fd.io/r/#/c/13215/ >> 2. https://lists.fd.io/g/vpp-dev/message/11257 >> 3. https://lists.fd.io/mt/28144643/675079 >> 4. mailto:vpp-dev+ow...@lists.fd.io >> 5. https://lists.fd.io/g/vpp-dev/unsub >> 6. mailto:dwallac...@gmail.com > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#11270): https://lists.fd.io/g/vpp-dev/message/11270 > Mute This Topic: https://lists.fd.io/mt/28144643/675152 > Group Owner: vpp-dev+ow...@lists.fd.io > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [fcoras.li...@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11271): https://lists.fd.io/g/vpp-dev/message/11271 Mute This Topic: https://lists.fd.io/mt/28144643/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
