Re: [vpp-dev] [csit-dev] vpp-csit-verify-virl-master - some more build failures

Dave Wallace Fri, 21 Oct 2016 09:42:28 -0700

Ed,

You are correct.


Thanks,
-daw-

On 10/21/16 11:48 AM, Edward Warnicke wrote:

Jan,

So just to make sure... this does not appear to be an infra issue atthis time, but rather a timeout waiting for a vpp restart? Correct?

Ed

On Fri, Oct 21, 2016 at 8:46 AM, Dave Wallace <dwallac...@gmail.com<mailto:dwallac...@gmail.com>> wrote:


    Jan,

    I see thatthere is a "sleep 3" in
    ...csit/resources/libraries/bash/dut_setup.sh after the vpp
    service restart in order to allow VPP tocomplete the restart
    case.  I'm wondering if VPP sometimes takes longer thanthat to
    restart?

    Do you recall if there was ever an attempt to find a determinant
    test to verify that VPP has completed initialization?

    I'm going to do some VPP restart testing on the VIRL testbed using
    the images built as part of
    https://jenkins.fd.io/job/vpp-csit-verify-virl-master/1851/
    <https://jenkins.fd.io/job/vpp-csit-verify-virl-master/1851/>.
    However if my theory is correct, I'm not comfortable resolving the
    problem by simply extending the sleep time.

    IMHO, theVPP restart/initialization time should bemeasuredand
    verified tobe within a specified period of time. It may be
    necessary to add instrumentation to VPP to achieve this goal.There
    should really be a separateset of unit tests (with different
    hugepage allocations) which pass/failprior to feature testing. At
    the very least, we should not be waiting an arbitrary amount of
    time during test setup and the actualtime to restart/complete
    initialization should be measured and logged.

    Thanks,
    -daw-

    On 10/21/16 10:00 AM, Jan Gelety -X (jgelety - PANTHEON
    TECHNOLOGIES at Cisco) wrote:


    Hello Dave,

    I had a look to failed CSIT runs under
    https://gerrit.fd.io/r/#/c/3511/ <https://gerrit.fd.io/r/#/c/3511/> :

    https://jenkins.fd.io/job/vpp-csit-verify-virl-master/1851/
    <https://jenkins.fd.io/job/vpp-csit-verify-virl-master/1851/>

    https://jenkins.fd.io/job/vpp-csit-verify-virl-master/1853/
    <https://jenkins.fd.io/job/vpp-csit-verify-virl-master/1853/>

    In both cases the timeout exception occurred during test case or
    test suite setup phase after restarting vpp (part of the script
    dut_setup.sh):

    Command_done_exec] 'cat /etc/vpp/startup.conf'

    [Command_start_exec] 'sudo -S service vpp restart'

    [Command_outputs]

    Current contents of stderr buffer:

    It seems that one of the DUTs has stop to communicate and not
    responding even after cca 30s later in the tear down of the test…
    But there was not a crash of DUT/VM as the frozen process for
    dut_setup.sh script is still there when the connection is working
    again (the next test case setup phase):

    20:54:26.422 FAIL timeout: Timeout exception.

    Current contents of stdout buffer:

    [Command_desc] Starting
    /tmp/openvpp-testing/resources/libraries/bash/dut_setup.sh

    [Command_start_exec] 'dpkg -l vpp\*'

    [Command_outputs] Desired=Unknown/Install/Remove/Purge/Hold

    |
    Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend

    |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)

    ||/ Name           Version Architecture Description

    
+++-==============-============================-============-=============================================

ii vpp 16.12-rc0~238-g0d1509f~b1851 amd64Vector Packet Processing--executables

ii vpp-dbg 16.12-rc0~238-g0d1509f~b1851 amd64Vector Packet Processing--debug symbols

ii vpp-dev 16.12-rc0~238-g0d1509f~b1851 amd64Vector Packet Processing--development support

ii vpp-dpdk-dev 16.12-rc0~238-g0d1509f~b1851 amd64Vector Packet Processing--development support


    ii vpp-dpdk-dkms  16.12-rc0~238-g0d1509f~b1851 amd64        DPDK
    2.1 igb_uio_driver

ii vpp-lib 16.12-rc0~238-g0d1509f~b1851 amd64Vector Packet Processing--runtime libraries

ii vpp-plugins 16.12-rc0~238-g0d1509f~b1851 amd64Vector Packet Processing--runtime plugins


    [Command_done_exec] 'dpkg -l vpp\*'

    [Command_start_exec] 'ps aux | grep vpp'

    [Command_outputs]

    USER PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

    root 10323 99.3  0.8 2477148 33580 ?       Rsl 14:54   0:13
    /usr/bin/vpp -c /etc/vpp/startup.conf

    root 10461  0.0  0.0  47296  3684 ?        Ss 14:54   0:00 sudo
    -Sn bash /tmp/openvpp-testing/resources/libraries/bash/dut_setup.sh

    root 10462  0.0  0.0  11540  2964 ?        S 14:54   0:00 bash
    /tmp/openvpp-testing/resources/libraries/bash/dut_setup.sh

    root 10465  0.0  0.0  13232   924 ?        S 14:54   0:00 grep vpp

    [Command_done_exec] 'ps aux | grep vpp'

    ...

    (the next tests case setup phase)

    20:55:06.951 FAIL timeout: Timeout exception.

    Current contents of stdout buffer:

    [Command_desc] Starting
    /tmp/openvpp-testing/resources/libraries/bash/dut_setup.sh

    [Command_start_exec] 'dpkg -l vpp\*'

    [Command_outputs] Desired=Unknown/Install/Remove/Purge/Hold

    |
    Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend

    |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)

    ||/ Name           Version Architecture Description

    
+++-==============-============================-============-=============================================