If 1 makes the arm jobs pass and supposedly run much faster, I’d be fine with that as well.
Florin > On Sep 10, 2018, at 7:48 AM, Marco Varlese <mvarl...@suse.de> wrote: > > On Mon, 2018-09-10 at 14:29 +0000, Ed Kern (ejk) wrote: >> At least three different possible actions to take at this point: (outside >> of fixing the issue) >> >> 1. remove make test attempt from arm build (return it to the way it was >> before a week ago). >> 2. lower timeout further my first thought would be in the 75 minute range >> (from 120) > I don't like this option mainly because it would still imply that developers > will have to wait for 75 minutes to see a patch verified... > The worst is that the job is a non-voting one hence having it running does > add any meaningful insight into the build result. > I'm in favour for the issue being resolved in the sandbox and eventually > moved to production when stable. > >> 3. remove job altogether. > I believe this should be the option to purse unless it could be fixed quickly. > As it is it only causes delays to the overall build / review / merging > process... > >> >> Im happy to push a patch that accomplishes the above or other options I >> haven’t thought of >> for this mail. >> >> Just let me know.. >> >> Ed >> >> >> >>> On Sep 10, 2018, at 4:31 AM, Marco Varlese <mvarl...@suse.de >>> <mailto:mvarl...@suse.de>> wrote: >>> Last example (today) can be found here >>> >>> 10:47:06 Not running extended tests (some tests will be skipped) >>> 10:47:06 Perform 4 attempts to pass the suite... >>> 10:47:10 *** Error in `python': double free or corruption (out): >>> 0x0000ffff75d483f0 *** >>> 12:27:54 Build timed out (after 120 minutes). Marking the build as failed. >>> >>> Patch: https://gerrit.fd.io/r/#/c/14744/ <https://gerrit.fd.io/r/#/c/14744/> >>> >>> Other jobs have finished more than one and half-hour ago and the patch >>> cannot be >>> marked Verified+1 because Jenkins is still waiting for the ARM job to >>> complete >>> (timeout = 120 minutes). IMHO it makes the overall patch submission and >>> review >>> very painful for authors and commiters. >>> >>> I would recommend (it has been done in the past for other jobs) to disable >>> this >>> job in production, move it again to sandbox, get it fixed and eventually >>> moved >>> again to production... >>> >>> - Marco >>> >>> On Fri, 2018-09-07 at 12:04 -0700, Florin Coras wrote: >>>> ARM jobs have not been working for some days now. That’s why their result >>>> is >>>> skipped. Timeout is 2h but probably we should drop that even further … >>>> >>>> Florin >>>> >>>>> On Sep 7, 2018, at 11:59 AM, Ole Troan <otr...@employees.org >>>>> <mailto:otr...@employees.org>> wrote: >>>>> Trying out a change in: >>>>> https://gerrit.fd.io/r/#/c/14732/ <https://gerrit.fd.io/r/#/c/14732/> >>>>> >>>>> All others succeed but ARM this doesn’t look too good. >>>>> Stuck apparently. >>>>> >>>>> https://jenkins.fd.io/job/vpp-arm-verify-master-ubuntu1604/2157/console >>>>> 20:16:04 >>>>> ============================================================================ >>>>> == >>>>> >>>>> 20:16:04 >>>>> ARP Test Case >>>>> >>>>> 20:16:04 >>>>> ============================================================================ >>>>> == >>>>> >>>>> 20:16:07 >>>>> *** Error in `python': free(): invalid pointer: 0x0000ffff553263a8 *** >>>>> >>>>> 20:16:09 >>>>> ARP >>>>> OK >>>>> >>>>> 20:16:09 >>>>> ARP Duplicates >>>>> OK >>>>> >>>>> 20:16:09 >>>>> test_arp_incomplete (test_neighbor.ARPTestCase) >>>>> OK >>>>> >>>>> 20:16:09 >>>>> ARP Static >>>>> OK >>>>> >>>>> 20:16:15 >>>>> ARP reply with VRRP virtual src hw addr >>>>> OK >>>>> >>>>> 20:16:15 >>>>> GARP >>>>> OK >>>>> >>>>> 20:16:15 >>>>> MPLS >>>>> OK >>>>> >>>>> >>>>> Cheers, >>>>> Ole >>>>> >>>>> >>>>>> On 7 Sep 2018, at 18:50, Ole Troan <otr...@employees.org> wrote: >>>>>> >>>>>> Ed, >>>>>> >>>>>> Let me take a closer look at these. >>>>>> It appears if VPP is slow to start it might not have created the socket >>>>>> yet. Let me try to put in a retry loop and see if that fixes verify. >>>>>> >>>>>> Cheers >>>>>> Ole >>>>>> >>>>>> On 7 Sep 2018, at 18:06, Ed Kern via Lists.Fd.Io < >>>>>> ejk=cisco....@lists.fd.io> wrote: >>>>>> >>>>>>> make test failures due to the below causing pretty consistent failures. >>>>>>> Note: For whatever reason the failures are not 100%. >>>>>>> The failures and with the retries on nonconcurrent merge jobs may lead >>>>>>> to long build queues. >>>>>>> >>>>>>> These are not infra issues but ill be keeping an eye on it. >>>>>>> >>>>>>> >>>>>>> Ed >>>>>>> >>>>>>> >>>>>>> >>>>>>> 13:08:25 >>>>>>> Using /var/cache/vpp/python/virtualenv/lib/python2.7/site-packages >>>>>>> >>>>>>> 13:08:25 >>>>>>> Finished processing dependencies for vpp-papi==1.6.1 >>>>>>> >>>>>>> 13:08:27 >>>>>>> Traceback (most recent call last): >>>>>>> >>>>>>> 13:08:27 >>>>>>> File "sanity_run_vpp.py", line 21, in <module> >>>>>>> >>>>>>> 13:08:27 >>>>>>> tc.setUpClass() >>>>>>> >>>>>>> 13:08:27 >>>>>>> File "/w/workspace/vpp-merge-master-ubuntu1604/test/framework.py", line >>>>>>> 394, in setUpClass >>>>>>> >>>>>>> 13:08:27 >>>>>>> cls.statistics = VPPStats(socketname=cls.tempdir+'/stats.sock') >>>>>>> >>>>>>> 13:08:27 >>>>>>> File "build/bdist.linux-x86_64/egg/vpp_papi/vpp_stats.py", line 117, in >>>>>>> __init__ >>>>>>> >>>>>>> 13:08:27 >>>>>>> IOError >>>>>>> >>>>>>> 13:08:27 >>>>>>> ******************************************************************* >>>>>>> >>>>>>> 13:08:27 >>>>>>> * Sanity check failed, cannot run vpp >>>>>>> >>>>>>> 13:08:27 >>>>>>> ******************************************************************* >>>>>>> >>>>>>> -=-=-=-=-=-=-=-=-=-=-=- >>>>>>> Links: You receive all messages sent to this group. >>>>>>> >>>>>>> View/Reply Online (#10432): https://lists.fd.io/g/vpp-dev/message/10432 >>>>>>> Mute This Topic: https://lists.fd.io/mt/25308161/675193 >>>>>>> Group Owner: vpp-dev+ow...@lists.fd.io >>>>>>> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsubb >>>>>>> <https://lists.fd.io/g/vpp-dev/unsubb> [otr...@employees.org >>>>>>> <mailto:otr...@employees.org> >>>>>>> ] >>>>>>> -=-=-=-=-=-=-=-=-=-=-=- >>>>>> -=-=-=-=-=-=-=-=-=-=-=- >>>>>> Links: You receive all messages sent to this group. >>>>>> >>>>>> View/Reply Online (#10435): https://lists.fd.io/g/vpp-dev/message/10435 >>>>>> <https://lists.fd.io/g/vpp-dev/message/10435> >>>>>> Mute This Topic: https://lists.fd.io/mt/25308161/675193 >>>>>> <https://lists.fd.io/mt/25308161/675193> >>>>>> Group Owner: vpp-dev+ow...@lists.fd.io <mailto:vpp-dev+ow...@lists.fd.io> >>>>>> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsubb >>>>>> <https://lists.fd.io/g/vpp-dev/unsubb> [otr...@employees.org >>>>>> <mailto:otr...@employees.org>] >>>>>> -=-=-=-=-=-=-=-=-=-=-=- >>>>> >>>>> -=-=-=-=-=-=-=-=-=-=-=- >>>>> Links: You receive all messages sent to this group. >>>>> >>>>> View/Reply Online (#10437): https://lists.fd.io/g/vpp-dev/message/10437 >>>>> <https://lists.fd.io/g/vpp-dev/message/10437> >>>>> Mute This Topic: https://lists.fd.io/mt/25308161/675152 >>>>> <https://lists.fd.io/mt/25308161/675152> >>>>> Group Owner: vpp-dev+ow...@lists.fd.io <mailto:vpp-dev+ow...@lists.fd.io> >>>>> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsubb >>>>> <https://lists.fd.io/g/vpp-dev/unsubb> [fcoras.li...@gmail.com >>>>> <mailto:fcoras.li...@gmail.com>] >>>>> -=-=-=-=-=-=-=-=-=-=-=- >>>> >>>> -=-=-=-=-=-=-=-=-=-=-=- >>>> Links: You receive all messages sent to this group. >>>> >>>> View/Reply Online (#10438): https://lists.fd.io/g/vpp-dev/message/10438 >>>> <https://lists.fd.io/g/vpp-dev/message/10438> >>>> Mute This Topic: https://lists.fd.io/mt/25308161/675056 >>>> <https://lists.fd.io/mt/25308161/675056> >>>> Group Owner: vpp-dev+ow...@lists.fd.io <mailto:vpp-dev+ow...@lists.fd.io> >>>> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsubb >>>> <https://lists.fd.io/g/vpp-dev/unsubb> [mvarl...@suse.de >>>> <mailto:mvarl...@suse.de>] >>>> -=-=-=-=-=-=-=-=-=-=-=- >>>> >>> -- >>> Marco V >>> >>> SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton >>> HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg >>> >>> -=-=-=-=-=-=-=-=-=-=-=- >>> Links: You receive all messages sent to this group. >>> >>> View/Reply Online (#10453): https://lists.fd.io/g/vpp-dev/message/10453 >>> <https://lists.fd.io/g/vpp-dev/message/10453> >>> Mute This Topic: https://lists.fd.io/mt/25308161/675649 >>> <https://lists.fd.io/mt/25308161/675649> >>> Group Owner: vpp-dev+ow...@lists.fd.io <mailto:vpp-dev+ow...@lists.fd.io> >>> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub >>> <https://lists.fd.io/g/vpp-dev/unsub> [e...@cisco.com >>> <mailto:e...@cisco.com>] >>> -=-=-=-=-=-=-=-=-=-=-=- >> > -- > Marco V > > SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton > HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10462): https://lists.fd.io/g/vpp-dev/message/10462 Mute This Topic: https://lists.fd.io/mt/25308161/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
