after a couple hours of testing on the sandbox and getting good builds I went with option 1 for now. (Should be getting merged as I type).
If we run into any more issues Ill remove the job from the main gerrit trigger so builds will only be started via comment. Ed On Sep 10, 2018, at 12:48 PM, Florin Coras <fcoras.li...@gmail.com<mailto:fcoras.li...@gmail.com>> wrote: If 1 makes the arm jobs pass and supposedly run much faster, I’d be fine with that as well. Florin On Sep 10, 2018, at 7:48 AM, Marco Varlese <mvarl...@suse.de<mailto:mvarl...@suse.de>> wrote: On Mon, 2018-09-10 at 14:29 +0000, Ed Kern (ejk) wrote: At least three different possible actions to take at this point: (outside of fixing the issue) 1. remove make test attempt from arm build (return it to the way it was before a week ago). 2. lower timeout further my first thought would be in the 75 minute range (from 120) I don't like this option mainly because it would still imply that developers will have to wait for 75 minutes to see a patch verified... The worst is that the job is a non-voting one hence having it running does add any meaningful insight into the build result. I'm in favour for the issue being resolved in the sandbox and eventually moved to production when stable. 3. remove job altogether. I believe this should be the option to purse unless it could be fixed quickly. As it is it only causes delays to the overall build / review / merging process... Im happy to push a patch that accomplishes the above or other options I haven’t thought of for this mail. Just let me know.. Ed On Sep 10, 2018, at 4:31 AM, Marco Varlese <mvarl...@suse.de<mailto:mvarl...@suse.de>> wrote: Last example (today) can be found here 10:47:06 Not running extended tests (some tests will be skipped) 10:47:06 Perform 4 attempts to pass the suite... 10:47:10 *** Error in `python': double free or corruption (out): 0x0000ffff75d483f0 *** 12:27:54 Build timed out (after 120 minutes). Marking the build as failed. Patch: https://gerrit.fd.io/r/#/c/14744/ Other jobs have finished more than one and half-hour ago and the patch cannot be marked Verified+1 because Jenkins is still waiting for the ARM job to complete (timeout = 120 minutes). IMHO it makes the overall patch submission and review very painful for authors and commiters. I would recommend (it has been done in the past for other jobs) to disable this job in production, move it again to sandbox, get it fixed and eventually moved again to production... - Marco On Fri, 2018-09-07 at 12:04 -0700, Florin Coras wrote: ARM jobs have not been working for some days now. That’s why their result is skipped. Timeout is 2h but probably we should drop that even further … Florin On Sep 7, 2018, at 11:59 AM, Ole Troan <otr...@employees.org<mailto:otr...@employees.org>> wrote: Trying out a change in: https://gerrit.fd.io/r/#/c/14732/ All others succeed but ARM this doesn’t look too good. Stuck apparently. https://jenkins.fd.io/job/vpp-arm-verify-master-ubuntu1604/2157/console 20:16:04 ============================================================================ == 20:16:04 ARP Test Case 20:16:04 ============================================================================ == 20:16:07 *** Error in `python': free(): invalid pointer: 0x0000ffff553263a8 *** 20:16:09 ARP OK 20:16:09 ARP Duplicates OK 20:16:09 test_arp_incomplete (test_neighbor.ARPTestCase) OK 20:16:09 ARP Static OK 20:16:15 ARP reply with VRRP virtual src hw addr OK 20:16:15 GARP OK 20:16:15 MPLS OK Cheers, Ole On 7 Sep 2018, at 18:50, Ole Troan <otr...@employees.org<mailto:otr...@employees.org>> wrote: Ed, Let me take a closer look at these. It appears if VPP is slow to start it might not have created the socket yet. Let me try to put in a retry loop and see if that fixes verify. Cheers Ole On 7 Sep 2018, at 18:06, Ed Kern via Lists.Fd.Io < ejk=cisco....@lists.fd.io<mailto:ejk=cisco....@lists.fd.io>> wrote: make test failures due to the below causing pretty consistent failures. Note: For whatever reason the failures are not 100%. The failures and with the retries on nonconcurrent merge jobs may lead to long build queues. These are not infra issues but ill be keeping an eye on it. Ed 13:08:25 Using /var/cache/vpp/python/virtualenv/lib/python2.7/site-packages 13:08:25 Finished processing dependencies for vpp-papi==1.6.1 13:08:27 Traceback (most recent call last): 13:08:27 File "sanity_run_vpp.py", line 21, in <module> 13:08:27 tc.setUpClass() 13:08:27 File "/w/workspace/vpp-merge-master-ubuntu1604/test/framework.py", line 394, in setUpClass 13:08:27 cls.statistics = VPPStats(socketname=cls.tempdir+'/stats.sock') 13:08:27 File "build/bdist.linux-x86_64/egg/vpp_papi/vpp_stats.py", line 117, in __init__ 13:08:27 IOError 13:08:27 ******************************************************************* 13:08:27 * Sanity check failed, cannot run vpp 13:08:27 ******************************************************************* -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10432): https://lists.fd.io/g/vpp-dev/message/10432 Mute This Topic: https://lists.fd.io/mt/25308161/675193 Group Owner: vpp-dev+ow...@lists.fd.io<mailto:vpp-dev+ow...@lists.fd.io> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsubb [otr...@employees.org<mailto:otr...@employees.org> ] -=-=-=-=-=-=-=-=-=-=-=- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10435): https://lists.fd.io/g/vpp-dev/message/10435 Mute This Topic: https://lists.fd.io/mt/25308161/675193 Group Owner: vpp-dev+ow...@lists.fd.io<mailto:vpp-dev+ow...@lists.fd.io> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsubb [otr...@employees.org<mailto:otr...@employees.org>] -=-=-=-=-=-=-=-=-=-=-=- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10437): https://lists.fd.io/g/vpp-dev/message/10437 Mute This Topic: https://lists.fd.io/mt/25308161/675152 Group Owner: vpp-dev+ow...@lists.fd.io<mailto:vpp-dev+ow...@lists.fd.io> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsubb [fcoras.li...@gmail.com<mailto:fcoras.li...@gmail.com>] -=-=-=-=-=-=-=-=-=-=-=- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10438): https://lists.fd.io/g/vpp-dev/message/10438 Mute This Topic: https://lists.fd.io/mt/25308161/675056 Group Owner: vpp-dev+ow...@lists.fd.io<mailto:vpp-dev+ow...@lists.fd.io> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsubb [mvarl...@suse.de<mailto:mvarl...@suse.de>] -=-=-=-=-=-=-=-=-=-=-=- -- Marco V SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10453): https://lists.fd.io/g/vpp-dev/message/10453 Mute This Topic: https://lists.fd.io/mt/25308161/675649 Group Owner: vpp-dev+ow...@lists.fd.io<mailto:vpp-dev+ow...@lists.fd.io> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [e...@cisco.com<mailto:e...@cisco.com>] -=-=-=-=-=-=-=-=-=-=-=- -- Marco V SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10463): https://lists.fd.io/g/vpp-dev/message/10463 Mute This Topic: https://lists.fd.io/mt/25308161/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
