This has nothing to do with the LF (outside of jobs getting stuck (which I haven’t seen since the upgrade yesterday)).
The queue has been ballooning and shrinking the last couple of days for a few reasons: a. a very large number of 1908 verify and then merge jobs. b. increase recently in the number and in some cases long duration of csit jobs c. merge jobs that have to wait for the previous so they just (correctly) sit in the build queue. d. multi hour shutdown yesterday which always causes a bubble… There is no current way to prioritize master runs over 1908 cherrypicks so for that wait I apologize. Current longest wait time I’ve seen on a verify based job was around 80 minutes to start. Andrew has said that he had a backlog of cherrypicks and he expects the number at any one time to reduce. We have more than doubled the amount of jobs running at any one time over the last three weeks. The system is fine but certainly the amount of ’slack’ is considerably less then there used to be… Currently thought is still to repurpose ‘retiring’ virl hardware thats in place into generic x86 infra build to get us a lot more ’slack’. Ed On Sep 26, 2019, at 6:56 AM, Jan Gelety via Lists.Fd.Io <jgelety=cisco....@lists.fd.io<mailto:jgelety=cisco....@lists.fd.io>> wrote: Hello, Could you, please, check FD.io<http://fd.io/> Jenkins? There’s a quite a huge queue of wating jobs (87 at the moment) there. Thanks, Jan -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14059): https://lists.fd.io/g/vpp-dev/message/14059 Mute This Topic: https://lists.fd.io/mt/34298430/675649 Group Owner: vpp-dev+ow...@lists.fd.io<mailto:vpp-dev+ow...@lists.fd.io> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [e...@cisco.com<mailto:e...@cisco.com>] -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14064): https://lists.fd.io/g/vpp-dev/message/14064 Mute This Topic: https://lists.fd.io/mt/34298430/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
