Yep, thanks Ed for a detailed comment and my apologies for the spike... When I catch up with the cherrypicks then it will be hopefully better. Still tweaking the process...
--a > On 26 Sep 2019, at 15:57, Ed Kern via Lists.Fd.Io <ejk=cisco....@lists.fd.io> > wrote: > > This has nothing to do with the LF (outside of jobs getting stuck (which I > haven’t seen since the upgrade yesterday)). > > The queue has been ballooning and shrinking the last couple of days for a few > reasons: > a. a very large number of 1908 verify and then merge jobs. > b. increase recently in the number and in some cases long duration of csit > jobs > c. merge jobs that have to wait for the previous so they just (correctly) sit > in the build queue. > d. multi hour shutdown yesterday which always causes a bubble… > > There is no current way to prioritize master runs over 1908 cherrypicks so > for that wait I apologize. Current longest > wait time I’ve seen on a verify based job was around 80 minutes to start. > > Andrew has said that he had a backlog of cherrypicks and he expects the > number at any one time to reduce. > > We have more than doubled the amount of jobs running at any one time over the > last three weeks. The system is fine > but certainly the amount of ’slack’ is considerably less then there used to > be… > > Currently thought is still to repurpose ‘retiring’ virl hardware thats in > place into generic x86 infra build to get us a lot more > ’slack’. > > Ed > > > >> On Sep 26, 2019, at 6:56 AM, Jan Gelety via Lists.Fd.Io >> <jgelety=cisco....@lists.fd.io> wrote: >> >> Hello, >> >> Could you, please, check FD.io Jenkins? There’s a quite a huge queue of >> wating jobs (87 at the moment) there. >> >> Thanks, >> Jan >> -=-=-=-=-=-=-=-=-=-=-=- >> Links: You receive all messages sent to this group. >> >> View/Reply Online (#14059): https://lists.fd.io/g/vpp-dev/message/14059 >> Mute This Topic: https://lists.fd.io/mt/34298430/675649 >> Group Owner: vpp-dev+ow...@lists.fd.io >> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [e...@cisco.com] >> -=-=-=-=-=-=-=-=-=-=-=- > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#14064): https://lists.fd.io/g/vpp-dev/message/14064 > Mute This Topic: https://lists.fd.io/mt/34298430/675608 > Group Owner: vpp-dev+ow...@lists.fd.io > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [ayour...@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14065): https://lists.fd.io/g/vpp-dev/message/14065 Mute This Topic: https://lists.fd.io/mt/34298430/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
