Thanks for your efforts, Marco! On Wed, Nov 21, 2018 at 4:02 PM Anirudh Subramanian <anirudh2...@gmail.com> wrote:
> Thanks for the quick response and mitigation! > > On Wed, Nov 21, 2018 at 3:55 PM Marco de Abreu > <marco.g.ab...@googlemail.com.invalid> wrote: > > > Hello, > > > > today, CI had some issues and I had to cancel all jobs a few minutes ago. > > This was basically caused by the high load that is currently being put on > > our CI system due to the pre-release efforts for this Friday. > > > > It's really unfortunate that we just had outages of three core components > > within the last two days - sorry about that!. To recap, we had the > > following outages (which are unrelated to the parallel refactor of the > > Jenkins pipeline): > > - (yesterday evening) The Jenkins master ran out of disk space and thus > > processed requests at reduced capacity > > - (this morning) The Jenkins master got updated which broke our > > autoscalings upscaling capabilities. > > - (new, this evening) Jenkins API was irresponsive: Due to the high > number > > of jobs and a bad API design in the Jenkins REST API, the time-complexity > > of a simple create or delete request was quadratic which resulted in all > > requests timing out (that was the current outage). This resulted in our > > auto scaling to be unable to interface with the Jenkins master. > > > > I have now made improvements to our REST API calls which reduced the > > complexity from O(N^2) to O(1). The reason was an underlying redirect > loop > > in the Jenkins createNode and deleteNode REST API in combination with > > unrolling the entire slave and job graph (which got quite huge during > > extensive load) upon every single request. Since we had about 150 > > registered slaves and 1000 jobs in the queue, the duration for a single > > REST API call rose to up to 45 seconds (we execute up to a few hundred > > queries per auto scaling loop). This lead to our auto scaling timing out. > > > > Everything should be back to normal now. I'm closely observing the > > situation and I'll let you know if I encounter any additional issues. > > > > Again, sorry for any caused inconveniences. > > > > Best regards, > > Marco > > > > On Wed, Nov 21, 2018 at 5:10 PM Gavin M Bell <gavin.max.b...@gmail.com> > > wrote: > > > > > Yes, let me add to the kudos, very nice work Marco. > > > > > > > > > "I'm trying real hard to be the shepherd." -Jules Winnfield > > > > > > > > > > On Nov 21, 2018, at 5:04 PM, Sunderland, Kellen > > > <kell...@amazon.de.INVALID> wrote: > > > > > > > > Appreciate the big effort in bring the CI back so quickly. Thanks > > Marco. > > > > > > > > On Nov 21, 2018 5:52 AM, Marco de Abreu < > marco.g.ab...@googlemail.com > > .INVALID> > > > wrote: > > > > Thanks Aaron! Just for the record, the new Jenkins jobs were > unrelated > > to > > > > that incident. > > > > > > > > If somebody is interested in the details around the outage: > > > > > > > > Due to a required maintenance (disk running full), we had to upgrade > > our > > > > Jenkins master because it was running on Ubuntu 17.04 (for an unknown > > > > reason, it used to be 16.04) and we needed to install some packages. > > > Since > > > > the support for Ubuntu 17.04 was stopped, this resulted in all > package > > > > updates and installations to fail because the repositories were taken > > > > offline. Due to the unavailable maintenance package and other issues > > with > > > > the installed OpenJDK8 version, we made the decision to upgrade the > > > Jenkins > > > > master to Ubuntu 18.04 LTS in order to get back to a supported > version > > > with > > > > maintenance tools. During this upgrade, Jenkins was automatically > > updated > > > > by APT as part of the dist-upgrade process. > > > > > > > > In the latest version of Jenkins, some labels have been changed which > > we > > > > depend on for our auto scaling. To be more specific: > > > >> Waiting for next available executor on mxnetlinux-gpu > > > > has been changed to > > > >> Waiting for next available executor on ‘mxnetlinux-gpu’ > > > > Notice the quote characters. > > > > > > > > Jenkins does not offer a better way than to parse these messages > > > > unfortunately - there's no standardized way to express queue items. > > Since > > > > our parser expected the above message without quote signs, this > message > > > was > > > > discarded. > > > > > > > > We support various queue reasons (5 of them to be exact) that > indicate > > > > resource starvation. If we run super low on capacity, the queue > reason > > is > > > > different and we would still be able to scale up, but most of the > cases > > > > would have printed the unsupported message. This resulted in reduced > > > > capacity (to be specific, the limit during that time was 1 slave per > > > type). > > > > > > > > We have now fixed our autoscaling to automatically strip these > > characters > > > > and added that message to our test suite. > > > > > > > > Best regards, > > > > Marco > > > > > > > > On Wed, Nov 21, 2018 at 2:49 PM Aaron Markham < > > aaron.s.mark...@gmail.com > > > > > > > > wrote: > > > > > > > >> Marco, thanks for your hard work on this. I'm super excited about > the > > > new > > > >> Jenkins jobs. This is going to be very helpful and improve sanity > for > > > our > > > >> PRs and ourselves! > > > >> > > > >> Cheers, > > > >> Aaron > > > >> > > > >> On Wed, Nov 21, 2018, 05:37 Marco de Abreu > > > >> <marco.g.ab...@googlemail.com.invalid wrote: > > > >> > > > >>> Hello, > > > >>> > > > >>> the CI is now back up and running. Auto scaling is working as > > expected > > > >> and > > > >>> it passed our load tests. > > > >>> > > > >>> Please excuse the caused inconveniences. > > > >>> > > > >>> Best regards, > > > >>> Marco > > > >>> > > > >>> On Wed, Nov 21, 2018 at 5:24 AM Marco de Abreu < > > > >>> marco.g.ab...@googlemail.com> > > > >>> wrote: > > > >>> > > > >>>> Hello, > > > >>>> > > > >>>> I'd like to let you know that our CI was impaired and down for the > > > last > > > >>>> few hours. After getting the CI back up, I noticed that our auto > > > >> scaling > > > >>>> broke due to a silent update of Jenkins which broke our > > > >>> upscale-detection. > > > >>>> Manual scaling is currently not possible and stopping the scaling > > > won't > > > >>>> help either because there are currently no p3 instances available, > > > >> which > > > >>>> means that all jobs will fail none the less. In a few hours, the > > auto > > > >>>> scaling will have recycled all slaves through the down-scale > > mechanism > > > >>> and > > > >>>> we will be out of capacity. This will lead to resource starvation > > and > > > >>> thus > > > >>>> timeouts. > > > >>>> > > > >>>> Your PRs will be properly registered by Jenkins, but please expect > > the > > > >>>> jobs to time out and thus fail your PRs. > > > >>>> > > > >>>> I will fix the auto scaling as soon as I'm awake again. > > > >>>> > > > >>>> Sorry for the caused inconveniences. > > > >>>> > > > >>>> Best regards, > > > >>>> Marco > > > >>>> > > > >>>> > > > >>>> P.S. Sorry for the brief email and my lack of further fixes, but > > it's > > > >>>> 5:30AM now and I've been working for 17 hours. > > > >>>> > > > >>> > > > >> > > > > > >
