Hello, the CI is now back up and running. Auto scaling is working as expected and it passed our load tests.
Please excuse the caused inconveniences. Best regards, Marco On Wed, Nov 21, 2018 at 5:24 AM Marco de Abreu <marco.g.ab...@googlemail.com> wrote: > Hello, > > I'd like to let you know that our CI was impaired and down for the last > few hours. After getting the CI back up, I noticed that our auto scaling > broke due to a silent update of Jenkins which broke our upscale-detection. > Manual scaling is currently not possible and stopping the scaling won't > help either because there are currently no p3 instances available, which > means that all jobs will fail none the less. In a few hours, the auto > scaling will have recycled all slaves through the down-scale mechanism and > we will be out of capacity. This will lead to resource starvation and thus > timeouts. > > Your PRs will be properly registered by Jenkins, but please expect the > jobs to time out and thus fail your PRs. > > I will fix the auto scaling as soon as I'm awake again. > > Sorry for the caused inconveniences. > > Best regards, > Marco > > > P.S. Sorry for the brief email and my lack of further fixes, but it's > 5:30AM now and I've been working for 17 hours. >
