Re: [build system] half of the jenkins workers are down

2021-08-09 Thread Xiao Li
Thank you, Shane! Xiao shane knapp ☠ 于2021年8月9日周一 下午1:26写道: > turns out that minikube/k8s and friends were being oom-killed and this was > causing all sorts of weirdnesses. > > i've upped the ram limits on all of the k8s jobs to 8G (from 6G), and > we'll keep an eye on things and see how they

Re: [build system] half of the jenkins workers are down

2021-08-09 Thread shane knapp ☠
turns out that minikube/k8s and friends were being oom-killed and this was causing all sorts of weirdnesses. i've upped the ram limits on all of the k8s jobs to 8G (from 6G), and we'll keep an eye on things and see how they go. On Mon, Aug 9, 2021 at 12:02 PM shane knapp ☠ wrote: > as workers

Re: [build system] half of the jenkins workers are down

2021-08-09 Thread shane knapp ☠
as workers are continuing to fail, i've stopped jenkins from accepting new builds for the time being. more updates as they come. On Mon, Aug 9, 2021 at 9:17 AM shane knapp ☠ wrote: > happy monday! > > the server gods did not smile upon us this weekend, and 4 of the workers > are down. we'll

[build system] half of the jenkins workers are down

2021-08-09 Thread shane knapp ☠
happy monday! the server gods did not smile upon us this weekend, and 4 of the workers are down. we'll most likely need to head to our colo some time today and give them an in-person kick and see what's going on. i'll send an update when they're back up. shane -- Shane Knapp Computer Guy /