our very first ubuntu-based PRB is running: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131701/
crossing my fingers! :) On Tue, Nov 24, 2020 at 1:30 PM shane knapp ☠ <skn...@berkeley.edu> wrote: > due to scheduling, upcoming holiday and in-the-colo work requirements, all > of the centos workers are being wiped NOW. > > this is great, as the sooner we can get started on fixing builds the > better. i'm not going anywhere over the holiday, so i'll get a good > head-start on things. > > thank you jon! > > shane > > On Tue, Nov 24, 2020 at 11:24 AM shane knapp ☠ <skn...@berkeley.edu> > wrote: > >> this is a lengthy, but important read for everyone here. >> >> in the next few days, the remaining centos machines (PRB/SBT workers AND >> primary) will have be reimaged from centos6.9 to ubuntu 20.04LTS. >> >> this means three important things on the very near horizon: >> 1 -- the PRB and SBT tests WILL BE BROKEN (by thanksgiving) >> 2 -- jenkins itself will be down for a while as we move the jenkins >> installation to it's new home. >> 3 -- those of you with accounts here will temporarily lose access >> >> regarding (1), brian (cced) will be helping me debug and fix any >> system-level bugs (python envs, missing packages, etc). jon (cced) will be >> doing the reimaging and cobbling together of hardware to keep us on our >> feet. their help is going to be invaluable to getting us back on the >> ground. >> >> we already have two ubuntu 20 workers up and building >> (research-jenkins-worker-0[1,2]), and the SparkPullRequestBuilder-K8s build >> is already green. i'll keep an eye on these workers to ensure i didn't >> miss anything. >> >> once we have a couple of more ubuntu 20 machines up, i'll move the PRB >> and SBT builds there and let them fail as often as possible so we can use >> the build logs during the migration of the primary. >> >> then we shut down jenkins and move to the new primary. >> >> this will all be happening in the next week to week-and-a-half. >> >> nearish on the horizon, we need to do two things: >> 1 -- reimage the ubuntu 16 workers >> 2 -- clean up the all of the breakages within jenkins plugin universe. >> there's a lot of stacktraces everywhere after the upgrade, but things are >> still building so i'm inclined to push this out. >> 3 -- fix the PRB/SBT builds. >> >> further off, once we're stable, we (the spark community) will need to >> have an honest conversation about where the build system lives. we don't >> currently have enough resources here to manage the system in a way that it >> deserves, and i can't forsee getting the staffing for long-term support any >> time soon. >> >> however, with the ansible configs (which i plan on moving to the spark >> repo), it should be much easier to replicate the build system. >> >> by this time next year, i would like to have helped find the build system >> a new home, and sunset jenkins. over the past 11 years (i think), this >> system has built spark. it's getting a little tired and needs a well >> deserved break. :) >> >> shane >> -- >> Shane Knapp >> Computer Guy / Voice of Reason >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> > > > -- > Shane Knapp > Computer Guy / Voice of Reason > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp Computer Guy / Voice of Reason UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
