On Sat, Nov 5, 2016 at 3:32 PM, Michael Scherer <msche...@redhat.com> wrote:
> Hi, > > so people might have seen that last week, a rather severe vuln was > found: https://dirtycow.ninja/ > > I was at Openstack summit when it was found, and the updated kernel > package wasn't on the CDN until I was out for holiday[1]. The main > reason is that RH test kernel patchs a bit more than others, especially > for something as critical. And Centos wait on RH to push update > > So while this was not uber urgent as shellshock or heartbleed, it was > still rather critical to fix as I have a rather minimal trust in Jenkins > and Gerrit to be secure. > > So once I was back on friday, and after dealing with others fires and > infra, I did reboot stuff that wouldn't impact too much production (like > rsyslog, freeipa servers, the salt server, the virt hosts with builders) > and decided to push for a reboot of jenkins and gerrit for the weekend. > > In retrospect, I tought I did discuss on irc, but I forgot, sorry about > that. > > Of course, because I like to live dangerously, I did that in the > saturday morning, on a travel day. It should have been fast[2]. > > However, things never go as expected and we did face a few issues: > > - myrmicinae.rht.gluster.org, the host running our VM decided to take 1h > to boot. At the firmware/BIOS level. That's slightly inacceptable, but I > have also a limited capacity to fix, since this would requires 1) to > test reboot (so lose 1h) 2) to fiddle in the Bios (and so reboot). > > So that's why jenkins/gerrit were down around 10h CET until 11h. > > - jenkins didn't (as usual) restart. I found the root cause, this was > due to NetworkManager and network init script kinda doing the same > stuff, but in different way. This is now fixed, and jenkins VM should > reboot without a human to fix stuff around. > > - gerrit for some reason do not start at boot. I am not sure what was > the way it was done before, but I suspect something related > to /etc/init.d that got wiped after a upgrade or something, because > gerrit initscript is not a real initscript. So I did some hack > in /etc/rc.local, since the upgrade to EL7 is around the corner, and I > had better things to do in the weekend that fixing some bash stuff (like > fixing python stuff). > > - gerrit VM DNS was incorrect, and no one told me until 6h after the > reboot (why no one told on irc and or on the list and or bugzilla is a > issue that I will surely have to investigate). Why did the DNS got > changed (or if it didn't changed, how did it worked before ?) is the > part that I still cannot explain. But it got for some reason reverted to > the old setting, using the libvirt gateway as dns, which wasn't working > with the current setup. So this was fixed after Nigel pinged me on my > phone, and I managed to connect from the train to fix it. > > So I suspect that's all for today, I will try to schedule my next > vacation outside of the unexpected release of a critical kernel patch. > > [1] yes, it was nice, thanks for asking. > > [2] famous last word > -- > Michael Scherer > Sysadmin, Community Infrastructure and Platform, OSAS > > > > _______________________________________________ > Gluster-infra mailing list > gluster-in...@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-infra > Hey, thanks for the update! We'll work on moving those kernel patches timelines around. - amye -- Amye Scavarda | a...@redhat.com | Gluster Community Lead
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel