I also boosted the number of cloud slaves at a time, though I need to work on that further. =)
A. On Sunday, July 5, 2015, Hervé BOUTEMY <[email protected]> wrote: > great work! > the build queue seems smaller now > > thank you Andrew > > Le samedi 4 juillet 2015 09:30:49 Andrew Bayer a écrit : > > Hi all - > > > > Finally! builds.apache.org is all the way back up and running. It's > feeling > > at least a bit peppier than it did before, and startup time was > > significantly faster. Sorry again for the inconvenience of the extended > > downtime - getting off that problematic ZFS volume that made the backup > so > > slow is exactly why we wanted to do this! I'd expect to see a little > > weirdness with most recent builds in some jobs, but nothing that > shouldn't > > clear up after a new build runs. If you see anything more strange, please > > let me know and I'll investigate. Thanks again! > > > > A. > > > > On Fri, Jul 3, 2015 at 6:29 PM, Andrew Bayer <[email protected] > <javascript:;>> wrote: > > > Hi all - > > > > > > Regrettably, the restore from backup is taking dramatically longer than > > > expected - we're now not even halfway through, so I don't expect to be > > > back > > > online until Saturday morning UTC. > > > > > > A. > > > > > > On Fri, Jul 3, 2015 at 9:35 AM, Andrew Bayer <[email protected] > <javascript:;>> > > > > > > wrote: > > >> Hi all - > > >> > > >> Sorry for the delay in updating - we were finally able to get the OS > > >> installed and booting properly, and are now in the process of > restoring > > >> Jenkins from backup. This'll take a few hours at least - sorry for the > > >> inconvenience, but we should be back up this evening Europe time at > the > > >> outside. > > >> > > >> A. > > >> > > >> On Thu, Jul 2, 2015 at 5:17 PM, Andrew Bayer <[email protected] > <javascript:;>> > > >> > > >> wrote: > > >>> Hi all - > > >>> > > >>> builds.apache.org is still down - we're having problems rebuilding > it. > > >>> We don't yet have a timeframe for when it'll be back up, but once > it's > > >>> been > > >>> reinstalled, I'll send another update. Thanks. > > >>> > > >>> A. > > >>> > > >>> > > >>> On Thu, Jul 2, 2015 at 1:28 PM, Andrew Bayer <[email protected] > <javascript:;>> > > >>> > > >>> wrote: > > >>>> Hooray - a correction. =) I've been able to get all the projects to > > >>>> backup in a reasonable timeframe, so we should hopefully not be > missing > > >>>> builds. Sorry for the false alarm! > > >>>> > > >>>> A. > > >>>> > > >>>> On Thu, Jul 2, 2015 at 12:37 PM, Andrew Bayer < > [email protected] <javascript:;>> > > >>>> > > >>>> wrote: > > >>>>> builds.apache.org has been down for the last two hours as we work > on > > >>>>> finalizing the backup - due to the absolutely atrocious performance > > >>>>> we've > > >>>>> seen on this box with ZFS on Linux and directory traversal, we may > not > > >>>>> be > > >>>>> able to back up Maven project type builds from around 5pm UTC > > >>>>> yesterday > > >>>>> until Jenkins was brought offline at around 8:30am UTC today. To > get > > >>>>> all of > > >>>>> those backed up, it might take another 4 hours of rsyncing. The > same > > >>>>> build > > >>>>> numbers won't be repeated - just that any builds run in that time > may > > >>>>> not > > >>>>> be preserved. This will only be the case for Maven project type > > >>>>> builds, not > > >>>>> freestyle jobs, and all previous builds and job configuration etc > will > > >>>>> still be there regardless. More updates when we're further along. > > >>>>> > > >>>>> A. > > >>>>> > > >>>>> On Tue, Jun 30, 2015 at 11:52 AM, Andrew Bayer <[email protected] > <javascript:;>> > > >>>>> > > >>>>> wrote: > > >>>>>> A reminder - builds.apache.org will be down on July 2nd, starting > > >>>>>> around 9am UTC/11am CEST/10am BST/5am EDT/2am PDT. We'll be taking > > >>>>>> down > > >>>>>> Jenkins, backing it up, rebuilding the machine, and restoring > > >>>>>> Jenkins. > > >>>>>> Hopefully this will be done within 5-6 hours, but could take > longer. > > >>>>>> We'll > > >>>>>> send updates on the status as we go. Thanks. > > >>>>>> > > >>>>>> If you have any questions please contact > [email protected] <javascript:;>. > > >>>>>> > > >>>>>> A. > >
