Re: JENKINS: downtime next week, wed and thurs mornings (9-23 and 9-24)

[shane knapp]

this is happening now.

On Tue, Sep 22, 2015 at 10:07 AM, shane knapp  wrote:
> ok, here's the updated downtime schedule for this week:
>
> wednesday, sept 23rd:
>
> firewall maintenance cancelled, as jon took care of the update
> saturday morning while we were bringing jenkins back up after the colo
> fire
>
> thursday, sept 24th:
>
> jenkins maintenance is still scheduled, but abbreviated as some of the
> maintenance was performed saturday morning as well
> * new builds will stop being accepted ~630am PDT
>   - i'll kill any hangers-on at 730am, and after maintenance is done,
> i will retrigger any killed jobs
> * jenkins worker system package updates
>   - amp-jenkins-master was completed on saturday
>   - this will NOT include kernel updates as moving to
> 2.6.32-573.3.1.el6 bricked amp-jenkins-master
> * moving default system java for builds from jdk1.7.0_71 to jdk1.7.0_79
> * all systems get a reboot
> * expected downtime:  3.5 hours or so
>
> i'll post updates as i progress.
>
> also, i'll post a copy of our post-mortem once the dust settles.  it's
> been, shall we say, a pretty crazy few days.
>
> http://news.berkeley.edu/2015/09/19/campus-network-outage/
>
> :)
>
> On Mon, Sep 21, 2015 at 10:11 AM, shane knapp  wrote:
>> quick update:  we actually did some of the maintenance on our systems
>> after the berkeley-wide outage caused by one of our (non-jenkins)
>> servers halting and catching fire.
>>
>> we'll still have some downtime early wednesday, but tomorrow's will be
>> cancelled.  i'll send out another update real soon now with what we'll
>> be covering on wednesday once we get our current situation more under
>> control.  :)
>>
>> On Wed, Sep 16, 2015 at 12:15 PM, shane knapp  wrote:
 630am-10am thursday, 9-24-15:
 * jenknins update to 1.629 (we're a few months behind in versions, and
 some big bugs have been fixed)
 * jenkins master and worker system package updates
 * all systems get a reboot (lots of hanging java processes have been
 building up over the months)
 * builds will stop being accepted ~630am, and i'll kill any hangers-on
 at 730am, and retrigger once we're done
 * expected downtime:  3.5 hours or so
 * i will also be testing out some of my shiny new ansible playbooks
 for the system updates!

>>> i forgot one thing:
>>>
>>> * moving default system java for builds from jdk1.7.0_71 to jdk1.7.0_79

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: JENKINS: downtime next week, wed and thurs mornings (9-23 and 9-24)

[shane knapp]

...and we're finished and now building!

On Thu, Sep 24, 2015 at 7:19 AM, shane knapp  wrote:
> this is happening now.
>
> On Tue, Sep 22, 2015 at 10:07 AM, shane knapp  wrote:
>> ok, here's the updated downtime schedule for this week:
>>
>> wednesday, sept 23rd:
>>
>> firewall maintenance cancelled, as jon took care of the update
>> saturday morning while we were bringing jenkins back up after the colo
>> fire
>>
>> thursday, sept 24th:
>>
>> jenkins maintenance is still scheduled, but abbreviated as some of the
>> maintenance was performed saturday morning as well
>> * new builds will stop being accepted ~630am PDT
>>   - i'll kill any hangers-on at 730am, and after maintenance is done,
>> i will retrigger any killed jobs
>> * jenkins worker system package updates
>>   - amp-jenkins-master was completed on saturday
>>   - this will NOT include kernel updates as moving to
>> 2.6.32-573.3.1.el6 bricked amp-jenkins-master
>> * moving default system java for builds from jdk1.7.0_71 to jdk1.7.0_79
>> * all systems get a reboot
>> * expected downtime:  3.5 hours or so
>>
>> i'll post updates as i progress.
>>
>> also, i'll post a copy of our post-mortem once the dust settles.  it's
>> been, shall we say, a pretty crazy few days.
>>
>> http://news.berkeley.edu/2015/09/19/campus-network-outage/
>>
>> :)
>>
>> On Mon, Sep 21, 2015 at 10:11 AM, shane knapp  wrote:
>>> quick update:  we actually did some of the maintenance on our systems
>>> after the berkeley-wide outage caused by one of our (non-jenkins)
>>> servers halting and catching fire.
>>>
>>> we'll still have some downtime early wednesday, but tomorrow's will be
>>> cancelled.  i'll send out another update real soon now with what we'll
>>> be covering on wednesday once we get our current situation more under
>>> control.  :)
>>>
>>> On Wed, Sep 16, 2015 at 12:15 PM, shane knapp  wrote:
> 630am-10am thursday, 9-24-15:
> * jenknins update to 1.629 (we're a few months behind in versions, and
> some big bugs have been fixed)
> * jenkins master and worker system package updates
> * all systems get a reboot (lots of hanging java processes have been
> building up over the months)
> * builds will stop being accepted ~630am, and i'll kill any hangers-on
> at 730am, and retrigger once we're done
> * expected downtime:  3.5 hours or so
> * i will also be testing out some of my shiny new ansible playbooks
> for the system updates!
>
 i forgot one thing:

 * moving default system java for builds from jdk1.7.0_71 to jdk1.7.0_79

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: JENKINS: downtime next week, wed and thurs mornings (9-23 and 9-24)

[shane knapp]

ok, here's the updated downtime schedule for this week:

wednesday, sept 23rd:

firewall maintenance cancelled, as jon took care of the update
saturday morning while we were bringing jenkins back up after the colo
fire

thursday, sept 24th:

jenkins maintenance is still scheduled, but abbreviated as some of the
maintenance was performed saturday morning as well
* new builds will stop being accepted ~630am PDT
  - i'll kill any hangers-on at 730am, and after maintenance is done,
i will retrigger any killed jobs
* jenkins worker system package updates
  - amp-jenkins-master was completed on saturday
  - this will NOT include kernel updates as moving to
2.6.32-573.3.1.el6 bricked amp-jenkins-master
* moving default system java for builds from jdk1.7.0_71 to jdk1.7.0_79
* all systems get a reboot
* expected downtime:  3.5 hours or so

i'll post updates as i progress.

also, i'll post a copy of our post-mortem once the dust settles.  it's
been, shall we say, a pretty crazy few days.

http://news.berkeley.edu/2015/09/19/campus-network-outage/

:)

On Mon, Sep 21, 2015 at 10:11 AM, shane knapp  wrote:
> quick update:  we actually did some of the maintenance on our systems
> after the berkeley-wide outage caused by one of our (non-jenkins)
> servers halting and catching fire.
>
> we'll still have some downtime early wednesday, but tomorrow's will be
> cancelled.  i'll send out another update real soon now with what we'll
> be covering on wednesday once we get our current situation more under
> control.  :)
>
> On Wed, Sep 16, 2015 at 12:15 PM, shane knapp  wrote:
>>> 630am-10am thursday, 9-24-15:
>>> * jenknins update to 1.629 (we're a few months behind in versions, and
>>> some big bugs have been fixed)
>>> * jenkins master and worker system package updates
>>> * all systems get a reboot (lots of hanging java processes have been
>>> building up over the months)
>>> * builds will stop being accepted ~630am, and i'll kill any hangers-on
>>> at 730am, and retrigger once we're done
>>> * expected downtime:  3.5 hours or so
>>> * i will also be testing out some of my shiny new ansible playbooks
>>> for the system updates!
>>>
>> i forgot one thing:
>>
>> * moving default system java for builds from jdk1.7.0_71 to jdk1.7.0_79

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: JENKINS: downtime next week, wed and thurs mornings (9-23 and 9-24)

[shane knapp]

quick update:  we actually did some of the maintenance on our systems
after the berkeley-wide outage caused by one of our (non-jenkins)
servers halting and catching fire.

we'll still have some downtime early wednesday, but tomorrow's will be
cancelled.  i'll send out another update real soon now with what we'll
be covering on wednesday once we get our current situation more under
control.  :)

On Wed, Sep 16, 2015 at 12:15 PM, shane knapp  wrote:
>> 630am-10am thursday, 9-24-15:
>> * jenknins update to 1.629 (we're a few months behind in versions, and
>> some big bugs have been fixed)
>> * jenkins master and worker system package updates
>> * all systems get a reboot (lots of hanging java processes have been
>> building up over the months)
>> * builds will stop being accepted ~630am, and i'll kill any hangers-on
>> at 730am, and retrigger once we're done
>> * expected downtime:  3.5 hours or so
>> * i will also be testing out some of my shiny new ansible playbooks
>> for the system updates!
>>
> i forgot one thing:
>
> * moving default system java for builds from jdk1.7.0_71 to jdk1.7.0_79

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: JENKINS: downtime next week, wed and thurs mornings (9-23 and 9-24)

[Reynold Xin]

Thanks Shane and Jon for the heads up.

On Wednesday, September 16, 2015, shane knapp  wrote:

> good morning, denizens of the aether!
>
> your hard working build system (and some associated infrastructure)
> has been in need of some updates and housecleaning for quite a while
> now.  we will be splitting the maintenance over two mornings to
> minimize impact.
>
> here's the plan:
>
> 7am-9am wednesday, 9-24-15  (or 24-9-15 for those not in amurrica):
> * firewall taken offline for system and firewall updates
> * expected downtime:  maybe an hour, but we'll say two just in case
> * this will be done by jkuroda (CCed on this message)
>
> 630am-10am thursday, 9-24-15:
> * jenknins update to 1.629 (we're a few months behind in versions, and
> some big bugs have been fixed)
> * jenkins master and worker system package updates
> * all systems get a reboot (lots of hanging java processes have been
> building up over the months)
> * builds will stop being accepted ~630am, and i'll kill any hangers-on
> at 730am, and retrigger once we're done
> * expected downtime:  3.5 hours or so
> * i will also be testing out some of my shiny new ansible playbooks
> for the system updates!
>
>
> please let me know if you have any questions, or requests to postpone
> this maintenance.  thanks in advance!
>
> shane & jon
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org 
> For additional commands, e-mail: dev-h...@spark.apache.org 
>
>

Re: JENKINS: downtime next week, wed and thurs mornings (9-23 and 9-24)

[shane knapp]

> 630am-10am thursday, 9-24-15:
> * jenknins update to 1.629 (we're a few months behind in versions, and
> some big bugs have been fixed)
> * jenkins master and worker system package updates
> * all systems get a reboot (lots of hanging java processes have been
> building up over the months)
> * builds will stop being accepted ~630am, and i'll kill any hangers-on
> at 730am, and retrigger once we're done
> * expected downtime:  3.5 hours or so
> * i will also be testing out some of my shiny new ansible playbooks
> for the system updates!
>
i forgot one thing:

* moving default system java for builds from jdk1.7.0_71 to jdk1.7.0_79

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org