it's looking like everything except the pull request builders are working. i'm going to be working on getting this resolved today.
On Fri, Sep 5, 2014 at 8:18 AM, Nicholas Chammas <nicholas.cham...@gmail.com > wrote: > Hmm, looks like at least some builds > <https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/19804/consoleFull> > are working now, though this last one was from ~5 hours ago. > > > On Fri, Sep 5, 2014 at 1:02 AM, shane knapp <skn...@berkeley.edu> wrote: > >> yep. that's exactly the behavior i saw earlier, and will be figuring out >> first thing tomorrow morning. i bet it's an environment issues on the >> slaves. >> >> >> On Thu, Sep 4, 2014 at 7:10 PM, Nicholas Chammas < >> nicholas.cham...@gmail.com> wrote: >> >>> Looks like during the last build >>> <https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/19797/console> >>> Jenkins was unable to execute a git fetch? >>> >>> >>> On Thu, Sep 4, 2014 at 7:58 PM, shane knapp <skn...@berkeley.edu> wrote: >>> >>>> i'm going to restart jenkins and see if that fixes things. >>>> >>>> >>>> On Thu, Sep 4, 2014 at 4:56 PM, shane knapp <skn...@berkeley.edu> >>>> wrote: >>>> >>>>> looking >>>>> >>>>> >>>>> On Thu, Sep 4, 2014 at 4:21 PM, Nicholas Chammas < >>>>> nicholas.cham...@gmail.com> wrote: >>>>> >>>>>> It appears that our main man is having trouble >>>>>> <https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/> >>>>>> hearing new requests >>>>>> <https://github.com/apache/spark/pull/2277#issuecomment-54549106>. >>>>>> >>>>>> Do we need some smelling salts? >>>>>> >>>>>> >>>>>> On Thu, Sep 4, 2014 at 5:49 PM, shane knapp <skn...@berkeley.edu> >>>>>> wrote: >>>>>> >>>>>>> i'd ping the Jenkinsmench... the master was completely offline, so >>>>>>> any new >>>>>>> jobs wouldn't have reached it. any jobs that were queued when power >>>>>>> was >>>>>>> lost probably started up, but jobs that were running would fail. >>>>>>> >>>>>>> >>>>>>> On Thu, Sep 4, 2014 at 2:45 PM, Nicholas Chammas < >>>>>>> nicholas.cham...@gmail.com >>>>>>> > wrote: >>>>>>> >>>>>>> > Woohoo! Thanks Shane. >>>>>>> > >>>>>>> > Do you know if queued PR builds will automatically be picked up? >>>>>>> Or do we >>>>>>> > have to ping the Jenkinmensch manually from each PR? >>>>>>> > >>>>>>> > Nick >>>>>>> > >>>>>>> > >>>>>>> > On Thu, Sep 4, 2014 at 5:37 PM, shane knapp <skn...@berkeley.edu> >>>>>>> wrote: >>>>>>> > >>>>>>> >> AND WE'RE UP! >>>>>>> >> >>>>>>> >> sorry that this took so long... i'll send out a more detailed >>>>>>> explanation >>>>>>> >> of what happened soon. >>>>>>> >> >>>>>>> >> now, off to back up jenkins. >>>>>>> >> >>>>>>> >> shane >>>>>>> >> >>>>>>> >> >>>>>>> >> On Thu, Sep 4, 2014 at 1:27 PM, shane knapp <skn...@berkeley.edu> >>>>>>> wrote: >>>>>>> >> >>>>>>> >> > it's a faulty power switch on the firewall, which has been >>>>>>> swapped out. >>>>>>> >> > we're about to reboot and be good to go. >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > On Thu, Sep 4, 2014 at 1:19 PM, shane knapp < >>>>>>> skn...@berkeley.edu> >>>>>>> >> wrote: >>>>>>> >> > >>>>>>> >> >> looks like some hardware failed, and we're swapping in a >>>>>>> replacement. >>>>>>> >> i >>>>>>> >> >> don't have more specific information yet -- including *what* >>>>>>> failed, >>>>>>> >> as our >>>>>>> >> >> sysadmin is super busy ATM. the root cause was an incorrect >>>>>>> circuit >>>>>>> >> being >>>>>>> >> >> switched off during building maintenance. >>>>>>> >> >> >>>>>>> >> >> on a side note, this incident will be accelerating our plan to >>>>>>> move the >>>>>>> >> >> entire jenkins infrastructure in to a managed datacenter >>>>>>> environment. >>>>>>> >> this >>>>>>> >> >> will be our major push over the next couple of weeks. more >>>>>>> details >>>>>>> >> about >>>>>>> >> >> this, also, as soon as i get them. >>>>>>> >> >> >>>>>>> >> >> i'm very sorry about the downtime, we'll get everything up and >>>>>>> running >>>>>>> >> >> ASAP. >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> On Thu, Sep 4, 2014 at 12:27 PM, shane knapp < >>>>>>> skn...@berkeley.edu> >>>>>>> >> wrote: >>>>>>> >> >> >>>>>>> >> >>> looks like a power outage in soda hall. more updates as they >>>>>>> happen. >>>>>>> >> >>> >>>>>>> >> >>> >>>>>>> >> >>> On Thu, Sep 4, 2014 at 12:25 PM, shane knapp < >>>>>>> skn...@berkeley.edu> >>>>>>> >> >>> wrote: >>>>>>> >> >>> >>>>>>> >> >>>> i am trying to get things up and running, but it looks like >>>>>>> either >>>>>>> >> the >>>>>>> >> >>>> firewall gateway or jenkins server itself is down. i'll >>>>>>> update as >>>>>>> >> soon as >>>>>>> >> >>>> i know more. >>>>>>> >> >>>> >>>>>>> >> >>> >>>>>>> >> >>> >>>>>>> >> >> >>>>>>> >> > >>>>>>> >> >>>>>>> > >>>>>>> > -- >>>>>>> > You received this message because you are subscribed to the Google >>>>>>> Groups >>>>>>> > "amp-infra" group. >>>>>>> > To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an >>>>>>> > email to amp-infra+unsubscr...@googlegroups.com. >>>>>>> > For more options, visit https://groups.google.com/d/optout. >>>>>>> > >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
