How's it going? It looks like during the last build <https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/lastBuild/console> from about 30 min ago Jenkins was still having trouble fetching from GitHub. It also looks like not all requests for testing are triggering builds.
On Fri, Sep 5, 2014 at 1:23 PM, shane knapp <skn...@berkeley.edu> wrote: > it's looking like everything except the pull request builders are working. > i'm going to be working on getting this resolved today. > > > On Fri, Sep 5, 2014 at 8:18 AM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> Hmm, looks like at least some builds >> <https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/19804/consoleFull> >> are working now, though this last one was from ~5 hours ago. >> >> >> On Fri, Sep 5, 2014 at 1:02 AM, shane knapp <skn...@berkeley.edu> wrote: >> >>> yep. that's exactly the behavior i saw earlier, and will be figuring >>> out first thing tomorrow morning. i bet it's an environment issues on the >>> slaves. >>> >>> >>> On Thu, Sep 4, 2014 at 7:10 PM, Nicholas Chammas < >>> nicholas.cham...@gmail.com> wrote: >>> >>>> Looks like during the last build >>>> <https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/19797/console> >>>> Jenkins was unable to execute a git fetch? >>>> >>>> >>>> On Thu, Sep 4, 2014 at 7:58 PM, shane knapp <skn...@berkeley.edu> >>>> wrote: >>>> >>>>> i'm going to restart jenkins and see if that fixes things. >>>>> >>>>> >>>>> On Thu, Sep 4, 2014 at 4:56 PM, shane knapp <skn...@berkeley.edu> >>>>> wrote: >>>>> >>>>>> looking >>>>>> >>>>>> >>>>>> On Thu, Sep 4, 2014 at 4:21 PM, Nicholas Chammas < >>>>>> nicholas.cham...@gmail.com> wrote: >>>>>> >>>>>>> It appears that our main man is having trouble >>>>>>> <https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/> >>>>>>> hearing new requests >>>>>>> <https://github.com/apache/spark/pull/2277#issuecomment-54549106>. >>>>>>> >>>>>>> Do we need some smelling salts? >>>>>>> >>>>>>> >>>>>>> On Thu, Sep 4, 2014 at 5:49 PM, shane knapp <skn...@berkeley.edu> >>>>>>> wrote: >>>>>>> >>>>>>>> i'd ping the Jenkinsmench... the master was completely offline, so >>>>>>>> any new >>>>>>>> jobs wouldn't have reached it. any jobs that were queued when >>>>>>>> power was >>>>>>>> lost probably started up, but jobs that were running would fail. >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Sep 4, 2014 at 2:45 PM, Nicholas Chammas < >>>>>>>> nicholas.cham...@gmail.com >>>>>>>> > wrote: >>>>>>>> >>>>>>>> > Woohoo! Thanks Shane. >>>>>>>> > >>>>>>>> > Do you know if queued PR builds will automatically be picked up? >>>>>>>> Or do we >>>>>>>> > have to ping the Jenkinmensch manually from each PR? >>>>>>>> > >>>>>>>> > Nick >>>>>>>> > >>>>>>>> > >>>>>>>> > On Thu, Sep 4, 2014 at 5:37 PM, shane knapp <skn...@berkeley.edu> >>>>>>>> wrote: >>>>>>>> > >>>>>>>> >> AND WE'RE UP! >>>>>>>> >> >>>>>>>> >> sorry that this took so long... i'll send out a more detailed >>>>>>>> explanation >>>>>>>> >> of what happened soon. >>>>>>>> >> >>>>>>>> >> now, off to back up jenkins. >>>>>>>> >> >>>>>>>> >> shane >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> On Thu, Sep 4, 2014 at 1:27 PM, shane knapp <skn...@berkeley.edu> >>>>>>>> wrote: >>>>>>>> >> >>>>>>>> >> > it's a faulty power switch on the firewall, which has been >>>>>>>> swapped out. >>>>>>>> >> > we're about to reboot and be good to go. >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > On Thu, Sep 4, 2014 at 1:19 PM, shane knapp < >>>>>>>> skn...@berkeley.edu> >>>>>>>> >> wrote: >>>>>>>> >> > >>>>>>>> >> >> looks like some hardware failed, and we're swapping in a >>>>>>>> replacement. >>>>>>>> >> i >>>>>>>> >> >> don't have more specific information yet -- including *what* >>>>>>>> failed, >>>>>>>> >> as our >>>>>>>> >> >> sysadmin is super busy ATM. the root cause was an incorrect >>>>>>>> circuit >>>>>>>> >> being >>>>>>>> >> >> switched off during building maintenance. >>>>>>>> >> >> >>>>>>>> >> >> on a side note, this incident will be accelerating our plan >>>>>>>> to move the >>>>>>>> >> >> entire jenkins infrastructure in to a managed datacenter >>>>>>>> environment. >>>>>>>> >> this >>>>>>>> >> >> will be our major push over the next couple of weeks. more >>>>>>>> details >>>>>>>> >> about >>>>>>>> >> >> this, also, as soon as i get them. >>>>>>>> >> >> >>>>>>>> >> >> i'm very sorry about the downtime, we'll get everything up >>>>>>>> and running >>>>>>>> >> >> ASAP. >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> >> >> On Thu, Sep 4, 2014 at 12:27 PM, shane knapp < >>>>>>>> skn...@berkeley.edu> >>>>>>>> >> wrote: >>>>>>>> >> >> >>>>>>>> >> >>> looks like a power outage in soda hall. more updates as >>>>>>>> they happen. >>>>>>>> >> >>> >>>>>>>> >> >>> >>>>>>>> >> >>> On Thu, Sep 4, 2014 at 12:25 PM, shane knapp < >>>>>>>> skn...@berkeley.edu> >>>>>>>> >> >>> wrote: >>>>>>>> >> >>> >>>>>>>> >> >>>> i am trying to get things up and running, but it looks like >>>>>>>> either >>>>>>>> >> the >>>>>>>> >> >>>> firewall gateway or jenkins server itself is down. i'll >>>>>>>> update as >>>>>>>> >> soon as >>>>>>>> >> >>>> i know more. >>>>>>>> >> >>>> >>>>>>>> >> >>> >>>>>>>> >> >>> >>>>>>>> >> >> >>>>>>>> >> > >>>>>>>> >> >>>>>>>> > >>>>>>>> > -- >>>>>>>> > You received this message because you are subscribed to the >>>>>>>> Google Groups >>>>>>>> > "amp-infra" group. >>>>>>>> > To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an >>>>>>>> > email to amp-infra+unsubscr...@googlegroups.com. >>>>>>>> > For more options, visit https://groups.google.com/d/optout. >>>>>>>> > >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
