Re: amplab jenkins is down

Sounds good! Thanks for the update Shane. On Wed, Oct 1, 2014 at 4:44 PM, shane knapp wrote: > as of this morning, i've got the new jenkins up, with all of the current > builds set up (but failing). i'm in the middle of playing setup/debug > whack-a-mole, but we're getting there. my guess woul

Re: amplab jenkins is down

as of this morning, i've got the new jenkins up, with all of the current builds set up (but failing). i'm in the middle of playing setup/debug whack-a-mole, but we're getting there. my guess would be early next week for the switchover. On Wed, Oct 1, 2014 at 12:53 PM, Nicholas Chammas < nicholas

Re: amplab jenkins is down

On Thu, Sep 4, 2014 at 4:19 PM, shane knapp wrote: > on a side note, this incident will be accelerating our plan to move the > entire jenkins infrastructure in to a managed datacenter environment. > this > will be our major push over the next couple of weeks. more details about > this, also, as

Re: amplab jenkins is down

i'll take a look at this today. On Mon, Sep 8, 2014 at 1:13 AM, Josh Rosen wrote: > Yeah, I think https://github.com/apache/spark/pull/2315 should have fixed > the Mima issue. We're still seeing some intermittent failures due to > DriverSuite and SparkSubmitSuite tests failing, so I'd apprecia

Re: amplab jenkins is down

Yeah, I think https://github.com/apache/spark/pull/2315 should have fixed the Mima issue. We're still seeing some intermittent failures due to DriverSuite and SparkSubmitSuite tests failing, so I'd appreciate any help in diagnosing that issue. On Sun, Sep 7, 2014 at 10:08 PM, Prashant Sharma wro

Re: amplab jenkins is down

Looks like this is already taken care of ? Prashant Sharma On Mon, Sep 8, 2014 at 4:37 AM, Josh Rosen wrote: > Does anyone know why some of the MiMa tests have started failing? > > See > https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19948/consoleFull > for > an example.

Re: amplab jenkins is down

Does anyone know why some of the MiMa tests have started failing? See  https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19948/consoleFull  for an example. On September 6, 2014 at 12:48:27 PM, Josh Rosen (rosenvi...@gmail.com) wrote: It looks like Jenkins is up and running, but

Re: amplab jenkins is down

It looks like Jenkins is up and running, but there seems to be a delay in responding to requests to re-test patches.  It seems like Jenkins is promptly testing new PRs, or new commits as they’re added to existing PRs, but taking a very long time to respond to requests to re-test PRs. I’m going

Re: amplab jenkins is down

We have successfully purged Jenkins’ build queue.  If you want a PR to be re-tested, please ask Jenkins again. On September 5, 2014 at 5:36:30 PM, shane knapp (skn...@berkeley.edu) wrote: yeah, it was a problem w/the PRB's OAuth key. josh rosen added a new key, and magique! we're about to c

Re: amplab jenkins is down

yeah, it was a problem w/the PRB's OAuth key. josh rosen added a new key, and magique! we're about to clear the queue of all builds as most aren't wanted/needed. On Fri, Sep 5, 2014 at 5:33 PM, Nicholas Chammas wrote: > Looks like Jenkins is back! > > lol The poor guy has like a million build

Re: amplab jenkins is down

Looks like Jenkins is back! lol The poor guy has like a million builds to catch up on. On Fri, Sep 5, 2014 at 4:15 PM, Nicholas Chammas wrote: > How's it going? > > It looks like during the las

Re: amplab jenkins is down

How's it going? It looks like during the last build from about 30 min ago Jenkins was still having trouble fetching from GitHub. It also looks like not all requests for testing are

Re: amplab jenkins is down

it's looking like everything except the pull request builders are working. i'm going to be working on getting this resolved today. On Fri, Sep 5, 2014 at 8:18 AM, Nicholas Chammas wrote: > Hmm, looks like at least some builds >

Re: amplab jenkins is down

Hmm, looks like at least some builds are working now, though this last one was from ~5 hours ago. On Fri, Sep 5, 2014 at 1:02 AM, shane knapp wrote: > yep. that's exactly the b

Re: amplab jenkins is down

yep. that's exactly the behavior i saw earlier, and will be figuring out first thing tomorrow morning. i bet it's an environment issues on the slaves. On Thu, Sep 4, 2014 at 7:10 PM, Nicholas Chammas wrote: > Looks like during the last build >

Re: amplab jenkins is down

Looks like during the last build Jenkins was unable to execute a git fetch? On Thu, Sep 4, 2014 at 7:58 PM, shane knapp wrote: > i'm going to restart jenkins and see if that fixes t

Re: amplab jenkins is down

i'm going to restart jenkins and see if that fixes things. On Thu, Sep 4, 2014 at 4:56 PM, shane knapp wrote: > looking > > > On Thu, Sep 4, 2014 at 4:21 PM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> It appears that our main man is having trouble >>

Re: amplab jenkins is down

looking On Thu, Sep 4, 2014 at 4:21 PM, Nicholas Chammas wrote: > It appears that our main man is having trouble > > hearing new requests >

Re: amplab jenkins is down

Hm yeah it seems that it hasn't been polling since 3:45. On Thu, Sep 4, 2014 at 4:21 PM, Nicholas Chammas wrote: > It appears that our main man is having trouble hearing new requests. > > Do we need some smelling salts? > > > On Thu, Sep 4, 2014 at 5:49 PM, shane knapp wrote: >> >> i'd ping the

Re: amplab jenkins is down

It appears that our main man is having trouble hearing new requests . Do we need some smelling salts? On Thu, Sep 4, 2014 at 5:49

Re: amplab jenkins is down

i'd ping the Jenkinsmench... the master was completely offline, so any new jobs wouldn't have reached it. any jobs that were queued when power was lost probably started up, but jobs that were running would fail. On Thu, Sep 4, 2014 at 2:45 PM, Nicholas Chammas wrote: > Woohoo! Thanks Shane. >

Re: amplab jenkins is down

Woohoo! Thanks Shane. Do you know if queued PR builds will automatically be picked up? Or do we have to ping the Jenkinmensch manually from each PR? Nick On Thu, Sep 4, 2014 at 5:37 PM, shane knapp wrote: > AND WE'RE UP! > > sorry that this took so long... i'll send out a more detailed expla

Re: amplab jenkins is down

AND WE'RE UP! sorry that this took so long... i'll send out a more detailed explanation of what happened soon. now, off to back up jenkins. shane On Thu, Sep 4, 2014 at 1:27 PM, shane knapp wrote: > it's a faulty power switch on the firewall, which has been swapped out. > we're about to re

Re: amplab jenkins is down

it's a faulty power switch on the firewall, which has been swapped out. we're about to reboot and be good to go. On Thu, Sep 4, 2014 at 1:19 PM, shane knapp wrote: > looks like some hardware failed, and we're swapping in a replacement. i > don't have more specific information yet -- including

Re: amplab jenkins is down

looks like some hardware failed, and we're swapping in a replacement. i don't have more specific information yet -- including *what* failed, as our sysadmin is super busy ATM. the root cause was an incorrect circuit being switched off during building maintenance. on a side note, this incident wi

Re: amplab jenkins is down

looks like a power outage in soda hall. more updates as they happen. On Thu, Sep 4, 2014 at 12:25 PM, shane knapp wrote: > i am trying to get things up and running, but it looks like either the > firewall gateway or jenkins server itself is down. i'll update as soon as > i know more. >

amplab jenkins is down

i am trying to get things up and running, but it looks like either the firewall gateway or jenkins server itself is down. i'll update as soon as i know more.