Re: Spark build is failing in amplab Jenkins

Disconnecting and reconnecting each Jenkins worker appears to have resolved the PATH issue: in the System Info page for each worker, I now see a PATH which includes Anaconda. To restart the worker processes, I only needed to hit the "Disconnect" button in the Jenkins master UI for each worker,

Re: Spark build is failing in amplab Jenkins

hello from the canary islands! ;) i just saw this thread, and another one about a quick power loss at the colo where our machines are hosted. the master is on UPS but the workers aren't... and when they come back, the PATH variable specified in the workers' configs get dropped and we see

Re: Spark build is failing in amplab Jenkins

Also another thing to look at is if you guys have any kinda of nightly cleanup scripts for these workers that completely nuke the conda environments. If there is one maybe that's why some of them recover after a while. I don't know enough about your infra right now to understand all the things

Re: Spark build is failing in amplab Jenkins

Thanks, I actually don't have access to the machines or build configs to do proper debugging on this. It looks like these workers are shared with other build configurations like avocado and cannoli as well and really any of the shared configs could be changing your JAVA_HOME and python

Re: Spark build is failing in amplab Jenkins

Hi Xin! Alyssa and I chatted just now and both reviewed the mango build scripts. We don’t see anything in the mango build scripts that looks concerning. To give a bit more context, Mango is a Spark-based application for visualizing genomics data that is built in Scala, but which has python

Re: Spark build is failing in amplab Jenkins

Hi Xin! Mango does install python dependencies, but they should all be inside of a conda environment. My guess is that we've got somewhere in the mango Jenkins build where something is getting installed outside of the conda environment. I'll be looking into this shortly. Regards, Frank

Re: Spark build is failing in amplab Jenkins

I'm not entirely sure if it's the cause because I can't see the build configurations, but just looking at the build logs it looks like they share a pool and those mango builds run some setup with python. On Sat, Nov 4, 2017 at 9:19 PM, Frank Austin Nothaft wrote: > Hi

Re: Spark build is failing in amplab Jenkins

Hi folks, Alyssa (cc’ed) and I manage the mango build on the AMPLab Jenkins. I will start to look into this to see what the connection between the mango builds and the failing Spark builds are. Regards, Frank Austin Nothaft fnoth...@berkeley.edu fnoth...@eecs.berkeley.edu 202-340-0466 > On

Re: Spark build is failing in amplab Jenkins

Sorry, mango wasn't added recently, but it looks like after successful builds of this specific configuration the workers break: https://amplab.cs.berkeley.edu/jenkins/job/mango/HADOOP_VERSION=2.6.0,SCALAVER=2.11,SPARK_VERSION=1.6.1,label=centos/ And then after another configuration runs it

Re: Spark build is failing in amplab Jenkins

It has happened with other workers as well, namely 3 and 4 and then recovered. Looking at the build history it looks like a project called mango has been added to this pool of machines recently: https://amplab.cs.berkeley.edu/jenkins/job/mango/ It looks like the slaves start to fail spark pull

Re: Spark build is failing in amplab Jenkins

I assume it is as it says: Python versions prior to 2.7 are not supported. Looks this happens in worker 2, 6 and 7 given my observation. On 4 Nov 2017 5:15 pm, "Sean Owen" wrote: Agree, seeing this somewhat regularly on the pull request builder. Do some machines

Re: Spark build is failing in amplab Jenkins

Agree, seeing this somewhat regularly on the pull request builder. Do some machines inadvertently have Python 2.6? some builds succeed, so may just be one or a few. CC Shane. On Thu, Nov 2, 2017 at 5:39 PM Pralabh Kumar wrote: > Hi Dev > > Spark build is failing in

Spark build is failing in amplab Jenkins

Hi Dev Spark build is failing in Jenkins https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83353/consoleFull Python versions prior to 2.7 are not supported. Build step 'Execute shell' marked build as failure Archiving artifacts Recording test results ERROR: Step ?Publish