To prevent this in the future you could set up something that checks if
each worker has the path. If a worker doesn't satisfy the criteria then
just mark the worker offline or restart the process automatically.It's
doable with a maintenance script on the jenkins master. When it fails like
thi
Disconnecting and reconnecting each Jenkins worker appears to have resolved
the PATH issue: in the System Info page for each worker, I now see a PATH
which includes Anaconda.
To restart the worker processes, I only needed to hit the "Disconnect"
button in the Jenkins master UI for each worker, wai
hello from the canary islands! ;)
i just saw this thread, and another one about a quick power loss at the
colo where our machines are hosted. the master is on UPS but the workers
aren't... and when they come back, the PATH variable specified in the
workers' configs get dropped and we see behavi
Also another thing to look at is if you guys have any kinda of nightly
cleanup scripts for these workers that completely nuke the conda
environments. If there is one maybe that's why some of them recover after
a while. I don't know enough about your infra right now to understand all
the things th
So, right now it looks like 2 and 6 are still broken, but 7 has recovered:
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/SparkPullRequestBuilder/buildTimeTrend
What I am suggesting is to just perhaps modify the SparkPullRequestBuilder
configuration and run "which python" and t
Thanks, I actually don't have access to the machines or build configs to do
proper debugging on this. It looks like these workers are shared with
other build configurations like avocado and cannoli as well and really any
of the shared configs could be changing your JAVA_HOME and python
environme
Hi Xin!
Alyssa and I chatted just now and both reviewed the mango build scripts. We
don’t see anything in the mango build scripts that looks concerning. To give a
bit more context, Mango is a Spark-based application for visualizing genomics
data that is built in Scala, but which has python lang
Hi Xin!
Mango does install python dependencies, but they should all be inside of a
conda environment. My guess is that we've got somewhere in the mango Jenkins
build where something is getting installed outside of the conda environment.
I'll be looking into this shortly.
Regards,
Frank Austin
I'm not entirely sure if it's the cause because I can't see the build
configurations, but just looking at the build logs it looks like they share
a pool and those mango builds run some setup with python.
On Sat, Nov 4, 2017 at 9:19 PM, Frank Austin Nothaft
wrote:
> Hi folks,
>
> Alyssa (cc’ed) a
Hi folks,
Alyssa (cc’ed) and I manage the mango build on the AMPLab Jenkins. I will start
to look into this to see what the connection between the mango builds and the
failing Spark builds are.
Regards,
Frank Austin Nothaft
fnoth...@berkeley.edu
fnoth...@eecs.berkeley.edu
202-340-0466
> On No
Sorry, mango wasn't added recently, but it looks like after successful
builds of this specific configuration the workers break:
https://amplab.cs.berkeley.edu/jenkins/job/mango/HADOOP_VERSION=2.6.0,SCALAVER=2.11,SPARK_VERSION=1.6.1,label=centos/
And then after another configuration runs it recove
It has happened with other workers as well, namely 3 and 4 and then
recovered. Looking at the build history it looks like a project called
mango has been added to this pool of machines recently:
https://amplab.cs.berkeley.edu/jenkins/job/mango/
It looks like the slaves start to fail spark pull re
I assume it is as it says:
Python versions prior to 2.7 are not supported.
Looks this happens in worker 2, 6 and 7 given my observation.
On 4 Nov 2017 5:15 pm, "Sean Owen" wrote:
Agree, seeing this somewhat regularly on the pull request builder. Do some
machines inadvertently have Python 2.6
Agree, seeing this somewhat regularly on the pull request builder. Do some
machines inadvertently have Python 2.6? some builds succeed, so may just be
one or a few. CC Shane.
On Thu, Nov 2, 2017 at 5:39 PM Pralabh Kumar wrote:
> Hi Dev
>
> Spark build is failing in Jenkins
>
>
> https://amplab.c
Hi Dev
Spark build is failing in Jenkins
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83353/consoleFull
Python versions prior to 2.7 are not supported.
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
ERROR: Step ?Publish JUnit
15 matches
Mail list logo