Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23218
it's building w/the proper java now. from the build log:
```
[EnvInject] - Injecting as environment variables the properties content
JENKINS_MASTER_HOSTNAME=amp-jenkins-master
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23218
ok, PRB builds updated w/the new JAVA_HOME
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23218
oh crap forgot about the PRB. updating the config now.
On Fri, Dec 7, 2018 at 2:58 PM Dongjoon Hyun
wrote:
> @shaneknapp <https://github.com/shaneknapp> .
&g
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23218
okie dokie, java8 update is done!
https://issues.apache.org/jira/browse/SPARK-26282
---
-
To unsubscribe, e-mail: reviews
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23218
https://issues.apache.org/jira/browse/SPARK-26282
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23218
shouldn't be too hard, but it will require some downtime.
On Wed, Dec 5, 2018 at 5:41 AM Sean Owen wrote:
> Ah OK, so all of them were a JVM crash. It would probably be a g
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23148
> This is already useful in that we can just ask people to run dev/scalafmt
(I'll update developer guids) as the output style looks _also_ just fine. I
won't try to have this automatically
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23148
sgtm, i'll be more than happy to review once i get in to the office. :)
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23148
i think the best place for this, imo, is in dev/run-tests.py (and called
via run-test-jenkins.py).
On Thu, Nov 29, 2018 at 6:56 AM Sean Owen wrote:
> Merged to mas
Github user shaneknapp commented on a diff in the pull request:
https://github.com/apache/spark/pull/23117#discussion_r236488706
--- Diff: dev/run-tests.py ---
@@ -434,6 +434,63 @@ def run_python_tests(test_modules, parallelism):
run_cmd(command)
+def
Github user shaneknapp commented on a diff in the pull request:
https://github.com/apache/spark/pull/23117#discussion_r236440252
--- Diff: dev/run-tests.py ---
@@ -434,6 +434,63 @@ def run_python_tests(test_modules, parallelism):
run_cmd(command)
+def
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23117
not yet, but i will carve out some time today and wednesday to look closer.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23117
i'll try and take a look at this over the next couple of days, but it's a
holiday weekend and i may not be able to get to this until monday
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22994
gonna hold off on backporting until i inspect each branch independently.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22994
okie dokie... this will be my first official push to the spark repo! :)
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22994
howdy howdy! i opened this nearly 2 weeks ago, and was wondering if i
could get another set of eyeballs on it...
@holdenk @srowen @felixcheung
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23061
thanks @vanzin
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23061
> BTW noticed this:
>
> ```
> Step 6/9 : COPY R ${SPARK_HOME}/R
> COPY failed: stat /var/lib/docker/tmp/docker-builder004084591/R: no such
file or directory
&g
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23061
edit the subject, add [k8s], then ask jenkins to trigger a build.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23017
btw i wiped all of my .ivy2 and .m2 dirs before building, just in case
we're looking at a poisoned artifact
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23017
> manually on a different machine or just on the workers?
on a different machine (your laptop, something local, etc).
i'm trying on a couple of different workers and it's alw
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23017
@vanzin @ifilonenko can you guys try and build the dist locally on your dev
laptops? here's a little wrapper script to make it easier:
you'll need to update your PATH to have some
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23017
yep it gets to the same spot when i try and build manually, and fails:
```
[INFO] --- maven-source-plugin:3.0.1:test-jar-no-fork (create-source-jar) @
spark-mllib-local_2.12 ---
[INFO
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23017
@vanzin i'm really not sure what's going on w/this. i noticed it happening
on research-jenkins-worker-07 yesterday, so i rebooted the box and that seemed
to fix it.
now it's back
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23026
> > if such a list exists it should be the same list that triggers regular
tests.
>
> I defer that to @shaneknapp
no, @vanzin is right. i'll update t
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22911
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22967
@dongjoon-hyun not a problem, i'll need to update the build config(s)
what branches will need 2.12 vs 2.11
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23012
@felixcheung @HyukjinKwon
yes: deprecation in this case means we test against R-3.1.1
---
-
To unsubscribe, e-mail
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23012
TL;DR: let's go w/deprecation.
still TL;DR: if i never have to install or manage R again, i will be a
happy person!
@HyukjinKwon upgrading R is easy. getting the right mix
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22994
nah, it's not urgent at all.
i also tested this locally by breaking various python scripts and
confirming that it caught errors as expected.
On Tue, Nov 13, 2018 at 5:03 PM
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22994
(weird, github ate my last comment)
@HyukjinKwon i think we're g2g for merging this in to master and backports.
you want to do this, or should i
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23019
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23012
howdy howdy!
unless we dockerize spark builds (someday!), we're going to be stuck
w/testing against one version of R on the jenkins workers... i've been
looking in to packrat to help
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23019
ok, i think i fixed the problem w/the jenkins config.
test this please
---
-
To unsubscribe, e-mail: reviews
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23019
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/23019
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22994
alright, i think we're g2g. i'll squash my commits now before merging.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22994
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22994
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22994
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22963
re https://github.com/apache/spark/pull/22963#issuecomment-437133365
i checked, and the only one we can seemingly download independently is
pycodestyle
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22994
i have no idea why the tests aren't passing btw. :\
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22994
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22994
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22994
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22994
see the output from the following build to get the gory details of what's
happening:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98664/console
i removed `set
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22994
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22994
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22994
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22963
orthogonal to this PR, but just as an FYI:
https://github.com/apache/spark/pull/22994
---
-
To unsubscribe, e-mail
GitHub user shaneknapp opened a pull request:
https://github.com/apache/spark/pull/22994
this is serious refactor
## What changes were proposed in this pull request?
`dev/lint-python` is a mess of nearly unreadable bash. i would like to fix
that as best as i can
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22963
> I don't have the context here to have a strong opinion, but, it seems
like we should make the test env setup self-contained if possible, to avoid
dependencies on and maintenance of the bu
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22963
pydocstyle tests passed w/o issue btw:
https://amplab.cs.berkeley.edu/jenkins/job/ubuntuSparkPRB/134/consoleFull
this is on ubuntu w/python 3.5, flake8 3.6.0, pydocstyle 3.0.0
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22963
interesting. from the flake8 webpage:
```
It is very important to install Flake8 on the correct version of Python for
your needs. If you want Flake8 to properly parse new language
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22963
ok, i missed that previous comment about flake8 and pycodestyle version
incompatibility.
since we're running flake8 3.50, i agree that this could be causing
problems w/pycodestyle
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22963
sorry to jump in late on this, but i just wanted to check in on some of
this stuff... `dev/lint-python` is a nightmare and i just wanted to discuss a
couple of things:
1) the script
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22931
sorry, ignore amplab's report. the build passed, but my hacking on the
integration test reports was what caused the failure
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22931
btw next k8s test that runs will actually have logs!
here's the integration test log from this run, which wasn't archived...
[integration-tests.log](https://github.com/apache/spark
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22931
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22145
closer. just started working again on the ubuntu port about a week ago.
hopefully before EOY.
On Thu, Nov 1, 2018 at 10:41 AM mccheah wrote:
> Just wanted to p
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22896
actually can we merge this? it's causing spurious lintr errors.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22896
thanks @HyukjinKwon !
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22896
appveyor timeout #fml
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22896
nteresting... still get the `StartTag` error in the build log:
```
Error: StartTag: invalid element name [68]
Execution halted
```
it's orthogonal to this PR
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22896
i am confused, though, as to why this wasn't caught during the PR that
added this file:
https://github.com/apache/spark/pull/22455
on the centos workers, there was an error message
GitHub user shaneknapp opened a pull request:
https://github.com/apache/spark/pull/22896
found some extra whitespace in the R tests
## What changes were proposed in this pull request?
during my ubuntu-port testing, i found some extra whitespace that for some
reason wasn't
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22146
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22146
@mccheah fixed. for some reason, and on this build node only, that dir
owner was set to root:root. this is fixed
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22824
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22824
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22854
yeah... wasn't worried about the PRB failing. thanks for the
merge/backport!
---
-
To unsubscribe, e-mail: reviews
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22854
build is green, and everything looks to be behaving normally!
https://amplab.cs.berkeley.edu/jenkins/job/sknapp-testing-spark-branch-2.4-test-maven-hadoop-2.7/12/console
ready
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22854
> Merged build finished. Test FAILed.
that was me killing the 1st PRB build
---
-
To unsubscribe, e-mail: revi
Github user shaneknapp commented on a diff in the pull request:
https://github.com/apache/spark/pull/22854#discussion_r228616782
--- Diff: build/mvn ---
@@ -163,8 +163,14 @@ export MAVEN_OPTS=${MAVEN_OPTS:-"$_COMPILE_JVM_OPTS"}
echo "Using \`mvn\` from pa
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22854
let's not merge until this build passes:
https://amplab.cs.berkeley.edu/jenkins/job/sknapp-testing-spark-branch-2.4-test-maven-hadoop-2.7/12
Github user shaneknapp commented on a diff in the pull request:
https://github.com/apache/spark/pull/22854#discussion_r228607477
--- Diff: build/mvn ---
@@ -163,8 +163,14 @@ export MAVEN_OPTS=${MAVEN_OPTS:-"$_COMPILE_JVM_OPTS"}
echo "Using \`mvn\` from pa
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22854
`mvn clean package` script logic worked!
```
[INFO]
[INFO] BUILD SUCCESS
[INFO
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22854
here's the test build run that's actually testing the changes to mvn:
https://amplab.cs.berkeley.edu/jenkins/job/sknapp-testing-spark-branch-2.4-test-maven-hadoop-2.7/12
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22854
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user shaneknapp commented on a diff in the pull request:
https://github.com/apache/spark/pull/22854#discussion_r228591941
--- Diff: build/mvn ---
@@ -163,8 +163,19 @@ export MAVEN_OPTS=${MAVEN_OPTS:-"$_COMPILE_JVM_OPTS"}
echo "Using \`mvn\` from pa
Github user shaneknapp commented on a diff in the pull request:
https://github.com/apache/spark/pull/22854#discussion_r228584294
--- Diff: build/mvn ---
@@ -163,8 +163,19 @@ export MAVEN_OPTS=${MAVEN_OPTS:-"$_COMPILE_JVM_OPTS"}
echo "Using \`mvn\` from pa
GitHub user shaneknapp opened a pull request:
https://github.com/apache/spark/pull/22854
[SPARK-25854] fix mvn to not always exit 1
## What changes were proposed in this pull request?
the final line in the mvn helper script in build/ attempts to shut down the
zinc server
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22501
@cloud-fan -- pip isn't broken... the actual error is found right above
what you cut and pasted:
`UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position
2719
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22748
k8s tests are triggered by the subject of the PR. if the test isn't being
triggered (which it appears to be), then it will require a jenkins restart
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22615
also, i will be deleting the following jobs:
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-master-test-sbt-hadoop-2.6/
https://amplab.cs.berkeley.edu/jenkins
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22615
ok just to revisit this: i'm going to push out the new jenkins jobs
configs now, and not gate on moving these to the spark repo
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22615
i haven't had a chance to do any of the jenkins stuff... after being
sidetracked by the conversation to move the configs to the spark repo, plus
planning for our big event that starts tomorrow
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22615
>
>
> I want to see the configurations ..
>
>
> they're just some absolutely breath-taking bits of yaml to define all of
the spark jenkins jobs.
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22615
@srowen fair 'nuf... i'll create a jira for this tomorrow and we can hash
out final design shite there (rather than overloading this PR
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22615
@vanzin i'm not opposed to hosting these configs somewhere else.
@JoshRosen did this a few years back just to "get shit done"...
i'd be leery of putting this in to the main
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22615
https://github.com/databricks/spark-jenkins-configurations/pull/47
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22615
@srowen sure, manually removing the failing jobs is one option... but
since we auto-generate the job configs, any time we add a new branch they'll
come back.
i'd much rather do
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22615
consider me pinged. ;)
i will need to do some refactoring in the jenkins job builder configs for
this, so we'll definitely need to coordinate before this is merged.
most likely
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22145
yes, hopefully soon. i won't be able to start on this for at least another
week due to our lab having a big event this coming week.
On Sat, Oct 6, 2018 at 5:55 PM Felix Cheung
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22061
no, not yet. i'll give a bigger update later today on the status of this,
but the TL;DR is that it's a bigger project than i expected. :\
On Mon, Sep 24, 2018 at 9:18 AM, Sean Owen
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22333
moving any parts of the spark build infrastructure to use docker is a big
project and not happening in the next few months
Github user shaneknapp commented on a diff in the pull request:
https://github.com/apache/spark/pull/22266#discussion_r214117901
--- Diff: dev/lint-python ---
@@ -82,6 +82,26 @@ else
rm "$PYCODESTYLE_REPORT_PATH"
fi
+python -m pip install flake8
Github user shaneknapp commented on a diff in the pull request:
https://github.com/apache/spark/pull/22266#discussion_r214114395
--- Diff: dev/lint-python ---
@@ -82,6 +82,26 @@ else
rm "$PYCODESTYLE_REPORT_PATH"
fi
+python -m pip install flake8
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22257
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22257
the network was disabled, so i fixed that:
```
virsh # net-list --all
Name State Autostart Persistent
1 - 100 of 654 matches
Mail list logo
