EBernhardson has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/387831 )
Change subject: [WIP] respect XDG_CACHE_DIR for ivy cache in test
......................................................................
[WIP] respect XDG_CACHE_DIR for ivy cache in test
By default spark jar resolution uses ivy and defaults
the cache to the users home directory. In the docker
CI pipeline the user has no home directory and this
fails. Additionally CI wants to keep all caches in
a single directory to make it easy to re-use in a
standardized way between tests.
Change-Id: I597bdab6b58ca66444bb447ecfb9dd788a46a3fd
---
M jvm/pom.xml
M mjolnir/test/conftest.py
2 files changed, 5 insertions(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/search/MjoLniR
refs/changes/31/387831/1
diff --git a/jvm/pom.xml b/jvm/pom.xml
index b2a7f71..365bc65 100644
--- a/jvm/pom.xml
+++ b/jvm/pom.xml
@@ -11,7 +11,7 @@
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
<maven.version>3.3.9</maven.version>
- <spark.version>2.1.0</spark.version>
+ <spark.version>2.1.2</spark.version>
<scala.version>2.11.8</scala.version>
<scala.binary.version>2.11</scala.binary.version>
<xgboost.version>0.7-wmf-1</xgboost.version>
diff --git a/mjolnir/test/conftest.py b/mjolnir/test/conftest.py
index dff4406..e74132a 100644
--- a/mjolnir/test/conftest.py
+++ b/mjolnir/test/conftest.py
@@ -42,6 +42,7 @@
os.environ['PYSPARK_SUBMIT_ARGS'] = '--repositories %s pyspark-shell' % (
','.join(['https://archiva.wikimedia.org/repository/%s' % (repo)
for repo in ['releases', 'snapshots', 'mirrored']]))
+
conf = (
SparkConf()
.setMaster("local[2]")
@@ -55,6 +56,9 @@
# way too many for our small test cases. This cuts execution
# time of the tests in half.
.set('spark.sql.shuffle.partitions', 4))
+ if 'XDG_CACHE_DIR' in os.environ:
+ conf.set('spark.jars.ivy', os.path.join(os.environ['XDG_CACHE_DIR'],
'ivy2'))
+
sc = SparkContext(conf=conf)
yield sc
sc.stop()
--
To view, visit https://gerrit.wikimedia.org/r/387831
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I597bdab6b58ca66444bb447ecfb9dd788a46a3fd
Gerrit-PatchSet: 1
Gerrit-Project: search/MjoLniR
Gerrit-Branch: master
Gerrit-Owner: EBernhardson <ebernhard...@wikimedia.org>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
