-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48298/
-----------------------------------------------------------
Review request for oozie.
Bugs: OOZIE-2547
https://issues.apache.org/jira/browse/OOZIE-2547
Repository: oozie-git
Description
-------
Currently, we pass jars using --jars option while submitting spark job. Also,
we add spark.yarn.dist.files option in case of yarn-client mode.
Instead of that, we can have only --files option and pass on the files which
are present in mapreduce.job.cache.files. While doing so, we make sure that
spark won't make another copy of the files if files exist on the hdfs. We saw
the issues when files are getting copied multiple times and causing exceptions
such as :
Diagnostics: Resource
hdfs://localhost/user/saley/.sparkStaging/application_1234_123/oozie-examples.jar
changed on src filesystem
Diffs
-----
sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java
0e6e271
Diff: https://reviews.apache.org/r/48298/diff/
Testing
-------
I have tested it in local, yarn-cluster and yarn-client.
File Attachments
----------------
OOZIE-2547-1.patch
https://reviews.apache.org/media/uploaded/files/2016/06/06/6d718b0f-4533-4e9e-992a-03aea079a6c7__OOZIE-2547-1.patch
Thanks,
Satish Saley
