----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48298/ -----------------------------------------------------------
(Updated June 8, 2016, 1:08 p.m.) Review request for oozie. Bugs: OOZIE-2547 https://issues.apache.org/jira/browse/OOZIE-2547 Repository: oozie-git Description ------- Currently, we pass jars using --jars option while submitting spark job. Also, we add spark.yarn.dist.files option in case of yarn-client mode. Instead of that, we can have only --files option and pass on the files which are present in mapreduce.job.cache.files. While doing so, we make sure that spark won't make another copy of the files if files exist on the hdfs. We saw the issues when files are getting copied multiple times and causing exceptions such as : Diagnostics: Resource hdfs://localhost/user/saley/.sparkStaging/application_1234_123/oozie-examples.jar changed on src filesystem Diffs (updated) ----- docs/src/site/twiki/DG_SparkActionExtension.twiki d7d75a1 sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java 0e6e271 Diff: https://reviews.apache.org/r/48298/diff/ Testing ------- I have tested it in local, yarn-cluster and yarn-client. File Attachments ---------------- OOZIE-2547-1.patch https://reviews.apache.org/media/uploaded/files/2016/06/06/6d718b0f-4533-4e9e-992a-03aea079a6c7__OOZIE-2547-1.patch Making changes suggested by Rohini https://reviews.apache.org/media/uploaded/files/2016/06/08/e26bb7e3-7011-4549-9026-0b7cb4262f76__OOZIE-2547-2.patch Thanks, Satish Saley