----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48298/#review136473 -----------------------------------------------------------
sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java <https://reviews.apache.org/r/48298/#comment201517> Can we retain the comments about different modes. That is useful information sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java <https://reviews.apache.org/r/48298/#comment201520> What about this case? mapreduce.cache.files will not contain hadoop jars. How will that work now? sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java (line 129) <https://reviews.apache.org/r/48298/#comment201519> What if user has already specified --files option? Should we append to it like we do for other options? - Rohini Palaniswamy On June 6, 2016, 11:26 p.m., Satish Saley wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48298/ > ----------------------------------------------------------- > > (Updated June 6, 2016, 11:26 p.m.) > > > Review request for oozie. > > > Bugs: OOZIE-2547 > https://issues.apache.org/jira/browse/OOZIE-2547 > > > Repository: oozie-git > > > Description > ------- > > Currently, we pass jars using --jars option while submitting spark job. Also, > we add spark.yarn.dist.files option in case of yarn-client mode. > Instead of that, we can have only --files option and pass on the files which > are present in mapreduce.job.cache.files. While doing so, we make sure that > spark won't make another copy of the files if files exist on the hdfs. We saw > the issues when files are getting copied multiple times and causing > exceptions such as : > > Diagnostics: Resource > hdfs://localhost/user/saley/.sparkStaging/application_1234_123/oozie-examples.jar > changed on src filesystem > > > Diffs > ----- > > sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java > 0e6e271 > > Diff: https://reviews.apache.org/r/48298/diff/ > > > Testing > ------- > > I have tested it in local, yarn-cluster and yarn-client. > > > File Attachments > ---------------- > > OOZIE-2547-1.patch > > https://reviews.apache.org/media/uploaded/files/2016/06/06/6d718b0f-4533-4e9e-992a-03aea079a6c7__OOZIE-2547-1.patch > > > Thanks, > > Satish Saley > >
