----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27614/#review60961 -----------------------------------------------------------
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java <https://reviews.apache.org/r/27614/#comment102383> type should be the second argument. i.e. LOG.debug("class for [{0}] Action: [{1}]", type, function); core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java <https://reviews.apache.org/r/27614/#comment102385> Can we name this something else? It's easy to confuse it with LauncherMapper.CONF_OOZIE_ACTION_MAIN_CLASS core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java <https://reviews.apache.org/r/27614/#comment102384> Do we actually need to use this? In my experience, it causes nothing but trouble... core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java <https://reviews.apache.org/r/27614/#comment102389> These should have more unique names (like we do in the other actions). This helps prevent collisions as they go into the actionConf. e.g. oozie.spark.spark oozie.spark.master oozie.spark.mode etc core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java <https://reviews.apache.org/r/27614/#comment102387> We have this for DistCp because the classes are different between different versions of MapReduce and there's also two versions of DistCp. I'm not familiar enough with Spark, but is there any reason for a user to use a different Spark class? If not, then you can get rid of this to simplify things. In fact, SparkMain should subclass LauncherMain instead of JavaMain, and you can call SparkSubmit.main(...) directly instead of using reflection. Look at HiveMain or SqoopMain for examples. core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java <https://reviews.apache.org/r/27614/#comment102388> Do we actually need to use this property? In my experience, it causes nothing but trouble... sharelib/spark/pom.xml <https://reviews.apache.org/r/27614/#comment102390> whitespace sharelib/spark/pom.xml <https://reviews.apache.org/r/27614/#comment102391> version numbers should go in the root pom sharelib/spark/pom.xml <https://reviews.apache.org/r/27614/#comment102392> versions numbers should go in the root pom sharelib/spark/src/main/java/org.apache.oozie.action.hadoop/SparkMain.java <https://reviews.apache.org/r/27614/#comment102393> See earlier comment where I explain that this should subclass LauncherMain and other changes sharelib/spark/src/main/java/org.apache.oozie.action.hadoop/SparkMain.java <https://reviews.apache.org/r/27614/#comment102394> Most of the other actions have some additional logic to send information back to Oozie (e.g. launched child job ids, other stats). I'm not sure but I believe Spark jobs have an ID, right? Is it possible to return this back to Oozie as the child job? Look at some of the other actions to see how we do this. - Robert Kanter On Nov. 11, 2014, 6:35 a.m., pavan kumar kolamuri wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/27614/ > ----------------------------------------------------------- > > (Updated Nov. 11, 2014, 6:35 a.m.) > > > Review request for oozie and shwethags. > > > Bugs: OOZIE-1983 > https://issues.apache.org/jira/browse/OOZIE-1983 > > > Repository: oozie-git > > > Description > ------- > > Add spark action executor in oozie. Spark jobs can be run using oozie > > > Diffs > ----- > > client/src/main/java/org/apache/oozie/cli/OozieCLI.java 9c2d14b > client/src/main/resources/spark-action-0.1.xsd PRE-CREATION > core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java > 42f2965 > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java > 7349d3f > core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java > PRE-CREATION > core/src/main/resources/oozie-default.xml 17155a1 > pom.xml 1e79186 > sharelib/pom.xml aa479a8 > sharelib/spark/pom.xml PRE-CREATION > sharelib/spark/src/main/java/org.apache.oozie.action.hadoop/SparkMain.java > PRE-CREATION > > sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java > PRE-CREATION > > sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkMain.java > PRE-CREATION > src/main/assemblies/sharelib.xml 4a46b90 > webapp/pom.xml 35776c5 > > Diff: https://reviews.apache.org/r/27614/diff/ > > > Testing > ------- > > Both unit testing and end to end testing done > > > Thanks, > > pavan kumar kolamuri > >
