[ https://issues.apache.org/jira/browse/OOZIE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14367700#comment-14367700 ]
Hadoop QA commented on OOZIE-2170: ---------------------------------- Testing JIRA OOZIE-2170 Cleaning local git workspace ---------------------------- {color:green}+1 PATCH_APPLIES{color} {color:green}+1 CLEAN{color} {color:green}+1 RAW_PATCH_ANALYSIS{color} . {color:green}+1{color} the patch does not introduce any @author tags . {color:green}+1{color} the patch does not introduce any tabs . {color:green}+1{color} the patch does not introduce any trailing spaces . {color:green}+1{color} the patch does not introduce any line longer than 132 . {color:green}+1{color} the patch does adds/modifies 2 testcase(s) {color:green}+1 RAT{color} . {color:green}+1{color} the patch does not seem to introduce new RAT warnings {color:green}+1 JAVADOC{color} . {color:green}+1{color} the patch does not seem to introduce new Javadoc warnings {color:green}+1 COMPILE{color} . {color:green}+1{color} HEAD compiles . {color:green}+1{color} patch compiles . {color:green}+1{color} the patch does not seem to introduce new javac warnings {color:green}+1 BACKWARDS_COMPATIBILITY{color} . {color:green}+1{color} the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations . {color:green}+1{color} the patch does not modify JPA files {color:red}-1 TESTS{color} . Tests run: 1626 . Tests failed: 1 . Tests errors: 0 . The patch failed the following testcases: . testCoordRerunDateNeg(org.apache.oozie.command.coord.TestCoordRerunXCommand) {color:green}+1 DISTRO{color} . {color:green}+1{color} distro tarball builds with the patch ---------------------------- {color:red}*-1 Overall result, please check the reported -1(s)*{color} The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/2261/ > Oozie should automatically set configs to make Spark jobs show up in the > Spark History Server > --------------------------------------------------------------------------------------------- > > Key: OOZIE-2170 > URL: https://issues.apache.org/jira/browse/OOZIE-2170 > Project: Oozie > Issue Type: Improvement > Components: action > Affects Versions: trunk > Reporter: Robert Kanter > Assignee: Robert Kanter > Attachments: OOZIE-2170.patch, OOZIE-2170.patch > > > If you use "yarn-cluster" for the Spark action's master, the Spark jobs don't > show up in the Spark History Server or properly link to it from the Spark AM. > The user needs to set this in their Spark action in the workflow.xml: > {code:xml} > <spark-opts>--conf spark.yarn.historyServer.address=http://SPH18088 --conf > spark.eventLog.dir=hdfs://NN:8020/user/spark/applicationHistory --conf > spark.eventLog.enabled=true</spark-opts> > {code} > It would be nice if Oozie did this automatically via some oozie-site.xml > config(s). We could do something similar how the hadoop configs are setup > where it will load a Spark .conf file from a directory based on the RM > specified in the <job-tracker>. > While we're at it, it might be good to document how to use Spark on YARN: > # Include the spark-assembly jar with your workflow (this is unfortunately > not published in maven) > # Specify "yarn-cluster" as the master > Also, the Spark example should delete the output dir in {{<prepare>}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)