[ https://issues.apache.org/jira/browse/OOZIE-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
abhishek bafna updated OOZIE-2277: ---------------------------------- Fix Version/s: (was: trunk) 4.3.0 > Honor oozie.action.sharelib.for.spark in Spark jobs > --------------------------------------------------- > > Key: OOZIE-2277 > URL: https://issues.apache.org/jira/browse/OOZIE-2277 > Project: Oozie > Issue Type: Improvement > Reporter: Ryan Brush > Assignee: Robert Kanter > Priority: Minor > Fix For: 4.3.0 > > Attachments: OOZIE-2277.001.patch, OOZIE-2277.002.patch, > OOZIE-2277.004.patch > > > Shared libraries specified by oozie.action.sharelib.for.spark are not visible > in the Spark job itself. For instance, setting > oozie.action.sharelib.for.spark to "spark,hcat" will not make the hcat jars > usable in the Spark job. This is inconsistent with other actions (such as > Java and MapReduce actions). > Since the Spark action just calls SparkSubmit, it looks like we would need to > explicitly pass the jars for the specified sharelibs into the SparkSubmit > operation so they are available to the Spark operation itself. > One option: we can just pass the HDFS URLs to that command via the --jars > parameter. This is actually what I've done to work around this issue; it > makes for a long SparkSubmit command but works. -- This message was sent by Atlassian JIRA (v6.3.4#6332)