[
https://issues.apache.org/jira/browse/PIG-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14733564#comment-14733564
]
Srikanth Sundarrajan commented on PIG-4667:
-------------------------------------------
Here is a quick update on this
It seems a bit tricky to get multiple jars passed to ExecutorLancher (Yarn-AM)
except for spark jar and an app jar. Am trying to seek help from the spark
community to see if there is a way to ship additional jars that needs to be on
AM class path. (There is a way to add files to dist cache, but they dont seem
to be added automatically to the classpath)
Easy workaround for this is to work with spark-assembly.jar, however the same
isn't available on maven repo for the pig code base to work with by default.
Am concurrently trying to see if it is possible to download the spark
dependencies and create a shaded jar to work with.
> Enable Pig on Spark to run on Yarn Client/Cluster mode
> ------------------------------------------------------
>
> Key: PIG-4667
> URL: https://issues.apache.org/jira/browse/PIG-4667
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: Srikanth Sundarrajan
> Assignee: Srikanth Sundarrajan
> Fix For: spark-branch
>
> Attachments: PIG-4667-logs.tgz
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
