[
https://issues.apache.org/jira/browse/PIG-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737394#comment-14737394
]
Xuefu Zhang commented on PIG-4667:
----------------------------------
[~sriksun], jar problem is indeed very complicated. I'm a little surprised that
Spark doesn't ship the dependent jars to the AM because that's something
required in any case. Regardless, I'm wondering:
1. would spark-assembly.jar help in this case? If so, it's fine to require this
jar to be in Pig /lib directory.
2. SPARK_HOME is also fine if that makes thing easy. However, I'm not sure what
you mean by wiring up bin/pig to the spark jars, which sounds a little
complicated.
Our spark expert, [~vanzin], knows very well in this area. In fact, he helped
Hive on Spark on Yarn, which was much complicated. I'm sure he has better ideas
on this. Could you please share your thought on this, [~vanzin]? Thanks.
> Enable Pig on Spark to run on Yarn Client/Cluster mode
> ------------------------------------------------------
>
> Key: PIG-4667
> URL: https://issues.apache.org/jira/browse/PIG-4667
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: Srikanth Sundarrajan
> Assignee: Srikanth Sundarrajan
> Fix For: spark-branch
>
> Attachments: PIG-4667-logs.tgz
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
