[
https://issues.apache.org/jira/browse/PIG-4233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Praveen Rachabattuni updated PIG-4233:
--------------------------------------
Description:
Currently we have a fat jar created in legacy directory which contains pig
along with dependencies.
Would need to modify build.xml to add spark dependency jars to include in
legacy fat jar.
Running job on Spark cluster:
1. export SPARK_HOME=/path/to/spark
2. export
SPARK_PIG_JAR=$PIG_HOME/legacy/pig-0.14.0-SNAPSHOT-withouthadoop-h1.jar
3. export SPARK_MASTER=spark://localhost:7077
4 export HADOOP_HOME=/path/to/hadoop
5. Launch the job using ./bin/pig -x spark
was:
Currently we have a fat jar created in legacy directory which contains pig
along with dependencies.
Would need to modify build.xml to add spark dependency jars to include in
legacy fat jar.
> Package pig along with dependencies into a fat jar while job submission to
> Spark cluster
> ----------------------------------------------------------------------------------------
>
> Key: PIG-4233
> URL: https://issues.apache.org/jira/browse/PIG-4233
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: Praveen Rachabattuni
> Assignee: Praveen Rachabattuni
> Attachments: PIG-4233.patch
>
>
> Currently we have a fat jar created in legacy directory which contains pig
> along with dependencies.
> Would need to modify build.xml to add spark dependency jars to include in
> legacy fat jar.
> Running job on Spark cluster:
> 1. export SPARK_HOME=/path/to/spark
> 2. export
> SPARK_PIG_JAR=$PIG_HOME/legacy/pig-0.14.0-SNAPSHOT-withouthadoop-h1.jar
> 3. export SPARK_MASTER=spark://localhost:7077
> 4 export HADOOP_HOME=/path/to/hadoop
> 5. Launch the job using ./bin/pig -x spark
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
