[
https://issues.apache.org/jira/browse/PIG-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liyunzhang_intel updated PIG-4059:
----------------------------------
Description:
Setting up your development environment:
0. download spark release package(currently pig on spark only support spark
1.6).
1. Check out Pig Spark branch.
2. Build Pig by running "ant jar" and "ant -Dhadoopversion=23 jar" for
hadoop-2.x versions
3. Configure these environmental variables:
export HADOOP_USER_CLASSPATH_FIRST="true"
Now we support “local” and "yarn-client" mode, you can export system variable
“SPARK_MASTER” like:
export SPARK_MASTER=local or export SPARK_MASTER="yarn-client"
4. In local mode: ./pig -x spark_local xxx.pig
In yarn-client mode:
export SPARK_HOME=xx;
export SPARK_JAR=hdfs://example.com:8020/xxxx (the hdfs location where you
upload the spark-assembly*.jar)
./pig -x spark xxx.pig
was:
Setting up your development environment:
1. Check out Pig Spark branch.
2. Build Pig by running "ant jar" and "ant -Dhadoopversion=23 jar" for
hadoop-2.x versions
3. Configure these environmental variables:
export HADOOP_USER_CLASSPATH_FIRST="true"
Now we support “local” and "yarn-client" mode, you can export system variable
“SPARK_MASTER” like:
export SPARK_MASTER=local or export SPARK_MASTER="yarn-client"
4. In local mode: ./pig -x spark_local xxx.pig
In yarn-client mode:
export SPARK_HOME=xx;
export SPARK_JAR=hdfs://example.com:8020/xxxx (the hdfs location where you
upload the spark-assembly*.jar)
./pig -x spark xxx.pig
> Pig on Spark
> ------------
>
> Key: PIG-4059
> URL: https://issues.apache.org/jira/browse/PIG-4059
> Project: Pig
> Issue Type: New Feature
> Components: spark
> Reporter: Rohini Palaniswamy
> Assignee: Praveen Rachabattuni
> Labels: spork
> Fix For: spark-branch
>
> Attachments: Pig-on-Spark-Design-Doc.pdf, Pig-on-Spark-Scope.pdf
>
>
> Setting up your development environment:
> 0. download spark release package(currently pig on spark only support spark
> 1.6).
> 1. Check out Pig Spark branch.
> 2. Build Pig by running "ant jar" and "ant -Dhadoopversion=23 jar" for
> hadoop-2.x versions
> 3. Configure these environmental variables:
> export HADOOP_USER_CLASSPATH_FIRST="true"
> Now we support “local” and "yarn-client" mode, you can export system variable
> “SPARK_MASTER” like:
> export SPARK_MASTER=local or export SPARK_MASTER="yarn-client"
> 4. In local mode: ./pig -x spark_local xxx.pig
> In yarn-client mode:
> export SPARK_HOME=xx;
> export SPARK_JAR=hdfs://example.com:8020/xxxx (the hdfs location where
> you upload the spark-assembly*.jar)
> ./pig -x spark xxx.pig
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
