liyunzhang_intel created PIG-5246: ------------------------------------- Summary: Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2 Key: PIG-5246 URL: https://issues.apache.org/jira/browse/PIG-5246 Project: Pig Issue Type: Bug Reporter: liyunzhang_intel Assignee: liyunzhang_intel
in bin/pig. we copy assembly jar to pig's classpath in spark1.6. {code} # For spark mode: # Please specify SPARK_HOME first so that we can locate $SPARK_HOME/lib/spark-assembly*.jar, # we will add spark-assembly*.jar to the classpath. if [ "$isSparkMode" == "true" ]; then if [ -z "$SPARK_HOME" ]; then echo "Error: SPARK_HOME is not set!" exit 1 fi # Please specify SPARK_JAR which is the hdfs path of spark-assembly*.jar to allow YARN to cache spark-assembly*.jar on nodes so that it doesn't need to be distributed each time an application runs. if [ -z "$SPARK_JAR" ]; then echo "Error: SPARK_JAR is not set, SPARK_JAR stands for the hdfs location of spark-assembly*.jar. This allows YARN to cache spark-assembly*.jar on nodes so that it doesn't need to be distributed each time an application runs." exit 1 fi if [ -n "$SPARK_HOME" ]; then echo "Using Spark Home: " ${SPARK_HOME} SPARK_ASSEMBLY_JAR=`ls ${SPARK_HOME}/lib/spark-assembly*` CLASSPATH=${CLASSPATH}:$SPARK_ASSEMBLY_JAR fi fi {code} after upgrade to spark2.0, we may modify it -- This message was sent by Atlassian JIRA (v6.3.15#6346)