[
https://issues.apache.org/jira/browse/PIG-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15021703#comment-15021703
]
liyunzhang_intel commented on PIG-4741:
---------------------------------------
[~sriksun]: thanks for your detailed explanation. You are right. the variable
{{PWD}} in the SPARK_DIST_CLASSPATH will be translated to the container working
directory when running in yarn-client mode. So we need have '\' to escape
substition. Another question about PIG-4667: now we can run pig on spark on
yarn-client mode, can we make it run on yarn-cluster mode? I try to run on
yarn-cluster mode, but following error happens:
{noformat}
2015-11-23 11:23:01,239 [main] ERROR org.apache.spark.SparkContext - Error
initializing SparkContext.
199 org.apache.spark.SparkException: Detected yarn-cluster mode, but isn't
running on a cluster. Deployment to YARN is not supported directly by
SparkContext. Please use spark-submit.
{noformat}
> the value of $SPARK_DIST_CLASSPATH in pig file is invalid
> ---------------------------------------------------------
>
> Key: PIG-4741
> URL: https://issues.apache.org/jira/browse/PIG-4741
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-4741.patch
>
>
> the value of
> [$SPARK_DIST_CLASSPATH|https://github.com/apache/pig/blob/spark/bin/pig#L380]
> in bin/pig is invalid
> {code}
> SPARK_DIST_CLASSPATH=${SPARK_DIST_CLASSPATH}:\${PWD}/`basename $f`
> {code}
> there is no need to escape the {{PWD}}. If we add "\", the value of
> SPARK_DIST_CLASSPATH will like:
> {code}
>
> ${PWD}/akka-actor_2.10-2.3.4-spark.jar:${PWD}/akka-remote_2.10-2.3.4-spark.jar
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
