Hi Tom,
I'm on Hadoop 2.05. I can launch application spark 0.8 release
normally. However I switch to git master branch version with application
built with it, I got the jar not found exception and same happens to the
example application. I have tried both file:// protocol and hdfs://
protocol with jar in local file system and hdfs respectively, and even
tried jar list parameter when new spark context. The exception is slightly
different for hdfs protocol and local file path. My application launch
command is
SPARK_JAR=/home/work/guojiacheng/spark/assembly/target/scala-2.9.3/spark-assembly-0.9.0-incubating-SNAPSHOT-hadoop2.0.5-alpha.jar
/home/work/guojiacheng/spark/spark-class
org.apache.spark.deploy.yarn.Client --jar
/home/work/guojiacheng/spark-auc/target/scala-2.9.3/SparkAUC-assembly-0.1.jar
--class myClass.SparkAUC --args -c --args yarn-standalone --args -i
--args hdfs://{hdfs_host}:9000/user/work/guojiacheng/data --args -m --args
hdfs://{hdfs_host}:9000/user/work/guojiacheng/model_large --args -o --args
hdfs://{hdfs_host}:9000/user/work/guojiacheng/score --num-workers 60
--master-memory 6g --worker-memory 7g --worker-cores 1
And my build command is SPARK_HADOOP_VERSION=2.0.5-alpha SPARK_YARN=true
sbt/sbt assembly
Only thing I can think of might be related is on each cluster node, it has
a env SPARK_HOME point to a copy of 0.8 version's position, and its bin
fold is in Path environment variable. And 0.9 version is not there. It was
something left over, when cluster was setup. But I don't know whether it
is related, as my understand is the yarn version try to distribute spark
through yarn.
hdfs version error message:
appDiagnostics: Application application_1384588058297_0056 failed
1 times due to AM Container for appattempt_1384588058297_0056_000001 exited
with exitCode: -1000 due to: RemoteTrace:
java.io.FileNotFoundException: File
file:/home/work/.sparkStaging/application_1384588058297_0056/SparkAUC-assembly-0.1.jar
does not exist
local version error message.
appDiagnostics: Application application_1384588058297_0066 failed 1 times
due to AM Container for appattempt_1384588058297_0066_000001 exited with
exitCode: -1000 due to: java.io.FileNotFoundException: File
file:/home/work/guojiacheng/spark/assembly/target/scala-2.9.3/spark-assembly-0.9.0-incubating-SNAPSHOT-hadoop2.0.5-alpha.jar
does not exist
Best Regards,
Jiacheng GUo
On Mon, Nov 18, 2013 at 10:34 PM, Tom Graves <tgraves...@yahoo.com> wrote:
> Hey Jiacheng Guo,
>
> do you have SPARK_EXAMPLES_JAR env variable set? If you do, you have to
> add the --addJars parameter to the yarn client and point to the spark
> examples jar. Or just unset SPARK_EXAMPLES_JAR env variable.
>
> You should only have to set SPARK_JAR env variable.
>
> If that isn't the issue let me know the build command you used and hadoop
> version, and your defaultFs or hadoop.
>
> Tom
>
>
> On Saturday, November 16, 2013 2:32 AM, guojc <guoj...@gmail.com> wrote:
> hi,
> After reading about the exiting progress in consolidating shuffle, I'm
> eager to trying out the last master branch. However up to launch the
> example application, the job failed with prompt the app master failed to
> find the target jar. appDiagnostics: Application
> application_1384588058297_0017 failed 1 times due to AM Container for
> appattempt_1384588058297_0017_000001 exited with exitCode: -1000 due to:
> java.io.FileNotFoundException: File
> file:/${my_work_dir}/spark/examples/target/scala-2.9.3/spark-examples-assembly-0.9.0-incubating-SNAPSHOT.jar
> does not exist.
>
> Is there any change on how to launch a yarn job now?
>
> Best Regards,
> Jiacheng Guo
>
>
>
>
