Issue with SparkContext in cluster
I've created a spark app, which runs fine if I copy the corresponding
jar to the hadoop-server (where yarn is running) and submit it there.
If it try it to submit it from my local machine, I get the error which
I've attached below.
Submit cmd: spark-submit.cmd --class
ExamplesHadoop.SparkHbase.TruckEvents --master yarn-cluster
.\SparkHbase-1.0-SNAPSHOT-jar-with-dependencies.jar
Even after raising the time for the yarn log, I have the issue that
there is no log when I try to get it via yarn logs -applicationId
myApplicationId
Any hints how I could find the root cause of this issue?
Thanks,
Marco
15/01/28 10:25:06 INFO spark.SecurityManager: Changing modify acls to: user
15/01/28 10:25:06 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view
permissions: Set(user); users with modify permissions:
Set((user))
15/01/28 10:25:06 INFO yarn.Client: Submitting application 9 to ResourceManager
15/01/28 10:25:06 INFO impl.YarnClientImpl: Submitted application
application_1422368366192_0009
15/01/28 10:25:07 INFO yarn.Client: Application report for
application_1422368366192_0009 (state: ACCEPTED)
15/01/28 10:25:07 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1422437106550
final status: UNDEFINED
tracking URL: http://server:8088/proxy/application_1422368366192_0009/
user: root
15/01/28 10:25:08 INFO yarn.Client: Application report for
application_1422368366192_0009 (state: ACCEPTED)
15/01/28 10:25:09 INFO yarn.Client: Application report for
application_1422368366192_0009 (state: ACCEPTED)
15/01/28 10:25:10 INFO yarn.Client: Application report for
application_1422368366192_0009 (state: ACCEPTED)
15/01/28 10:25:11 INFO yarn.Client: Application report for
application_1422368366192_0009 (state: FAILED)
15/01/28 10:25:11 INFO yarn.Client:
client token: N/A
diagnostics: Application application_1422368366192_0009 failed 2
times due to AM Container for appattempt_1422368366192_0009_02
exited with exitCode: 1
For more detailed output, check application tracking
page:http://server:8088/proxy/application_1422368366192_0009/Then,
click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1422368366192_0009_02_01
Exit code: 1
Exception message:
/hadoop/yarn/local/usercache/root/appcache/application_1422368366192_0009/container_1422368366192_0009_02_01/launch_container.sh:
line 27: %PWD%;%PWD%/__spark__.jar;$H
ADOOP_CONF_DIR;/usr/hdp/current/hadoop-client/*;/usr/hdp/current/hadoop-client/lib/*;/usr/hdp/current/hadoop-hdfs-client/*;/usr/hdp/current/hadoop-hdfs-client/lib/*;/usr/hdp/current/hadoop-y
arn-client/*;/usr/hdp/current/hadoop-yarn-client/lib/*;$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/
hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/shar
e/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure;%PWD%/__app__.jar;%PWD%/*:
bad substitution
/bin/bash: line 0: fg: no job control
Stack trace: ExitCodeException exitCode=1:
/hadoop/yarn/local/usercache/root/appcache/application_1422368366192_0009/container_1422368366192_0009_02_01/launch_container.sh:
line 27: %PWD
%;%PWD%/__spark__.jar;$HADOOP_CONF_DIR;/usr/hdp/current/hadoop-client/*;/usr/hdp/current/hadoop-client/lib/*;/usr/hdp/current/hadoop-hdfs-client/*;/usr/hdp/current/hadoop-hdfs-client/lib/*;/
usr/hdp/current/hadoop-yarn-client/*;/usr/hdp/current/hadoop-yarn-client/lib/*;$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr
-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/
mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure;%PWD%/
__app__.jar;%PWD%/*: bad substitution
/bin/bash: line 0: fg: no job control
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at
Re: Issue with SparkContext in cluster
It's because you committed the job in Windows to a Hadoop cluster running
in Linux. Spark has not yet supported it. See
https://issues.apache.org/jira/browse/SPARK-1825
Best Regards,
Shixiong Zhu
2015-01-28 17:35 GMT+08:00 Marco marco@gmail.com:
I've created a spark app, which runs fine if I copy the corresponding
jar to the hadoop-server (where yarn is running) and submit it there.
If it try it to submit it from my local machine, I get the error which
I've attached below.
Submit cmd: spark-submit.cmd --class
ExamplesHadoop.SparkHbase.TruckEvents --master yarn-cluster
.\SparkHbase-1.0-SNAPSHOT-jar-with-dependencies.jar
Even after raising the time for the yarn log, I have the issue that
there is no log when I try to get it via yarn logs -applicationId
myApplicationId
Any hints how I could find the root cause of this issue?
Thanks,
Marco
15/01/28 10:25:06 INFO spark.SecurityManager: Changing modify acls to: user
15/01/28 10:25:06 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view
permissions: Set(user); users with modify permissions:
Set((user))
15/01/28 10:25:06 INFO yarn.Client: Submitting application 9 to
ResourceManager
15/01/28 10:25:06 INFO impl.YarnClientImpl: Submitted application
application_1422368366192_0009
15/01/28 10:25:07 INFO yarn.Client: Application report for
application_1422368366192_0009 (state: ACCEPTED)
15/01/28 10:25:07 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1422437106550
final status: UNDEFINED
tracking URL:
http://server:8088/proxy/application_1422368366192_0009/
user: root
15/01/28 10:25:08 INFO yarn.Client: Application report for
application_1422368366192_0009 (state: ACCEPTED)
15/01/28 10:25:09 INFO yarn.Client: Application report for
application_1422368366192_0009 (state: ACCEPTED)
15/01/28 10:25:10 INFO yarn.Client: Application report for
application_1422368366192_0009 (state: ACCEPTED)
15/01/28 10:25:11 INFO yarn.Client: Application report for
application_1422368366192_0009 (state: FAILED)
15/01/28 10:25:11 INFO yarn.Client:
client token: N/A
diagnostics: Application application_1422368366192_0009 failed 2
times due to AM Container for appattempt_1422368366192_0009_02
exited with exitCode: 1
For more detailed output, check application tracking
page:http://server:8088/proxy/application_1422368366192_0009/Then,
click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1422368366192_0009_02_01
Exit code: 1
Exception message:
/hadoop/yarn/local/usercache/root/appcache/application_1422368366192_0009/container_1422368366192_0009_02_01/launch_container.sh:
line 27: %PWD%;%PWD%/__spark__.jar;$H
ADOOP_CONF_DIR;/usr/hdp/current/hadoop-client/*;/usr/hdp/current/hadoop-client/lib/*;/usr/hdp/current/hadoop-hdfs-client/*;/usr/hdp/current/hadoop-hdfs-client/lib/*;/usr/hdp/current/hadoop-y
arn-client/*;/usr/hdp/current/hadoop-yarn-client/lib/*;$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/
hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/shar
e/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure;%PWD%/__app__.jar;%PWD%/*:
bad substitution
/bin/bash: line 0: fg: no job control
Stack trace: ExitCodeException exitCode=1:
/hadoop/yarn/local/usercache/root/appcache/application_1422368366192_0009/container_1422368366192_0009_02_01/launch_container.sh:
line 27: %PWD
%;%PWD%/__spark__.jar;$HADOOP_CONF_DIR;/usr/hdp/current/hadoop-client/*;/usr/hdp/current/hadoop-client/lib/*;/usr/hdp/current/hadoop-hdfs-client/*;/usr/hdp/current/hadoop-hdfs-client/lib/*;/
usr/hdp/current/hadoop-yarn-client/*;/usr/hdp/current/hadoop-yarn-client/lib/*;$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr
-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/
mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure;%PWD%/
__app__.jar;%PWD%/*: bad substitution
/bin/bash: line 0: fg: no job control
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at
