Is it able to run on local mode ?
what mean?? standalone mode ?
------------------ ???????? ------------------
??????: "Yash Sharma";<yash...@gmail.com>;
????????: 2016??6??22??(??????) ????2:18
??????: "Saisai Shao"<sai.sai.s...@gmail.com>;
????: "????????"<958943...@qq.com>; "user"<user@spark.apache.org>;
????: Re: Could not find or load main class
org.apache.spark.deploy.yarn.ExecutorLauncher
Try providing the jar with the hdfs prefix. Its probably just because its not
able to find the jar on all nodes.
hdfs://master:9000/user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar
Is it able to run on local mode ?
On Wed, Jun 22, 2016 at 4:14 PM, Saisai Shao <sai.sai.s...@gmail.com> wrote:
spark.yarn.jar(none)The location of the Spark jar file, in case overriding the
default location is desired. By default, Spark on YARN will use a Spark jar
installed locally, but the Spark jar can also be in a world-readable location
on HDFS. This allows YARN to cache it on nodes so that it doesn't need to be
distributed each time an application runs. To point to a jar on HDFS, for
example, set this configuration to hdfs:///some/path.
spark.yarn.jar is used for spark run-time system jar, which is spark assembly
jar, not the application jar (example-assembly jar). So in your case you upload
the example-assembly jar into hdfs, in which spark system jars are not packed,
so ExecutorLaucher cannot be found.
Thanks
Saisai
On Wed, Jun 22, 2016 at 2:10 PM, ???????? <958943...@qq.com> wrote:
shihj@master:/usr/local/spark/spark-1.6.1-bin-hadoop2.6$ ./bin/spark-submit
--class org.apache.spark.examples.SparkPi --master yarn-client --driver-memory
512m --num-executors 2 --executor-memory 512m --executor-cores 2
/user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar 10
Warning: Local jar /user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar
does not exist, skipping.
java.lang.ClassNotFoundException: org.apache.spark.examples.SparkPi
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:174)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:689)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
get error at once
------------------ ???????? ------------------
??????: "Yash Sharma";<yash...@gmail.com>;
????????: 2016??6??22??(??????) ????2:04
??????: "????????"<958943...@qq.com>;
????: "user"<user@spark.apache.org>;
????: Re: Could not find or load main class
org.apache.spark.deploy.yarn.ExecutorLauncher
How about supplying the jar directly in spark submit -
./bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn-client \
--driver-memory 512m \
--num-executors 2 \
--executor-memory 512m \
--executor-cores 2 \
/user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar
On Wed, Jun 22, 2016 at 3:59 PM, ???????? <958943...@qq.com> wrote:
i config this para at spark-defaults.conf
spark.yarn.jar
hdfs://master:9000/user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar
then ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master
yarn-client --driver-memory 512m --num-executors 2 --executor-memory 512m
--executor-cores 2 10:
Error: Could not find or load main class
org.apache.spark.deploy.yarn.ExecutorLauncher
but i don't config that para ,there no error why???that para is only avoid
Uploading resource file(jar package)??
