Hi Eric (or rather: anyone who's experiencing similar situation),
I think your problem was, that the /--files/ parameter was provided after
the application jar. Your command should have looked like this, instead:
./bin/spark-submit --class edu.bjut.spark.SparkPageRank --master
yarn-cluster
Thanks for your reply!
Sorry for i forgot referring the spark which i'm using is *Version 1.0.2*
instead of 1.1.0.
Also the document of 1.0.2 seems not same like 1.1.0:
http://spark.apache.org/docs/1.0.2/running-on-yarn.html
And i tried your suggestion(upload ) but did not work:
*1. set my
Hi,
I want to check the DEBUG log of spark executor on YARN(using yarn-cluster
mode), but
1. yarn daemonlog setlevel DEBUG YarnChild.class
2. set log4j.properties in spark/conf folder on client node.
no means above works.
So how could i set the log level of spark executor* on YARN container
Hi Eric,
Check the Debugging Your Application section at:
http://spark.apache.org/docs/latest/running-on-yarn.html
Long story short: upload your log4j.properties using the --files
argument of spark-submit.
(Mental note: we could make the log level configurable via a system property...)
On
