I am using the following command to submit Spark job, I hope to send jar and config files to each executor and load it there
spark-submit --verbose \ --files=/tmp/metrics.properties \ --jars /tmp/datainsights-metrics-source-aembly-1.0.jar \ --total-executor-cores 4\ --conf "spark.metrics.conf=metrics.properties" \ --conf "spark.executor.extraClassPath=datainsights-metrics-source-assembly-1.0.jar" \ --class org.microsoft.ofe.datainsights.StartServiceSignalPipeline \ ./target/datainsights-1.0-jar-with-dependencies.jar --files and --jars is used to send files to executors, I found that the files are sent to the working directory of executor like 'worker/app-xxxxx-xxxx/0/ But when job is running, the executor always throws exception saying that it could not find the file 'metrics.properties'or the class which is contained in 'datainsights-metrics-source-assembly-1.0.jar'. It seems that the job is looking for files under another dir other than working directory. Do you know how to load the file which is sent to executors? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-get-the-working-directory-in-executor-tp25962.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
