Can you place metrics.properties and datainsights-metrics-source-assembly-1.0.jar on hdfs ?
Cheers On Wed, Jan 13, 2016 at 8:01 AM, Byron Wang <open...@gmail.com> wrote: > I am using the following command to submit Spark job, I hope to send jar > and > config files to each executor and load it there > > spark-submit --verbose \ > --files=/tmp/metrics.properties \ > --jars /tmp/datainsights-metrics-source-aembly-1.0.jar \ > --total-executor-cores 4\ > --conf "spark.metrics.conf=metrics.properties" \ > --conf > > "spark.executor.extraClassPath=datainsights-metrics-source-assembly-1.0.jar" > \ > --class org.microsoft.ofe.datainsights.StartServiceSignalPipeline \ > ./target/datainsights-1.0-jar-with-dependencies.jar > > --files and --jars is used to send files to executors, I found that the > files are sent to the working directory of executor like > 'worker/app-xxxxx-xxxx/0/ > > But when job is running, the executor always throws exception saying that > it > could not find the file 'metrics.properties'or the class which is contained > in 'datainsights-metrics-source-assembly-1.0.jar'. It seems that the job is > looking for files under another dir other than working directory. > > Do you know how to load the file which is sent to executors? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/How-to-get-the-working-directory-in-executor-tp25962.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
