Executor initialize before all resources are ready

Hi, I am building metrics system for Spark Streaming job, in the system, the metrics are collected in each executor, so a metrics source (a class used to collect metrics) needs to be initialized in each executor. The metrics source is packaged in a jar, when submitting a job, the jar is sent from

How to get the working directory in executor

I am using the following command to submit Spark job, I hope to send jar and config files to each executor and load it there spark-submit --verbose \ --files=/tmp/metrics.properties \ --jars /tmp/datainsights-metrics-source-aembly-1.0.jar \ --total-executor-cores 4\ --conf