Hi,
We have some code which worked with Spark 1.5.0, which allowed us to pass
system properties to the executors:
SparkConf sparkConf = new SparkConf().setAppName(appName);
...
for each property:
sparkConf.setExecutorEnv(propName, propValue);
Our current Spark Streaming driver is compiled with spark-streaming_2.11 :
2.0.0.
We no longer get the properties on the executor side. Tried the following
approaches and none of these worked:
sparkConf.set("spark.MyPropName", "MyPropValue");
sparkConf.setExecutorEnv("MyProp1", "MyPropVal1");
sparkConf.setExecutorEnv("SPARK_JAVA_OPTS", "-Daaa=bbb");
On the executor side, I'm retrieving with System.getenv; fall back on
System.getProperty() if the property ends up being in the properties rather
than the environment.
The only way I was able to get this to work is by defining a custom Spark
conf file, with every property name being prefixed with "spark." and then
passing the conf properties file via the --properties-file <location> to
spark-submit.
Is anyone aware of a change that may have caused setExecutorEnv to work
differently in 2.0.0?
Is there any other way to pass environment variables or system properties
to the executor side, preferably a programmatic way rather than
configuration-wise?
Thanks,
- Dmitry
