Re: [K8S] properties file via SPARK_CONF_DIR and --properties-file prevents definition of own properties via secrets/own mounts

I would like to discuss if we can introduce a change here - maybe it’s also possible to get that into 2.4.4? I’m keen to contribute that change. Please let me know your thoughts on this. King Regards Roland Johann Software Developer/Data Engineer phenetic GmbH Lütticher Straße 10, 50674 Köln,

[K8S] properties file via SPARK_CONF_DIR and --properties-file prevents definition of own properties via secrets/own mounts

Hi all, K8S resource manager dumps the config map to /opt/spark/conf/spark-defaults.conf and passes it to spark submit twice: • via env var SPARK_CONF_DIR=/opt/spark/conf/ • argument --properties-file /opt/spark/conf/spark-defaults.conf This prevents definition of user defined