Hi all,
K8S resource manager dumps the config map to
/opt/spark/conf/spark-defaults.conf and passes it to spark submit twice:
• via env var SPARK_CONF_DIR=/opt/spark/conf/
• argument --properties-file /opt/spark/conf/spark-defaults.conf
This prevents definition of user defined properties, such as from secrets -
currently it seems to be the only possibility to define spark config properties
from k8s secrets.
Is the current implementation intent? If yes, what are the reasons behind that
decision?
Thanks and kind regards
Roland Johann
Software Developer/Data Engineer
phenetic GmbH
Lütticher Straße 10, 50674 Köln, Germany
Mobil: +49 172 365 26 46
Mail: [email protected]
Web: phenetic.io
Handelsregister: Amtsgericht Köln (HRB 92595)
Geschäftsführer: Roland Johann, Uwe Reimann
