YE created SPARK-43454: -------------------------- Summary: SparkConf's get and getAllWithPrefix support variable substitution Key: SPARK-43454 URL: https://issues.apache.org/jira/browse/SPARK-43454 Project: Spark Issue Type: Improvement Components: Kubernetes, Spark Core Affects Versions: 3.4.0, 3.3.2, 3.2.4 Reporter: YE
After SPARK-16272: variable substitution is available for `ConfigEntry`s, this creates some confusion for get conf directly vs get conf from ConfigEntry. For example: {code:java} // suppose spark.a=${env:var_a} is set in sparkConf, and it's defined as a ConfigEntry as CONFIG_ENTRY_A // this would return ${env:var_a} val a = sparkConf.get("spark.a") // this would return the value of var_a in the current JVM process val configA = sparkConf.get(CONFIG_ENTRY_A) {code} I'd like to propose that we should support variable substitution for get config directly. I noticed this problem when running spark applications on k8s, and noticed some substitutions are not work as expected. For k8s particularly, it uses getAllWithPrefix extensively, which also suffers this problem -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org