i would like to change some defaults in spark without overriding the user if she/he wishes to change them.
for example currently spark.blacklist.enabled is by default false, which makes sense for backwards compatibility. i would like it to be by default true, but if the user provided --conf spark.spark.blacklist.enabled=false it should be set to false again. i am aware that i can do: val spark = SparkSession .builder() .appName("SomeExample") .config("spark.spark.blacklist.enabled", "true") .getOrCreate() however if i understand the order in which settings are applied correctly, this will override any setting the user provided with --conf, which is undesired. note that i do not have permission to change spark-defaults.conf because the spark install is also shared with some other applications for which i do not want to change the defaults. any suggestions? thanks