[ https://issues.apache.org/jira/browse/SPARK-27520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836730#comment-16836730 ]
Thomas Graves commented on SPARK-27520: --------------------------------------- Can we add more to the description to explain why we are doing this? I'm assuming this is to allow users to more easily change it. > Introduce a global config system to replace hadoopConfiguration > --------------------------------------------------------------- > > Key: SPARK-27520 > URL: https://issues.apache.org/jira/browse/SPARK-27520 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 3.0.0 > Reporter: Xingbo Jiang > Priority: Major > > hadoopConf can be accessed via `SparkContext.hadoopConfiguration` from both > user code and Spark internal. The configuration is mainly used to read files > from hadoop-supported file system(eg. get URI/get FileSystem/add security > credentials/get metastore connect url/etc.) > We shall keep a global config that users can set and use that to track the > hadoop configurations, and avoid using `SparkContext.hadoopConfiguration`, > maybe we shall mark it as deprecate. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org