[ https://issues.apache.org/jira/browse/SPARK-31935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17147873#comment-17147873 ]
Apache Spark commented on SPARK-31935: -------------------------------------- User 'cloud-fan' has created a pull request for this issue: https://github.com/apache/spark/pull/28948 > Hadoop file system config should be effective in data source options > --------------------------------------------------------------------- > > Key: SPARK-31935 > URL: https://issues.apache.org/jira/browse/SPARK-31935 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 2.4.7, 3.0.1, 3.1.0 > Reporter: Gengliang Wang > Assignee: Gengliang Wang > Priority: Major > Fix For: 3.0.1, 3.1.0 > > > Data source options should be propagated into the hadoop configuration of > method `checkAndGlobPathIfNecessary` > From org.apache.hadoop.fs.FileSystem.java: > {code:java} > public static FileSystem get(URI uri, Configuration conf) throws > IOException { > String scheme = uri.getScheme(); > String authority = uri.getAuthority(); > if (scheme == null && authority == null) { // use default FS > return get(conf); > } > if (scheme != null && authority == null) { // no authority > URI defaultUri = getDefaultUri(conf); > if (scheme.equals(defaultUri.getScheme()) // if scheme matches > default > && defaultUri.getAuthority() != null) { // & default has authority > return get(defaultUri, conf); // return default > } > } > > String disableCacheName = String.format("fs.%s.impl.disable.cache", > scheme); > if (conf.getBoolean(disableCacheName, false)) { > return createFileSystem(uri, conf); > } > return CACHE.get(uri, conf); > } > {code} > With this, we can specify URI schema and authority related configurations for > scanning file systems. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org