[ https://issues.apache.org/jira/browse/SPARK-36766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-36766. ---------------------------------- Resolution: Invalid > Spark SQL DDL does not recognize fs.s3.impl implied filesystem in LOCATION tag > ------------------------------------------------------------------------------ > > Key: SPARK-36766 > URL: https://issues.apache.org/jira/browse/SPARK-36766 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.0.2 > Reporter: Sungwon > Priority: Major > > CREATE TABLE with the LOCATION tag via SPARK SQL DDL does not seem to > recognize the fileSystem implied by setting fs.s3.impl. > The environment setting is set to: > |spark.hadoop.fs.s3.impl|org.apache.hadoop.fs.s3a.S3AFileSystem| > The following spark.sql ddl raises an exception: > spark.sql("""CREATE TABLE local.db.table ( > col1 string, > col2 string, > ) > LOCATION 's3://bucket/prefix' > """) > Exception: > Py4JJavaError: An error occurred while calling o86.sql. : > org.apache.iceberg.exceptions.CommitStateUnknownException: Got exception: > org.apache.hadoop.fs.UnsupportedFileSystemException No FileSystem for scheme > "s3" > This is inconsistent with other spark functions such as > spark.read.parquet("s3://bucket/prefix") which succeeds with the same > configuration -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org