Github user budde commented on the issue:
https://github.com/apache/spark/pull/16797
I'll double check, but I don't think
```spark.sql.hive.manageFilesourcePartitions=false``` would solve this issue
since we're still deriving the file relation's dataSchema parameter from the
schema of MetastoreRelation. The call to ```fileFormat.inferSchema()``` has
been removed entirely.
If Spark SQL is set on using a table property to store the case-sesnitive
schema then I think having a way to backfill this property for existing < 2.1
tables as well as tables not created or managed by Spark will be a necessity.
If the cleanest way to deal with this case sensitivity problem is to bring back
schema inference then I think a good option would be to introduce a
configuration param to indicate whether or not an inferred schema should be
written back to the table as a property.
We could also introduce another config param that allows a user to bypass
schema inference even if a case-sensitive schema can't be read from the table
properties. This could be helpful for users who would like to query external
Hive tables that aren't managed by Spark and that they know aren't backed by
files containing case-sensitive field names.
This would basically allow us to support the following use cases:
1) The MetastoreRelation is able to read a case-sensitive schema from the
table properties. No inference is necessary.
2) The MetastoreRelation can't read a case-sensitive schema from the table
properties. A case-sensitive schema is inferred and, if configured, written
back as a table property.
3) The MetastoreRelation can't read a case-sensitive schema from the table
properties. The user knows the underlying data files don't contain
case-sensitive field names and has explicitly set a config param to skip the
inference step.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
