Github user ericl commented on the issue:
https://github.com/apache/spark/pull/15515
Ok, I think this is now ready to review. I've pushed commits that add a
feature flag (consolidated with prior feature flags to make it easy to revert
back to 2.0 behavior), backwards compatibility, and tests cases.
The proposal for backwards compatibility is as follows.
- Newly created datasource tables will have the table property
`partitionProvider=hive` and will use the metastore for partition discovery.
- Older Spark versions will not respect the partitionProvider property, and
will always discover partitions from remote storage.
- When partitionProvider is set to hive, partition related hive DDLs (e.g.
SHOW PARTITIONS) will work. Otherwise, an exception will be thrown telling the
user to run `msck repair table` to import the table partitions into the hive
metastore (this will set partitionProvider = hive).
- When the feature flag is disabled, tables with hive partition provider
will behave like legacy tables. The partition DDLs will also be disabled for
all datasource tables.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
