[ https://issues.apache.org/jira/browse/SPARK-22306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16210105#comment-16210105 ]
Dongjoon Hyun commented on SPARK-22306: --------------------------------------- Hi, [~cloud_fan] and [~smilegator]. Should we change the default value to prevent this regression? > INFER_AND_SAVE overwrites important metadata in Metastore > --------------------------------------------------------- > > Key: SPARK-22306 > URL: https://issues.apache.org/jira/browse/SPARK-22306 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.2.0 > Environment: Hive 2.3.0 (PostgresQL metastore) > Spark 2.2.0 > Reporter: David Malinge > > I noticed some critical changes on my hive tables and realized that they were > caused by a simple select on SparkSQL. Looking at the logs, I found out that > this select was actually performing an update on the database "Saving > case-sensitive schema for table". > I then found out that Spark 2.2.0 introduces a new default value for > spark.sql.hive.caseSensitiveInferenceMode (see SPARK-20888): INFER_AND_SAVE > The issue is that this update changes critical metadata of the table, in > particular: > - changes the owner to the current user > - removes bucketing metadata (BUCKETING_COLS, SDS) > - removes sorting metadata (SORT_COLS) > Switching the property to: NEVER_INFER prevents the issue. > Also, note that the damage can be fix manually in Hive with e.g.: > {code:sql} > alter table [table_name] > clustered by ([col1], [col2]) > sorted by ([colA], [colB]) > into [n] buckets > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org