[ https://issues.apache.org/jira/browse/SPARK-18543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiao Li updated SPARK-18543: ---------------------------- Description: When the mode is OVERWRITE, we drop the Hive serde tables and create a data source table. This is not right. {code} val tableName = "tab1" withTable(tableName) { sql(s"CREATE TABLE $tableName STORED AS SEQUENCEFILE AS SELECT 1 AS key, 'abc' AS value") val df = sql(s"SELECT key, value FROM $tableName") df.write.mode(SaveMode.Overwrite).saveAsTable(tableName) val tableMeta = spark.sessionState.catalog.getTableMetadata(TableIdentifier(tableName)) assert(tableMeta.provider == Some(spark.sessionState.conf.defaultDataSourceName)) } {code} Based on the definition of OVERWRITE, no change should be made on the table definition. When recreate the table, we need to create a Hive serde table. was: When the mode is OVERWRITE, we drop the Hive serde tables and create a data source table. This is not right. Based on the definition of OVERWRITE, no change should be made on the table definition. When recreate the table, we need to create a Hive serde table. {code} val tableName = "tab1" withTable(tableName) { sql(s"CREATE TABLE $tableName STORED AS SEQUENCEFILE AS SELECT 1 AS key, 'abc' AS value") val df = sql(s"SELECT key, value FROM $tableName") df.write.mode(SaveMode.Overwrite).saveAsTable(tableName) val tableMeta = spark.sessionState.catalog.getTableMetadata(TableIdentifier(tableName)) assert(tableMeta.provider == Some(spark.sessionState.conf.defaultDataSourceName)) } {code} > SaveAsTable(CTAS) using overwrite could change table definition > --------------------------------------------------------------- > > Key: SPARK-18543 > URL: https://issues.apache.org/jira/browse/SPARK-18543 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.6.3, 2.0.2 > Reporter: Xiao Li > Assignee: Xiao Li > > When the mode is OVERWRITE, we drop the Hive serde tables and create a data > source table. This is not right. > {code} > val tableName = "tab1" > withTable(tableName) { > sql(s"CREATE TABLE $tableName STORED AS SEQUENCEFILE AS SELECT 1 AS > key, 'abc' AS value") > val df = sql(s"SELECT key, value FROM $tableName") > df.write.mode(SaveMode.Overwrite).saveAsTable(tableName) > val tableMeta = > spark.sessionState.catalog.getTableMetadata(TableIdentifier(tableName)) > assert(tableMeta.provider == > Some(spark.sessionState.conf.defaultDataSourceName)) > } > {code} > Based on the definition of OVERWRITE, no change should be made on the table > definition. When recreate the table, we need to create a Hive serde table. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org