[spark] branch master updated: [SPARK-27690][SQL] Remove materialized views first in `HiveClientImpl.reset`

[dongjoon]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new fee695d  [SPARK-27690][SQL] Remove materialized views first in 
`HiveClientImpl.reset`
fee695d is described below

commit fee695d0cf211e4119c7df7a984708628dc9368a
Author: Yuming Wang <yumw...@ebay.com>
AuthorDate: Tue May 14 09:05:22 2019 -0700

    [SPARK-27690][SQL] Remove materialized views first in `HiveClientImpl.reset`
    
    ## What changes were proposed in this pull request?
    
    We should remove materialized view first otherwise(note that Hive 3.1 could 
reproduce this issue):
    ```scala
    Cause: 
org.apache.derby.shared.common.error.DerbySQLIntegrityConstraintViolationException:
 DELETE on table 'TBLS' caused a violation of foreign key constraint 
'MV_TABLES_USED_FK2' for key (4).  The statement has been rolled back.
    at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown 
Source)
    at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
    at 
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown 
Source)
    at 
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown 
Source)
    at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown 
Source)
    at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown 
Source)
    at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown 
Source)
    at 
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeBatchElement(Unknown 
Source)
    at org.apache.derby.impl.jdbc.EmbedStatement.executeLargeBatch(Unknown 
Source)
    ```
    
    ## How was this patch tested?
    
    Existing test
    
    Closes #24592 from wangyum/SPARK-27690.
    
    Authored-by: Yuming Wang <yumw...@ebay.com>
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
 .../org/apache/spark/sql/hive/client/HiveClientImpl.scala | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
index bad2110..eaca03c 100644
--- 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
+++ 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
@@ -879,9 +879,20 @@ private[hive] class HiveClientImpl(
   }
 
   def reset(): Unit = withHiveState {
-    client.getAllTables("default").asScala.foreach { t =>
+    val allTables = client.getAllTables("default")
+    val (mvs, others) = allTables.asScala.map(t => client.getTable("default", 
t))
+      .partition(_.getTableType.toString.equals("MATERIALIZED_VIEW"))
+
+    // Remove materialized view first, otherwise caused a violation of foreign 
key constraint.
+    mvs.foreach { table =>
+      val t = table.getTableName
+      logDebug(s"Deleting materialized view $t")
+      client.dropTable("default", t)
+    }
+
+    others.foreach { table =>
+      val t = table.getTableName
       logDebug(s"Deleting table $t")
-      val table = client.getTable("default", t)
       try {
         client.getIndexes("default", t, 255).asScala.foreach { index =>
           shim.dropIndex(client, "default", t, index.getIndexName)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org