[GitHub] spark pull request #14801: [SPARK-17234] [SQL] Table Existence Checking when...
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/14801 [SPARK-17234] [SQL] Table Existence Checking when Index Table with the Same Name Exists ### What changes were proposed in this pull request? Hive Index tables are not supported by Spark SQL. Thus, we issue an exception when users try to access Hive Index tables. When the internal function `tableExists` tries to access Hive Index tables, it always gets the same error message: ```Hive index table is not supported```. This message could be confusing to users, since their SQL operations could be completely unrelated to Hive Index tables. For example, when users try to alter a table to a new name and there exists an index table with the same name, the expected exception should be a `TableAlreadyExistsException`. This PR made the following changes: - Introduced a new `AnalysisException` type: `SQLFeatureNotSupportedException`. When users try to access an `Index Table`, we will issue a `SQLFeatureNotSupportedException`. - `tableExists` returns `true` when hitting a `SQLFeatureNotSupportedException` and the feature is `Hive index table`. - Add a checking `requireTableNotExists` for `SessionCatalog`'s `createTable` API; otherwise, the current implementation relies on the Hive's internal checking. ### How was this patch tested? Added a test case You can merge this pull request into a Git repository by running: $ git pull https://github.com/gatorsmile/spark tableExists Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14801.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14801 commit 1af428b68c4341192bf8f66af7c434a7b89be61d Author: gatorsmile Date: 2016-08-25T06:26:00Z fix commit 664d6f1caa9b3d62eafbddb292991def722910ae Author: gatorsmile Date: 2016-08-25T06:34:16Z improve test cases commit c400c5292a32549cea80861adfaefeb41f4d90b3 Author: gatorsmile Date: 2016-08-25T07:12:57Z fix --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14801: [SPARK-17234] [SQL] Table Existence Checking when...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/14801#discussion_r76565215 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala --- @@ -450,7 +455,11 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, hadoopConf: Configurat } override def tableExists(db: String, table: String): Boolean = withClient { -client.getTableOption(db, table).isDefined +try { + client.getTableOption(db, table).isDefined +} catch { + case e: AnalysisException if e.message.contains("Hive index table is not supported") => true --- End diff -- this looks hacky. Actually why we use `getTableOption` to implement the semantic of `tableExists`? I'd like to make `HiveClient` provide the `tableExists` API directly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14801: [SPARK-17234] [SQL] Table Existence Checking when...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/14801#discussion_r76663011 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala --- @@ -450,7 +455,11 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, hadoopConf: Configurat } override def tableExists(db: String, table: String): Boolean = withClient { -client.getTableOption(db, table).isDefined +try { + client.getTableOption(db, table).isDefined +} catch { + case e: AnalysisException if e.message.contains("Hive index table is not supported") => true --- End diff -- Yeah, that is a good idea. Let me add it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14801: [SPARK-17234] [SQL] Table Existence Checking when...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/14801#discussion_r76719507 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala --- @@ -218,6 +218,12 @@ class VersionsSuite extends SparkFunSuite with Logging { holdDDLTime = false) } +test(s"$version: tableExists") { + // No exception should be thrown --- End diff -- hive's `tableExists` may throw exception? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14801: [SPARK-17234] [SQL] Table Existence Checking when...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/14801#discussion_r76719642 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala --- @@ -637,6 +639,36 @@ class HiveDDLSuite } } + test("create table with the same name as an index table") { +val tabName = "tab1" +val indexName = tabName + "_index" +val indexTabName = "default__tab1_tab1_index__" --- End diff -- instead of hard code it, can we use `listTables` to find it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14801: [SPARK-17234] [SQL] Table Existence Checking when...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/14801#discussion_r76720278 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala --- @@ -218,6 +218,12 @@ class VersionsSuite extends SparkFunSuite with Logging { holdDDLTime = false) } +test(s"$version: tableExists") { + // No exception should be thrown --- End diff -- https://github.com/apache/hive/blob/ecab0d072b50a8d85dca6e850e47425d96c1ac09/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1216 We can let Hive return an exception. However, we set the flag to [false](https://github.com/gatorsmile/spark/blob/ce35aa01894e83761226db38d3ca0ce720de5284/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala#L351). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14801: [SPARK-17234] [SQL] Table Existence Checking when...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/14801#discussion_r76720319 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala --- @@ -637,6 +639,36 @@ class HiveDDLSuite } } + test("create table with the same name as an index table") { +val tabName = "tab1" +val indexName = tabName + "_index" +val indexTabName = "default__tab1_tab1_index__" --- End diff -- Let me try it. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14801: [SPARK-17234] [SQL] Table Existence Checking when...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14801 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14801: [SPARK-17234] [SQL] Table Existence Checking when...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/14801#discussion_r89679622 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala --- @@ -171,9 +172,13 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, hadoopConf: Configurat ignoreIfExists: Boolean): Unit = withClient { assert(tableDefinition.identifier.database.isDefined) val db = tableDefinition.identifier.database.get +val table = tableDefinition.identifier.table requireDbExists(db) verifyTableProperties(tableDefinition) +if (tableExists(db, table) && !ignoreIfExists) { + throw new TableAlreadyExistsException(db = db, table = table) +} --- End diff -- After a few months, I found the above code looks weird. We should follow the same logics in InMemoryCatalog.scala. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org