[GitHub] spark pull request #14801: [SPARK-17234] [SQL] Table Existence Checking when...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/14801#discussion_r89679622
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -171,9 +172,13 @@ private[spark] class HiveExternalCatalog(conf: 
SparkConf, hadoopConf: Configurat
   ignoreIfExists: Boolean): Unit = withClient {
 assert(tableDefinition.identifier.database.isDefined)
 val db = tableDefinition.identifier.database.get
+val table = tableDefinition.identifier.table
 requireDbExists(db)
 verifyTableProperties(tableDefinition)
 
+if (tableExists(db, table) && !ignoreIfExists) {
+  throw new TableAlreadyExistsException(db = db, table = table)
+}
--- End diff --

After a few months, I found the above code looks weird. We should follow 
the same logics in InMemoryCatalog.scala.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14801: [SPARK-17234] [SQL] Table Existence Checking when...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/14801


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14801: [SPARK-17234] [SQL] Table Existence Checking when...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/14801#discussion_r76720319
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala 
---
@@ -637,6 +639,36 @@ class HiveDDLSuite
 }
   }
 
+  test("create table with the same name as an index table") {
+val tabName = "tab1"
+val indexName = tabName + "_index"
+val indexTabName = "default__tab1_tab1_index__"
--- End diff --

Let me try it. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14801: [SPARK-17234] [SQL] Table Existence Checking when...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/14801#discussion_r76720278
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala ---
@@ -218,6 +218,12 @@ class VersionsSuite extends SparkFunSuite with Logging 
{
 holdDDLTime = false)
 }
 
+test(s"$version: tableExists") {
+  // No exception should be thrown
--- End diff --


https://github.com/apache/hive/blob/ecab0d072b50a8d85dca6e850e47425d96c1ac09/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1216

We can let Hive return an exception. However, we set the flag to 
[false](https://github.com/gatorsmile/spark/blob/ce35aa01894e83761226db38d3ca0ce720de5284/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala#L351).
 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14801: [SPARK-17234] [SQL] Table Existence Checking when...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/14801#discussion_r76719642
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala 
---
@@ -637,6 +639,36 @@ class HiveDDLSuite
 }
   }
 
+  test("create table with the same name as an index table") {
+val tabName = "tab1"
+val indexName = tabName + "_index"
+val indexTabName = "default__tab1_tab1_index__"
--- End diff --

instead of hard code it, can we use `listTables` to find it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14801: [SPARK-17234] [SQL] Table Existence Checking when...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/14801#discussion_r76719507
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala ---
@@ -218,6 +218,12 @@ class VersionsSuite extends SparkFunSuite with Logging 
{
 holdDDLTime = false)
 }
 
+test(s"$version: tableExists") {
+  // No exception should be thrown
--- End diff --

hive's `tableExists` may throw exception?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14801: [SPARK-17234] [SQL] Table Existence Checking when...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/14801#discussion_r76663011
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -450,7 +455,11 @@ private[spark] class HiveExternalCatalog(conf: 
SparkConf, hadoopConf: Configurat
   }
 
   override def tableExists(db: String, table: String): Boolean = 
withClient {
-client.getTableOption(db, table).isDefined
+try {
+  client.getTableOption(db, table).isDefined
+} catch {
+  case e: AnalysisException if e.message.contains("Hive index table is 
not supported") => true
--- End diff --

Yeah, that is a good idea. Let me add it. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14801: [SPARK-17234] [SQL] Table Existence Checking when...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/14801#discussion_r76565215
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -450,7 +455,11 @@ private[spark] class HiveExternalCatalog(conf: 
SparkConf, hadoopConf: Configurat
   }
 
   override def tableExists(db: String, table: String): Boolean = 
withClient {
-client.getTableOption(db, table).isDefined
+try {
+  client.getTableOption(db, table).isDefined
+} catch {
+  case e: AnalysisException if e.message.contains("Hive index table is 
not supported") => true
--- End diff --

this looks hacky. Actually why we use `getTableOption` to implement the 
semantic of `tableExists`? I'd like to make `HiveClient` provide the 
`tableExists` API directly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14801: [SPARK-17234] [SQL] Table Existence Checking when...

[gatorsmile]

GitHub user gatorsmile opened a pull request:

https://github.com/apache/spark/pull/14801

[SPARK-17234] [SQL] Table Existence Checking when Index Table with the Same 
Name Exists

### What changes were proposed in this pull request?
Hive Index tables are not supported by Spark SQL. Thus, we issue an 
exception when users try to access Hive Index tables. When the internal 
function `tableExists` tries to access Hive Index tables, it always gets the 
same error message: ```Hive index table is not supported```. This message could 
be confusing to users, since their SQL operations could be completely unrelated 
to Hive Index tables. For example, when users try to alter a table to a new 
name and there exists an index table with the same name, the expected exception 
should be a `TableAlreadyExistsException`.

This PR made the following changes:
- Introduced a new `AnalysisException` type: 
`SQLFeatureNotSupportedException`. When users try to access an `Index Table`, 
we will issue a `SQLFeatureNotSupportedException`.
- `tableExists` returns `true` when hitting a 
`SQLFeatureNotSupportedException` and the feature is `Hive index table`.
- Add a checking `requireTableNotExists` for `SessionCatalog`'s 
`createTable` API; otherwise, the current implementation relies on the Hive's 
internal checking.

### How was this patch tested?
Added a test case

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gatorsmile/spark tableExists

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/14801.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #14801


commit 1af428b68c4341192bf8f66af7c434a7b89be61d
Author: gatorsmile 
Date:   2016-08-25T06:26:00Z

fix

commit 664d6f1caa9b3d62eafbddb292991def722910ae
Author: gatorsmile 
Date:   2016-08-25T06:34:16Z

improve test cases

commit c400c5292a32549cea80861adfaefeb41f4d90b3
Author: gatorsmile 
Date:   2016-08-25T07:12:57Z

fix




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org