[GitHub] spark pull request #17953: [SPARK-20680][SQL] Spark-sql do not support for v...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17953 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17953: [SPARK-20680][SQL] Spark-sql do not support for v...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17953#discussion_r120967472
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
---
@@ -1928,4 +1928,17 @@ class HiveDDLSuite
}
}
}
+
+ test("SPARK-20680: Spark-sql do not support for void column datatype of
view") {
+withTable("t", "tabNullType") {
+ val client =
spark.sharedState.externalCatalog.asInstanceOf[HiveExternalCatalog].client
+ client.runSqlHive("CREATE TABLE t (t1 int)")
+ client.runSqlHive("INSERT INTO t VALUES (3)")
+ client.runSqlHive("CREATE TABLE tabNullType AS SELECT NULL AS col
FROM t")
--- End diff --
IIRC, hive 2 does't support this. Let's test with `CREATE VIEW AS ...` to
be safer
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17953: [SPARK-20680][SQL] Spark-sql do not support for v...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17953#discussion_r117669841
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
---
@@ -1928,4 +1928,16 @@ class HiveDDLSuite
}
}
}
+
+ test("SPARK-20680: Spark-sql do not support for void column datatype of
view") {
+withTable("t", "tabNullType") {
+ val client =
spark.sharedState.externalCatalog.asInstanceOf[HiveExternalCatalog].client
+ client.runSqlHive("CREATE TABLE t (t1 int)")
+ client.runSqlHive("INSERT INTO t VALUES (3)")
+ client.runSqlHive("CREATE TABLE tabNullType AS SELECT NULL AS col
FROM t")
+ assert(spark.table("tabNullType").collect.nonEmpty)
+ // check desc sql expression doesn't throw exception
+ assert(spark.sql("desc tabNullType").collect.nonEmpty)
--- End diff --
After the above change, you can improve the test to
```Scala
val desc = sql("DESC tabNullType").collect().toSeq
assert(desc.contains(Row("col", "void", null)))
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17953: [SPARK-20680][SQL] Spark-sql do not support for v...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17953#discussion_r117669482
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
---
@@ -1928,4 +1928,16 @@ class HiveDDLSuite
}
}
}
+
+ test("SPARK-20680: Spark-sql do not support for void column datatype of
view") {
+withTable("t", "tabNullType") {
+ val client =
spark.sharedState.externalCatalog.asInstanceOf[HiveExternalCatalog].client
+ client.runSqlHive("CREATE TABLE t (t1 int)")
+ client.runSqlHive("INSERT INTO t VALUES (3)")
+ client.runSqlHive("CREATE TABLE tabNullType AS SELECT NULL AS col
FROM t")
+ assert(spark.table("tabNullType").collect.nonEmpty)
+ // check desc sql expression doesn't throw exception
+ assert(spark.sql("desc tabNullType").collect.nonEmpty)
--- End diff --
In class `NullType`, we can add the following line:
```
override def simpleString: String = "void"
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17953: [SPARK-20680][SQL] Spark-sql do not support for v...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17953#discussion_r117669189
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
---
@@ -1928,4 +1928,16 @@ class HiveDDLSuite
}
}
}
+
+ test("SPARK-20680: Spark-sql do not support for void column datatype of
view") {
+withTable("t", "tabNullType") {
+ val client =
spark.sharedState.externalCatalog.asInstanceOf[HiveExternalCatalog].client
+ client.runSqlHive("CREATE TABLE t (t1 int)")
+ client.runSqlHive("INSERT INTO t VALUES (3)")
+ client.runSqlHive("CREATE TABLE tabNullType AS SELECT NULL AS col
FROM t")
+ assert(spark.table("tabNullType").collect.nonEmpty)
--- End diff --
-> `checkAnswer(spark.table("tabNullType"), Row(null))`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17953: [SPARK-20680][SQL] Spark-sql do not support for v...
Github user LantaoJin commented on a diff in the pull request:
https://github.com/apache/spark/pull/17953#discussion_r116912802
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
---
@@ -1928,4 +1928,17 @@ class HiveDDLSuite
}
}
}
+
+ test("SPARK-20680: Spark-sql do not support for void column datatype of
view") {
+withTable("t", "tabNullType") {
+ val client =
spark.sharedState.externalCatalog.asInstanceOf[HiveExternalCatalog].client
+ client.runSqlHive("CREATE TABLE t (t1 int)")
+ client.runSqlHive("INSERT INTO t VALUES (3)")
+ client.runSqlHive("CREATE TABLE tabNullType AS SELECT NULL AS col
FROM t")
+ assert(spark.table("tabNullType").schema.contains(StructField("col",
NullType)))
--- End diff --
From the Jenkins, I saw below error message:
> org.scalatest.exceptions.TestFailedException:
StructType(StructField(col,NullType,true)) did not contain
StructField(col,NullType,true)
But it can passed from my spark-shell:
> scala> val schema = spark.table("tabNullType").schema
schema: org.apache.spark.sql.types.StructType =
StructType(StructField(col,NullType,true))
scala> schema.contains(StructField("col", NullType))
res7: Boolean = true
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17953: [SPARK-20680][SQL] Spark-sql do not support for v...
Github user LantaoJin commented on a diff in the pull request:
https://github.com/apache/spark/pull/17953#discussion_r116705233
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
---
@@ -1504,6 +1504,7 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef]
with Logging {
case ("decimal", precision :: Nil) =>
DecimalType(precision.getText.toInt, 0)
case ("decimal", precision :: scale :: Nil) =>
DecimalType(precision.getText.toInt, scale.getText.toInt)
+ case ("void", Nil) => NullType
--- End diff --
Sure
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17953: [SPARK-20680][SQL] Spark-sql do not support for v...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17953#discussion_r116607708
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
---
@@ -1504,6 +1504,7 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef]
with Logging {
case ("decimal", precision :: Nil) =>
DecimalType(precision.getText.toInt, 0)
case ("decimal", precision :: scale :: Nil) =>
DecimalType(precision.getText.toInt, scale.getText.toInt)
+ case ("void", Nil) => NullType
--- End diff --
Could you add a comment to explain this specific scenario?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17953: [SPARK-20680][SQL] Spark-sql do not support for v...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/17953#discussion_r116268498
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
---
@@ -1504,6 +1504,7 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef]
with Logging {
case ("decimal", precision :: Nil) =>
DecimalType(precision.getText.toInt, 0)
case ("decimal", precision :: scale :: Nil) =>
DecimalType(precision.getText.toInt, scale.getText.toInt)
+ case ("void", Nil) => NullType
--- End diff --
+1 for the test case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17953: [SPARK-20680][SQL] Spark-sql do not support for v...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17953#discussion_r116157223
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
---
@@ -1504,6 +1504,7 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef]
with Logging {
case ("decimal", precision :: Nil) =>
DecimalType(precision.getText.toInt, 0)
case ("decimal", precision :: scale :: Nil) =>
DecimalType(precision.getText.toInt, scale.getText.toInt)
+ case ("void", Nil) => NullType
--- End diff --
Hive 2.x disables it. Could you add some test cases by reading and writing
the tables? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17953: [SPARK-20680][SQL] Spark-sql do not support for v...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/17953#discussion_r116138305
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
---
@@ -1504,6 +1504,7 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef]
with Logging {
case ("decimal", precision :: Nil) =>
DecimalType(precision.getText.toInt, 0)
case ("decimal", precision :: scale :: Nil) =>
DecimalType(precision.getText.toInt, scale.getText.toInt)
+ case ("void", Nil) => NullType
--- End diff --
Apparently Hive can have null typed columns. So this should be the location
where you'd want to change this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17953: [SPARK-20680][SQL] Spark-sql do not support for v...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17953#discussion_r116127009
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
---
@@ -1504,6 +1504,7 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef]
with Logging {
case ("decimal", precision :: Nil) =>
DecimalType(precision.getText.toInt, 0)
case ("decimal", precision :: scale :: Nil) =>
DecimalType(precision.getText.toInt, scale.getText.toInt)
+ case ("void", Nil) => NullType
--- End diff --
This change really resolves your issue?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17953: [SPARK-20680][SQL] Spark-sql do not support for v...
GitHub user LantaoJin opened a pull request: https://github.com/apache/spark/pull/17953 [SPARK-20680][SQL] Spark-sql do not support for void column datatype ⦠â¦of view ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/LantaoJin/spark SPARK-20680 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/17953.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #17953 commit d14fc418c3c998280dbfb75fbf4bc529f1b83f30 Author: Alan Jin Date: 2017-05-11T12:02:56Z [SPARK-20680][SQL] Spark-sql do not support for void column datatype of view --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
