[GitHub] spark pull request #23255: [SPARK-26307] [SQL] Fix CTAS when INSERT a partit...

2018-12-09 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/23255


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23255: [SPARK-26307] [SQL] Fix CTAS when INSERT a partit...

2018-12-08 Thread cloud-fan 
Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/23255#discussion_r240026441
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala ---
@@ -752,6 +752,17 @@ class InsertSuite extends QueryTest with 
TestHiveSingleton with BeforeAndAfter
 }
   }
 
+  test("CTAS: INSERT a partitioned table using Hive serde") {
--- End diff --

+1


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23255: [SPARK-26307] [SQL] Fix CTAS when INSERT a partit...

2018-12-07 Thread dongjoon-hyun 
Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/23255#discussion_r239926758
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateHiveTableAsSelectCommand.scala
 ---
@@ -57,9 +57,11 @@ case class CreateHiveTableAsSelectCommand(
 return Seq.empty
   }
 
+  // For CTAS, there is no static partition values to insert.
+  val partition = tableDesc.partitionColumnNames.map(_ -> None).toMap
--- End diff --

Logically, this happens not only `CTAS` but also `INSERT`. Could you update 
the PR title and description to be clear?
```scala
scala> Seq(("a", 100)).toDF("part", 
"id").write.format("hive").partitionBy("part").mode("append").saveAsTable("newtable")
scala> Seq(("a", 100)).toDF("part", 
"id").write.format("hive").partitionBy("part").mode("append").saveAsTable("newtable")
org.apache.spark.SparkException: Requested partitioning does not match the 
newtable table:
Requested partitions:
```


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23255: [SPARK-26307] [SQL] Fix CTAS when INSERT a partit...

2018-12-07 Thread dongjoon-hyun 
Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/23255#discussion_r239922982
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala ---
@@ -752,6 +752,17 @@ class InsertSuite extends QueryTest with 
TestHiveSingleton with BeforeAndAfter
 }
   }
 
+  test("CTAS: INSERT a partitioned table using Hive serde") {
--- End diff --

Since this was a regression, can we have `SPARK-26307` in the test name?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23255: [SPARK-26307] [SQL] Fix CTAS when INSERT a partit...

2018-12-07 Thread gatorsmile 
GitHub user gatorsmile opened a pull request:

https://github.com/apache/spark/pull/23255

[SPARK-26307] [SQL] Fix CTAS when INSERT a partitioned table using Hive 
serde

## What changes were proposed in this pull request?

This was a  Spark 2.3 regression introduced in 
https://github.com/apache/spark/pull/20521. We should add the partition info 
for InsertIntoHiveTable in CreateHiveTableAsSelectCommand. Otherwise, we will 
hit the following error by running the newly added test case:

```
[info] - CTAS: INSERT a partitioned table using Hive serde *** FAILED *** 
(829 milliseconds)
[info]   org.apache.spark.SparkException: Requested partitioning does not 
match the tab1 table:
[info] Requested partitions: 
[info] Table partitions: part
[info]   at 
org.apache.spark.sql.hive.execution.InsertIntoHiveTable.processInsert(InsertIntoHiveTable.scala:179)
[info]   at 
org.apache.spark.sql.hive.execution.InsertIntoHiveTable.run(InsertIntoHiveTable.scala:107)
```

## How was this patch tested?

Added a test case.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gatorsmile/spark fixCTAS

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/23255.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #23255


commit 921967b369a8e48269da0825278a7e1095b9173e
Author: gatorsmile 
Date:   2018-12-07T17:59:36Z

fix.




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org