Github user krisgeus commented on the issue:
https://github.com/apache/spark/pull/21893
Sorry to hear you don't like to support this although in Hive this works
correctly.
If chances of getting this merged are so slim I will wait before
implementing a configuration switch and add
Github user krisgeus commented on a diff in the pull request:
https://github.com/apache/spark/pull/21893#discussion_r206236120
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
---
@@ -437,6 +437,8 @@ private[hive] class HiveClientImpl
Github user krisgeus commented on the issue:
https://github.com/apache/spark/pull/21893
@gatorsmile That's why this PR makes sure that in the case of differing
formats we use the HiveTableScanExec path and not the FileSourceScanExec
Github user krisgeus commented on the issue:
https://github.com/apache/spark/pull/21893
@gatorsmile Yes indeed. A singel partitioned table having multiple
partitions.
Each partition can have it own file format like parquet, orc, avro etc.
In our case new data comes in in AVRO
Github user krisgeus commented on the issue:
https://github.com/apache/spark/pull/21893
@xuanyuanking Thanks for another round of review comments. Addressed them
in the 4th commit. Hope it is ok now
Github user krisgeus commented on a diff in the pull request:
https://github.com/apache/spark/pull/21893#discussion_r206229739
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/MultiFormatTableSuite.scala
---
@@ -0,0 +1,514 @@
+/*
+ * Licensed
Github user krisgeus commented on a diff in the pull request:
https://github.com/apache/spark/pull/21893#discussion_r206229791
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/MultiFormatTableSuite.scala
---
@@ -0,0 +1,514 @@
+/*
+ * Licensed
Github user krisgeus commented on a diff in the pull request:
https://github.com/apache/spark/pull/21893#discussion_r206229680
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/MultiFormatTableSuite.scala
---
@@ -0,0 +1,514 @@
+/*
+ * Licensed
Github user krisgeus commented on a diff in the pull request:
https://github.com/apache/spark/pull/21893#discussion_r206229532
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/MultiFormatTableSuite.scala
---
@@ -0,0 +1,514 @@
+/*
+ * Licensed
Github user krisgeus commented on a diff in the pull request:
https://github.com/apache/spark/pull/21893#discussion_r206229611
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/MultiFormatTableSuite.scala
---
@@ -0,0 +1,514 @@
+/*
+ * Licensed
Github user krisgeus commented on a diff in the pull request:
https://github.com/apache/spark/pull/21893#discussion_r206229493
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/MultiFormatTableSuite.scala
---
@@ -0,0 +1,514 @@
+/*
+ * Licensed
Github user krisgeus commented on a diff in the pull request:
https://github.com/apache/spark/pull/21893#discussion_r206229212
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/MultiFormatTableSuite.scala
---
@@ -0,0 +1,514 @@
+/*
+ * Licensed
Github user krisgeus commented on a diff in the pull request:
https://github.com/apache/spark/pull/21893#discussion_r206228870
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParserUtils.scala
---
@@ -96,6 +96,9 @@ object ParserUtils
Github user krisgeus commented on the issue:
https://github.com/apache/spark/pull/21893
@xuanyuanking Processed all review comments and create jira for this. Feel
free to review again.
---
-
To unsubscribe, e-mail
Github user krisgeus commented on a diff in the pull request:
https://github.com/apache/spark/pull/21893#discussion_r205988632
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -857,6 +857,32 @@ class SparkSqlAstBuilder(conf: SQLConf
Github user krisgeus commented on a diff in the pull request:
https://github.com/apache/spark/pull/21893#discussion_r205988606
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/MultiFormatTableSuite.scala
---
@@ -0,0 +1,512 @@
+/*
+ * Licensed
Github user krisgeus commented on a diff in the pull request:
https://github.com/apache/spark/pull/21893#discussion_r205988600
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -857,6 +857,32 @@ class SparkSqlAstBuilder(conf: SQLConf
Github user krisgeus commented on a diff in the pull request:
https://github.com/apache/spark/pull/21893#discussion_r205988604
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ---
@@ -415,6 +415,51 @@ case class AlterTableSerDePropertiesCommand
Github user krisgeus commented on the issue:
https://github.com/apache/spark/pull/21893
Jira created: https://issues.apache.org/jira/browse/SPARK-24965
Working on the other comments
---
-
To unsubscribe, e-mail
GitHub user krisgeus opened a pull request:
https://github.com/apache/spark/pull/21893
Support selecting from partitioned tabels with partitions having different
data formats
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix
20 matches
Mail list logo
