Github user kmanamcheri commented on the issue:
https://github.com/apache/spark/pull/22614
> Based on my understanding, the solution of FB team is to retry the
following commands multiple times:
>
> ```
> getPartitionsByFilterMethod.invoke(hive, ta
Github user kmanamcheri commented on the issue:
https://github.com/apache/spark/pull/22614
@gatorsmile, @tejasapatil was reviewing the code before I added the new
config option. I have asked him to review the new code. Lets see what his
thoughts are on that. I have also asked him
Github user kmanamcheri commented on a diff in the pull request:
https://github.com/apache/spark/pull/22614#discussion_r223498018
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala ---
@@ -746,34 +746,20 @@ private[client] class Shim_v0_13 extends
Github user kmanamcheri commented on a diff in the pull request:
https://github.com/apache/spark/pull/22614#discussion_r223473324
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala ---
@@ -746,34 +746,45 @@ private[client] class Shim_v0_13 extends
Github user kmanamcheri commented on a diff in the pull request:
https://github.com/apache/spark/pull/22614#discussion_r223469446
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala
---
@@ -79,12 +82,30 @@ class HiveClientSuite(version: String
Github user kmanamcheri commented on a diff in the pull request:
https://github.com/apache/spark/pull/22614#discussion_r223429117
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala
---
@@ -79,12 +82,30 @@ class HiveClientSuite(version: String
Github user kmanamcheri commented on a diff in the pull request:
https://github.com/apache/spark/pull/22614#discussion_r223419868
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala ---
@@ -754,26 +755,38 @@ private[client] class Shim_v0_13 extends
Github user kmanamcheri commented on a diff in the pull request:
https://github.com/apache/spark/pull/22614#discussion_r223418766
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala ---
@@ -754,26 +755,38 @@ private[client] class Shim_v0_13 extends
Github user kmanamcheri commented on the issue:
https://github.com/apache/spark/pull/22614
@viirya I have updated the title and description.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user kmanamcheri commented on a diff in the pull request:
https://github.com/apache/spark/pull/22614#discussion_r223148534
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala
---
@@ -87,6 +90,18 @@ class HiveClientSuite(version: String
Github user kmanamcheri commented on a diff in the pull request:
https://github.com/apache/spark/pull/22614#discussion_r223148506
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -544,6 +544,15 @@ object SQLConf {
.booleanConf
Github user kmanamcheri commented on a diff in the pull request:
https://github.com/apache/spark/pull/22614#discussion_r223139874
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala
---
@@ -87,6 +90,18 @@ class HiveClientSuite(version: String
Github user kmanamcheri commented on a diff in the pull request:
https://github.com/apache/spark/pull/22614#discussion_r223136013
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -544,6 +544,15 @@ object SQLConf {
.booleanConf
Github user kmanamcheri commented on a diff in the pull request:
https://github.com/apache/spark/pull/22614#discussion_r223122115
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -544,6 +544,15 @@ object SQLConf {
.booleanConf
Github user kmanamcheri commented on the issue:
https://github.com/apache/spark/pull/22614
@gatorsmile I have added the config option and an additional test.
Here's the new behavior
- Setting spark.sql.metastorePartitionPruningFallback to 'false' will
Github user kmanamcheri commented on the issue:
https://github.com/apache/spark/pull/22614
> Let us add a conf to control it? Failing fast is better than hanging. If
users want to get all partitions, they can change the conf by themselves.
@gatorsmile We already have a con
Github user kmanamcheri commented on a diff in the pull request:
https://github.com/apache/spark/pull/22614#discussion_r222372452
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala ---
@@ -746,34 +746,20 @@ private[client] class Shim_v0_13 extends
Github user kmanamcheri commented on a diff in the pull request:
https://github.com/apache/spark/pull/22614#discussion_r222359233
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala ---
@@ -746,34 +746,20 @@ private[client] class Shim_v0_13 extends
Github user kmanamcheri commented on a diff in the pull request:
https://github.com/apache/spark/pull/22614#discussion_r222348679
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala ---
@@ -746,34 +746,20 @@ private[client] class Shim_v0_13 extends
Github user kmanamcheri commented on a diff in the pull request:
https://github.com/apache/spark/pull/22614#discussion_r222140323
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala ---
@@ -746,34 +746,20 @@ private[client] class Shim_v0_13 extends
Github user kmanamcheri commented on a diff in the pull request:
https://github.com/apache/spark/pull/22614#discussion_r222123426
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala ---
@@ -746,34 +746,20 @@ private[client] class Shim_v0_13 extends
Github user kmanamcheri commented on the issue:
https://github.com/apache/spark/pull/22614
@mallman @cloud-fan @ericl @rezasafi
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
GitHub user kmanamcheri opened a pull request:
https://github.com/apache/spark/pull/22614
HiveClient.getPartitionsByFilter should not throw an exception if HMS
retries directSql
## What changes were proposed in this pull request?
When using partition filter pushdown to HMS
Github user kmanamcheri commented on a diff in the pull request:
https://github.com/apache/spark/pull/15673#discussion_r221107390
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala ---
@@ -586,17 +587,31 @@ private[client] class Shim_v0_13 extends
24 matches
Mail list logo
