[jira] [Commented] (SPARK-25561) HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql
[ https://issues.apache.org/jira/browse/SPARK-25561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16631193#comment-16631193 ] Karthik Manamcheri commented on SPARK-25561: The root cause was from SPARK-17992 ping [~michael] what are your thoughts on this? > HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql > -- > > Key: SPARK-25561 > URL: https://issues.apache.org/jira/browse/SPARK-25561 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 >Reporter: Karthik Manamcheri >Priority: Major > > In HiveShim.scala, the current behavior is that if > hive.metastore.try.direct.sql is enabled, we expect the getPartitionsByFilter > call to succeed. If it fails, we'll throw a RuntimeException. > However, this might not always be the case. Hive's direct SQL functionality > is best-attempt. Meaning, it will fall back to ORM if direct sql fails. Spark > should handle that exception correctly if Hive falls back to ORM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25561) HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql
[ https://issues.apache.org/jira/browse/SPARK-25561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16632137#comment-16632137 ] Michael Allman commented on SPARK-25561: cc [~cloud_fan] [~ekhliang] Hi [~karthik.manamcheri]. Thanks for reporting this. I can't take a look right now, but I believe we have test cases that exercise this scenario. If not, it's certainly a whole in our coverage. If we do, it may be that Hive's behavior in this scenario is version-dependent, and we don't have coverage for your version of Hive. What version of Hive are you using? Thanks. > HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql > -- > > Key: SPARK-25561 > URL: https://issues.apache.org/jira/browse/SPARK-25561 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 >Reporter: Karthik Manamcheri >Priority: Major > > In HiveShim.scala, the current behavior is that if > hive.metastore.try.direct.sql is enabled, we expect the getPartitionsByFilter > call to succeed. If it fails, we'll throw a RuntimeException. > However, this might not always be the case. Hive's direct SQL functionality > is best-attempt. Meaning, it will fall back to ORM if direct sql fails. Spark > should handle that exception correctly if Hive falls back to ORM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25561) HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql
[ https://issues.apache.org/jira/browse/SPARK-25561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16632483#comment-16632483 ] Karthik Manamcheri commented on SPARK-25561: [~michael] thanks for the prompt reply. This is hard to test because the problem happens only in the case when HMS goes into fallback ORM mode. For this to happen, we need to have the direct SQL query fail in HMS. There are no consistent bugs (that I know of) which can be used to test this in a deterministic fashion. I was able to run into this running Hive 1.1.0. However, as I understand HMS behavior of falling back to ORM has been the same in Hive from the beginning. Not sure. > HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql > -- > > Key: SPARK-25561 > URL: https://issues.apache.org/jira/browse/SPARK-25561 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 >Reporter: Karthik Manamcheri >Priority: Major > > In HiveShim.scala, the current behavior is that if > hive.metastore.try.direct.sql is enabled, we expect the getPartitionsByFilter > call to succeed. If it fails, we'll throw a RuntimeException. > However, this might not always be the case. Hive's direct SQL functionality > is best-attempt. Meaning, it will fall back to ORM if direct sql fails. Spark > should handle that exception correctly if Hive falls back to ORM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25561) HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql
[ https://issues.apache.org/jira/browse/SPARK-25561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16634332#comment-16634332 ] Karthik Manamcheri commented on SPARK-25561: I am working on a patch for this and will post a PR as soon as possible. > HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql > -- > > Key: SPARK-25561 > URL: https://issues.apache.org/jira/browse/SPARK-25561 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 >Reporter: Karthik Manamcheri >Priority: Major > > In HiveShim.scala, the current behavior is that if > hive.metastore.try.direct.sql is enabled, we expect the getPartitionsByFilter > call to succeed. If it fails, we'll throw a RuntimeException. > However, this might not always be the case. Hive's direct SQL functionality > is best-attempt. Meaning, it will fall back to ORM if direct sql fails. Spark > should handle that exception correctly if Hive falls back to ORM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25561) HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql
[ https://issues.apache.org/jira/browse/SPARK-25561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16635946#comment-16635946 ] Apache Spark commented on SPARK-25561: -- User 'kmanamcheri' has created a pull request for this issue: https://github.com/apache/spark/pull/22614 > HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql > -- > > Key: SPARK-25561 > URL: https://issues.apache.org/jira/browse/SPARK-25561 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 >Reporter: Karthik Manamcheri >Priority: Major > > In HiveShim.scala, the current behavior is that if > hive.metastore.try.direct.sql is enabled, we expect the getPartitionsByFilter > call to succeed. If it fails, we'll throw a RuntimeException. > However, this might not always be the case. Hive's direct SQL functionality > is best-attempt. Meaning, it will fall back to ORM if direct sql fails. Spark > should handle that exception correctly if Hive falls back to ORM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25561) HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql
[ https://issues.apache.org/jira/browse/SPARK-25561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16635945#comment-16635945 ] Karthik Manamcheri commented on SPARK-25561: Created PR https://github.com/apache/spark/pull/22614 > HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql > -- > > Key: SPARK-25561 > URL: https://issues.apache.org/jira/browse/SPARK-25561 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 >Reporter: Karthik Manamcheri >Priority: Major > > In HiveShim.scala, the current behavior is that if > hive.metastore.try.direct.sql is enabled, we expect the getPartitionsByFilter > call to succeed. If it fails, we'll throw a RuntimeException. > However, this might not always be the case. Hive's direct SQL functionality > is best-attempt. Meaning, it will fall back to ORM if direct sql fails. Spark > should handle that exception correctly if Hive falls back to ORM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
