[jira] [Work logged] (HIVE-25659) Metastore direct sql queries with IN/(NOT IN) should be split based on max parameters allowed by SQL DB

[ASF GitHub Bot (Jira)]

     [ 
https://issues.apache.org/jira/browse/HIVE-25659?focusedWorklogId=675126&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-675126
 ]

ASF GitHub Bot logged work on HIVE-25659:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 04/Nov/21 00:14
            Start Date: 04/Nov/21 00:14
    Worklog Time Spent: 10m 
      Work Description: guptanikhil007 commented on a change in pull request 
#2758:
URL: https://github.com/apache/hive/pull/2758#discussion_r741713387



##########
File path: 
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/txn/TestTxnUtils.java
##########
@@ -150,12 +151,34 @@ public void testBuildQueryWithINClause() throws Exception 
{
     ret = TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, 
inList, "TXN_ID", false, false);
     Assert.assertEquals(3, queries.size());
     Assert.assertEquals(queries.size(), ret.size());
-    Assert.assertEquals(2255L, ret.get(0).longValue());
-    Assert.assertEquals(2033L, ret.get(1).longValue());
-    Assert.assertEquals(33L, ret.get(2).longValue());
+    Assert.assertEquals(2000L, ret.get(0).longValue());
+    Assert.assertEquals(2000L, ret.get(1).longValue());
+    Assert.assertEquals(321L, ret.get(2).longValue());
     runAgainstDerby(queries);
   }
 
+  @Test(expected = IllegalArgumentException.class)
+  public void testBuildQueryWithNOTINClauseFailure() throws Exception {

Review comment:
       If the not in query cannot be fit in one query, it should fail according 
to function definition

##########
File path: 
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/txn/TestTxnUtils.java
##########
@@ -150,12 +151,34 @@ public void testBuildQueryWithINClause() throws Exception 
{
     ret = TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, 
inList, "TXN_ID", false, false);
     Assert.assertEquals(3, queries.size());
     Assert.assertEquals(queries.size(), ret.size());
-    Assert.assertEquals(2255L, ret.get(0).longValue());
-    Assert.assertEquals(2033L, ret.get(1).longValue());
-    Assert.assertEquals(33L, ret.get(2).longValue());
+    Assert.assertEquals(2000L, ret.get(0).longValue());
+    Assert.assertEquals(2000L, ret.get(1).longValue());
+    Assert.assertEquals(321L, ret.get(2).longValue());
     runAgainstDerby(queries);
   }
 
+  @Test(expected = IllegalArgumentException.class)
+  public void testBuildQueryWithNOTINClauseFailure() throws Exception {

Review comment:
       If the `not in` query cannot be fit in one query, it should fail 
according to function definition

##########
File path: 
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/txn/TestTxnUtils.java
##########
@@ -150,12 +151,34 @@ public void testBuildQueryWithINClause() throws Exception 
{
     ret = TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, 
inList, "TXN_ID", false, false);
     Assert.assertEquals(3, queries.size());
     Assert.assertEquals(queries.size(), ret.size());
-    Assert.assertEquals(2255L, ret.get(0).longValue());
-    Assert.assertEquals(2033L, ret.get(1).longValue());
-    Assert.assertEquals(33L, ret.get(2).longValue());
+    Assert.assertEquals(2000L, ret.get(0).longValue());
+    Assert.assertEquals(2000L, ret.get(1).longValue());
+    Assert.assertEquals(321L, ret.get(2).longValue());
     runAgainstDerby(queries);
   }
 
+  @Test(expected = IllegalArgumentException.class)
+  public void testBuildQueryWithNOTINClauseFailure() throws Exception {

Review comment:
       If the `not in` query cannot be fit in one single query, it should fail 
according to function definition




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 675126)
    Time Spent: 1h 50m  (was: 1h 40m)

> Metastore direct sql queries with IN/(NOT IN) should be split based on max 
> parameters allowed by SQL DB
> -------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-25659
>                 URL: https://issues.apache.org/jira/browse/HIVE-25659
>             Project: Hive
>          Issue Type: Bug
>          Components: Standalone Metastore
>    Affects Versions: 3.1.0, 4.0.0
>            Reporter: Nikhil Gupta
>            Assignee: Nikhil Gupta
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 4.0.0
>
>          Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Function 
> org.apache.hadoop.hive.metastore.txn.TxnUtils#buildQueryWithINClauseStrings 
> can generate queries with huge number of parameters with very small value of 
> DIRECT_SQL_MAX_ELEMENTS_IN_CLAUSE and DIRECT_SQL_MAX_QUERY_LENGTH while 
> generating delete query for completed_compactions table
> Example:
> {code:java}
> DIRECT_SQL_MAX_ELEMENTS_IN_CLAUSE = 100
> DIRECT_SQL_MAX_QUERY_LENGTH = 10 (10 KB)
> Number of parameters in a single query = 4759
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)