[jira] [Work logged] (HIVE-25659) Metastore direct sql queries with IN/(NOT IN) should be split based on max parameters allowed by SQL DB

[ASF GitHub Bot (Jira)]

     [ 
https://issues.apache.org/jira/browse/HIVE-25659?focusedWorklogId=675771&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-675771
 ]

ASF GitHub Bot logged work on HIVE-25659:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 04/Nov/21 01:27
            Start Date: 04/Nov/21 01:27
    Worklog Time Spent: 10m 
      Work Description: sankarh commented on a change in pull request #2758:
URL: https://github.com/apache/hive/pull/2758#discussion_r741678476



##########
File path: 
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
##########
@@ -680,6 +680,9 @@ public static ConfVars getMetaConf(String name) {
     
DIRECT_SQL_MAX_ELEMENTS_VALUES_CLAUSE("metastore.direct.sql.max.elements.values.clause",
         "hive.direct.sql.max.elements.values.clause",
         1000, "The maximum number of values in a VALUES clause for INSERT 
statement."),
+    DIRECT_SQL_MAX_PARAMETERS("metastore.direct.sql.max.parameters",
+        "hive.direct.sql.max.parameters", 1000, "The maximum parameters the\n" 
+

Review comment:
       Rephrase: "... maximum query parameters..."

##########
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
##########
@@ -316,7 +317,7 @@ public static String getFullTableName(String dbName, String 
tableName) {
       // Compute the size of a query when the 'nextValue' is added to the 
current query.
       int querySize = querySizeExpected(buf.length(), nextValue.length(), 
suffix.length(), addParens);
 
-      if (querySize > maxQueryLength * 1024) {
+      if (querySize > maxQueryLength * 1024 || currentCount >= maxParameters) {

Review comment:
       nit: Keep each condition within ()

##########
File path: 
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/txn/TestTxnUtils.java
##########
@@ -150,12 +151,34 @@ public void testBuildQueryWithINClause() throws Exception 
{
     ret = TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, 
inList, "TXN_ID", false, false);
     Assert.assertEquals(3, queries.size());
     Assert.assertEquals(queries.size(), ret.size());
-    Assert.assertEquals(2255L, ret.get(0).longValue());
-    Assert.assertEquals(2033L, ret.get(1).longValue());
-    Assert.assertEquals(33L, ret.get(2).longValue());
+    Assert.assertEquals(2000L, ret.get(0).longValue());
+    Assert.assertEquals(2000L, ret.get(1).longValue());
+    Assert.assertEquals(321L, ret.get(2).longValue());
     runAgainstDerby(queries);
   }
 
+  @Test(expected = IllegalArgumentException.class)
+  public void testBuildQueryWithNOTINClauseFailure() throws Exception {

Review comment:
       Why would this test fail?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 675771)
    Time Spent: 2h 20m  (was: 2h 10m)

> Metastore direct sql queries with IN/(NOT IN) should be split based on max 
> parameters allowed by SQL DB
> -------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-25659
>                 URL: https://issues.apache.org/jira/browse/HIVE-25659
>             Project: Hive
>          Issue Type: Bug
>          Components: Standalone Metastore
>    Affects Versions: 3.1.0, 4.0.0
>            Reporter: Nikhil Gupta
>            Assignee: Nikhil Gupta
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 4.0.0
>
>          Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Function 
> org.apache.hadoop.hive.metastore.txn.TxnUtils#buildQueryWithINClauseStrings 
> can generate queries with huge number of parameters with very small value of 
> DIRECT_SQL_MAX_ELEMENTS_IN_CLAUSE and DIRECT_SQL_MAX_QUERY_LENGTH while 
> generating delete query for completed_compactions table
> Example:
> {code:java}
> DIRECT_SQL_MAX_ELEMENTS_IN_CLAUSE = 100
> DIRECT_SQL_MAX_QUERY_LENGTH = 10 (10 KB)
> Number of parameters in a single query = 4759
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)