veghlaci05 commented on code in PR #4421:
URL: https://github.com/apache/hive/pull/4421#discussion_r1231980882
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -554,6 +554,11 @@ public enum ConfVars {
360, TimeUnit.MINUTES,
"Logging frequency of ACID related metrics. Set this value to 0 to
completely turn off logging. " +
"Default time unit: minutes"),
+ COMPACTOR_FETCH_SIZE(
+ "metastore.compactor.fetch.size",
+ "hive.compactor.fetch.size",
+ 1000,
+ "Limits the number of items fetched during cleaning, abort and
finding potential compactions"),
Review Comment:
You should add a range validator to prevent extreme values. sth like
100-5000.
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java:
##########
@@ -204,7 +206,7 @@ public Set<CompactionInfo> findPotentialCompactions(int
abortedThreshold,
") \"C\" " +
"ON \"TC\".\"CTC_DATABASE\" = \"C\".\"CC_DATABASE\" AND
\"TC\".\"CTC_TABLE\" = \"C\".\"CC_TABLE\" " +
" AND (\"TC\".\"CTC_PARTITION\" = \"C\".\"CC_PARTITION\" OR
(\"TC\".\"CTC_PARTITION\" IS NULL AND \"C\".\"CC_PARTITION\" IS NULL)) " +
- "WHERE \"C\".\"CC_ID\" IS NOT NULL OR " +
isWithinCheckInterval("\"TC\".\"CTC_TIMESTAMP\"", checkInterval) : "");
+ "WHERE \"C\".\"CC_ID\" IS NOT NULL OR " +
isWithinCheckInterval("\"TC\".\"CTC_TIMESTAMP\"", checkInterval) : ""));
LOG.debug("Going to execute query <{}>", s);
rs = stmt.executeQuery(s);
Review Comment:
statement fetch size should be adjusted accordingly to the limit, to prevent
multiple roundtrips. Since every row will be read anyway, there's no memory
overhead of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
