[jira] [Commented] (HIVE-27637) Compare highest write ID of compaction records when trying to perform abort cleanup
[ https://issues.apache.org/jira/browse/HIVE-27637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17813677#comment-17813677 ] Sourabh Badhya commented on HIVE-27637: --- The commit is reverted via https://github.com/apache/hive/pull/5058. > Compare highest write ID of compaction records when trying to perform abort > cleanup > --- > > Key: HIVE-27637 > URL: https://issues.apache.org/jira/browse/HIVE-27637 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Zsolt Miskolczi >Assignee: Zsolt Miskolczi >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Compare highest write ID of compaction records when trying to get the > potential table/partitions for abort cleanup. > Idea: If there exists a highest write ID of a record in COMPACTION_QUEUE for > a table/partition which is greater than the max(aborted write ID) for that > table/partition, then we can potentially ignore abort cleanup for such > tables/partitions. This is because compaction will perform cleanup of > obsolete deltas and aborted deltas hence doing abort cleanup is redundant > here. > This is more of an optimisation since it can potentially save some filesystem > operations (mainly file-listing during construction of Acid state). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27637) Compare highest write ID of compaction records when trying to perform abort cleanup
[ https://issues.apache.org/jira/browse/HIVE-27637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17813561#comment-17813561 ] Zsolt Miskolczi commented on HIVE-27637: [~sbadhya] , thank you for the idea of improvement and a review. > Compare highest write ID of compaction records when trying to perform abort > cleanup > --- > > Key: HIVE-27637 > URL: https://issues.apache.org/jira/browse/HIVE-27637 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Zsolt Miskolczi >Assignee: Zsolt Miskolczi >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Compare highest write ID of compaction records when trying to get the > potential table/partitions for abort cleanup. > Idea: If there exists a highest write ID of a record in COMPACTION_QUEUE for > a table/partition which is greater than the max(aborted write ID) for that > table/partition, then we can potentially ignore abort cleanup for such > tables/partitions. This is because compaction will perform cleanup of > obsolete deltas and aborted deltas hence doing abort cleanup is redundant > here. > This is more of an optimisation since it can potentially save some filesystem > operations (mainly file-listing during construction of Acid state). -- This message was sent by Atlassian Jira (v8.20.10#820010)