[jira] [Comment Edited] (HBASE-16524) Clean procedure wal periodically instead of on every sync
[ https://issues.apache.org/jira/browse/HBASE-16524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15468754#comment-15468754 ] Appy edited comment on HBASE-16524 at 9/6/16 10:09 PM: --- Thinking about it, that should be easy to do. In our current logic, we basically do this: 1. make a copy of current global tracker, say T. 2. Iterate over log files: - keep the file if contain an update to proc P which is not marked deleted in T. - Delete P from T so we don't hold another older log file because of it. The new steps would be: 1. make a copy of current global tracker, say T. 2. *Mark all recently updated procs as deleted in T (so we don't hold logs because of these procs)* 3. (step 2 above in old logic) Iterate over log files: - keep the file if contain an update to proc P which is not marked deleted in T. - Delete P from T so we don't hold another older log file because of it. Sounds good? was (Author: appy): Thinking about it, that should be easy to do. In our current logic, we basically do this: 1. make a copy of current global tracker, say T. 2. Iterate over log files: - keep the file if contain an update to proc P which is not marked deleted in T. - Delete P from T so we don't hold another older log file because of it. We can simply add step 0, which marks all recently updated procs as deleted in T. Sounds good? > Clean procedure wal periodically instead of on every sync > - > > Key: HBASE-16524 > URL: https://issues.apache.org/jira/browse/HBASE-16524 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-16524.master.001.patch, flame1.svg > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16524) Clean procedure wal periodically instead of on every sync
[ https://issues.apache.org/jira/browse/HBASE-16524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15457397#comment-15457397 ] Matteo Bertozzi edited comment on HBASE-16524 at 9/2/16 3:42 AM: - before adding conditions to reduce the number of times we call removeInactiveLogs(), let's try to improve the algorithm. executing every roll does not provide the result we want. e.g. WAL-1: [1, 2], Current: [] as soon we update [1, 2] we can remove WAL-1 e.g. WAL-1: [1], WAL-2: [2], Current [] as soon we update [1, 2] we can remove WAL-1 and WAL-2 was (Author: mbertozzi): before adding conditions to reduce the number of times we call removeInactiveLogs(), let's try to improve the algorithm. executing every roll does not provide the result we want. e.g. WAL-1: [1, 2] Current: [] as soon we update [1, 2] we can remove WAL-1 e.g. WAL-1: [1], WAL-2: [2] as soon we update [1, 2] we can remove WAL-1 and WAL-2 > Clean procedure wal periodically instead of on every sync > - > > Key: HBASE-16524 > URL: https://issues.apache.org/jira/browse/HBASE-16524 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-16524.master.001.patch, flame1.svg > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
