[ https://issues.apache.org/jira/browse/HBASE-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13712577#comment-13712577 ]
Varun Sharma commented on HBASE-8599: ------------------------------------- 0.94 patch looks good. Thanks [~lhofhansl] > HLogs in ZK are not cleaned up when replication lag is minimal > -------------------------------------------------------------- > > Key: HBASE-8599 > URL: https://issues.apache.org/jira/browse/HBASE-8599 > Project: HBase > Issue Type: Improvement > Components: Replication > Affects Versions: 0.98.0, 0.94.7 > Reporter: Varun Sharma > Assignee: Varun Sharma > Fix For: 0.98.0, 0.95.2, 0.94.10 > > Attachments: 8599-0.94.patch, 8599-0.94-v2.txt, 8599-trunk.patch, > 8599-trunk-v2.patch, 8599-trunk-v3.patch > > > On a cluster with very low replication lag (as measured by ageOfLastShippedOp > on source), we found HLogs accumulating and not being cleaned up as new > WAL(s) are rolled. > Each time, we call logPositionAndCleanOldLogs() to clean older logs whenever > the current WAL is not being written to any more - as suggested by > currentWALBeingWrittenTo being false. However, when lags are small, we may > hit the following block first and continue onto the next WAL without clearing > the old WAL(s)... > ReplicationSource::run() { > if (readAllEntriesToReplicateOrNextFile(currentWALisBeingWrittenTo = > false)) { > // If we are here, then we advance to the next WAL without any > cleaning > // and close existing WAL > continue; > } > // Ship some edits and call logPositionAndCleanOldLogs > } > If we hit readAllEntriesToReplicateOrNextFile(false) only once - then older > logs are not cleaned out and persist in the zookeeper node since we simply > call "continue" and skip the subsequent logPositionAndCleanOldLogs call - if > its called more than once, we do end up clearing the old logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira