[ https://issues.apache.org/jira/browse/HBASE-9208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13750666#comment-13750666 ]
Dave Latham commented on HBASE-9208: ------------------------------------ Thanks for the attention Stack, and JD. {quote}Trunk patch looks good. Does what you have deployed look like this patch Dave Latham?{quote} What we deployed is a bit different. Added a startBatch and endBatch to the FileCleanerDelegate. That made it a bit less clean of an interface, which also has some hidden dependencies on client behavior (otherwise would have race conditions). However it was a smaller change that we needed rapidly that didn't require a refactor of CleanerChore like this patch does. I like this patch better and look forward to running it if it looks good to others. {quote}It doesn't have unit tests. You are relying on existing tests to ensure that base functionality remains undisturbed?{quote} That's right. {quote}Rather than indent whole method, just reverse this test and return early: if (entries != null) { {quote} I guess it depends on whether you prefer having indentation or an extra return statement. I'll add the extra return since you spoke up. {quote} Is this right? Delete all files if not configured? ... What it means is that this log cleaner says ok for those files, but other log cleaners can say otherwise. As long as one cleaner in the chain says "keep it", we keep it. But I'm not sure about this WARN, it could just be removed: LOG.warn("Not configured - allowing all hlogs to be deleted"); {quote} This case existed in the previous code with the same behavior that JD explains. However it can only occur if someone manually configures HBase to use the ReplicationLogCleaner but has replication disabled (normally the ReplicationLogCleaner is added programmatically in Replication.decorateMasterConfiguration when replication is enabled). The warning could prove useful to someone in such a case. However I will move it to the setConf method so it is only logged once at startup if there is concern over its level and verbosity. > ReplicationLogCleaner slow at large scale > ----------------------------------------- > > Key: HBASE-9208 > URL: https://issues.apache.org/jira/browse/HBASE-9208 > Project: HBase > Issue Type: Improvement > Components: Replication > Reporter: Dave Latham > Assignee: Dave Latham > Fix For: 0.94.12, 0.96.0 > > Attachments: HBASE-9208-0.94.patch, HBASE-9208.patch, > HBASE-9208-v2.patch > > > At a large scale the ReplicationLogCleaner fails to clean up .oldlogs as fast > as the cluster is producing them. For each old HLog file that has been > replicated and should be deleted the ReplicationLogCleaner checks every > replication queue in ZooKeeper before removing it. This means that as a > cluster scales up the number of files to delete scales as well as the time to > delete each file so the cleanup chore scales quadratically. In our case it > reached the point where the oldlogs were growing faster than they were being > cleaned up. > We're now running with a patch that allows the ReplicationLogCleaner to > refresh its list of files in the replication queues from ZooKeeper just once > for each batch of files the CleanerChore wants to evaluate. > I'd propose updating FileCleanerDelegate to take a List<FileStatus> rather > than a single one at a time. This would allow file cleaners that check an > external resource for references such as ZooKeeper (for > ReplicationLogCleaner) or HDFS (for SnapshotLogCleaner which looks like it > may also have similar trouble at scale) to load those references once per > batch rather than for every log. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira