[
https://issues.apache.org/jira/browse/HBASE-9208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13744172#comment-13744172
]
Hadoop QA commented on HBASE-9208:
----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12598799/HBASE-9208-0.94.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6814//console
This message is automatically generated.
> ReplicationLogCleaner slow at large scale
> -----------------------------------------
>
> Key: HBASE-9208
> URL: https://issues.apache.org/jira/browse/HBASE-9208
> Project: HBase
> Issue Type: Improvement
> Components: Replication
> Reporter: Dave Latham
> Assignee: Dave Latham
> Fix For: 0.94.12, 0.96.0
>
> Attachments: HBASE-9208-0.94.patch, HBASE-9208.patch,
> HBASE-9208-v2.patch
>
>
> At a large scale the ReplicationLogCleaner fails to clean up .oldlogs as fast
> as the cluster is producing them. For each old HLog file that has been
> replicated and should be deleted the ReplicationLogCleaner checks every
> replication queue in ZooKeeper before removing it. This means that as a
> cluster scales up the number of files to delete scales as well as the time to
> delete each file so the cleanup chore scales quadratically. In our case it
> reached the point where the oldlogs were growing faster than they were being
> cleaned up.
> We're now running with a patch that allows the ReplicationLogCleaner to
> refresh its list of files in the replication queues from ZooKeeper just once
> for each batch of files the CleanerChore wants to evaluate.
> I'd propose updating FileCleanerDelegate to take a List<FileStatus> rather
> than a single one at a time. This would allow file cleaners that check an
> external resource for references such as ZooKeeper (for
> ReplicationLogCleaner) or HDFS (for SnapshotLogCleaner which looks like it
> may also have similar trouble at scale) to load those references once per
> batch rather than for every log.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
