[
https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-18309:
--------------------------
Hadoop Flags: Reviewed
Release Note: After HBASE-18309 we could use multiple threads to scan
archive directories (including data and oldWALs) through config
hbase.cleaner.scan.dir.concurrent.size, which supports both integer (meaning
the concrete size) and double (between 0 and 1, meaning ratio of available cpu
cores) value and defaults to 0.5. Please take
hbase.regionserver.hfilecleaner.large.thread.count and
hbase.regionserver.hfilecleaner.small.thread.count into account when setting
this config to avoid thread flooding. We also support using multiple threads to
clean wals in a single directory through hbase.oldwals.cleaner.thread.size, 2
by default.
Fix Version/s: 2.0.0-beta-1
3.0.0
Description:
There is only one thread in LogCleaner to clean oldWALs and in our big cluster
we find this is not enough. The number of files under oldWALs reach the
max-directory-items limit of HDFS and cause region server crash, so we use
multi threads for LogCleaner and the crash not happened any more.
What's more, currently there's only one thread iterating the archive directory,
and we could use multiple threads cleaning sub directories in parallel to speed
it up.
was:There is only one thread in LogCleaner to clean oldWALs and in our big
cluster we find this is not enough. The number of files under oldWALs reach the
max-directory-items limit of HDFS and cause region server crash, so we use
multi threads for LogCleaner and the crash not happened any more.
Component/s: (was: wal)
[~reidchan] please check the release note and feel free to refine it. It's
recommended to add release note when introducing new properties, so people
could better know how to use them.
> Support multi threads in CleanerChore
> -------------------------------------
>
> Key: HBASE-18309
> URL: https://issues.apache.org/jira/browse/HBASE-18309
> Project: HBase
> Issue Type: Improvement
> Reporter: binlijin
> Assignee: Reid Chan
> Fix For: 3.0.0, 2.0.0-beta-1
>
> Attachments: HBASE-18309.master.001.patch,
> HBASE-18309.master.002.patch, HBASE-18309.master.004.patch,
> HBASE-18309.master.005.patch, HBASE-18309.master.006.patch,
> HBASE-18309.master.007.patch, HBASE-18309.master.008.patch,
> HBASE-18309.master.009.patch, HBASE-18309.master.010.patch,
> HBASE-18309.master.011.patch, HBASE-18309.master.012.patch,
> space_consumption_in_archive.png
>
>
> There is only one thread in LogCleaner to clean oldWALs and in our big
> cluster we find this is not enough. The number of files under oldWALs reach
> the max-directory-items limit of HDFS and cause region server crash, so we
> use multi threads for LogCleaner and the crash not happened any more.
> What's more, currently there's only one thread iterating the archive
> directory, and we could use multiple threads cleaning sub directories in
> parallel to speed it up.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
