[
https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reid Chan updated HBASE-18309:
------------------------------
Release Note:
After HBASE-18309,
1. we could use multiple threads to scan archive directories (including data
and oldWALs) through config hbase.cleaner.scan.dir.concurrent.size, which
supports both integer (meaning the concrete size, but no more than available
cpu cores) and double (between 0 and 1, meaning ratio of available cpu cores)
value and defaults to 0.5.
2. We also support using multiple threads to clean wals under oldWALs directory
through hbase.oldwals.cleaner.thread.size, 2 by default.
3. In addition, hbase.cleaner.scan.dir.concurrent.size and
hbase.oldwals.cleaner.thread.size support online re-configuration as
hbase.regionserver.hfilecleaner.large.thread.count and
hbase.regionserver.hfilecleaner.small.thread.count does.
4. Please take hbase.cleaner.scan.dir.concurrent.size,
hbase.regionserver.hfilecleaner.large.thread.count,
hbase.regionserver.hfilecleaner.small.thread.count and
hbase.oldwals.cleaner.thread.size into account when setting this config to
avoid thread flooding.
was:
After HBASE-18309 we could use multiple threads to scan archive directories
(including data and oldWALs) through config
hbase.cleaner.scan.dir.concurrent.size, which supports both integer (meaning
the concrete size, but no more than available cpu cores) and double (between 0
and 1, meaning ratio of available cpu cores) value and defaults to 0.5.
We also support using multiple threads to clean wals under oldWALs directory
through hbase.oldwals.cleaner.thread.size, 2 by default.
Please take hbase.cleaner.scan.dir.concurrent.size,
hbase.regionserver.hfilecleaner.large.thread.count,
hbase.regionserver.hfilecleaner.small.thread.count and
hbase.oldwals.cleaner.thread.size into account when setting this config to
avoid thread flooding.
In addition, hbase.cleaner.scan.dir.concurrent.size and
hbase.oldwals.cleaner.thread.size support online re-configuration as
hbase.regionserver.hfilecleaner.large.thread.count and
hbase.regionserver.hfilecleaner.small.thread.count does.
> Support multi threads in CleanerChore
> -------------------------------------
>
> Key: HBASE-18309
> URL: https://issues.apache.org/jira/browse/HBASE-18309
> Project: HBase
> Issue Type: Improvement
> Reporter: binlijin
> Assignee: Reid Chan
> Fix For: 3.0.0, 2.0.0-beta-1
>
> Attachments: HBASE-18309.master.001.patch,
> HBASE-18309.master.002.patch, HBASE-18309.master.004.patch,
> HBASE-18309.master.005.patch, HBASE-18309.master.006.patch,
> HBASE-18309.master.007.patch, HBASE-18309.master.008.patch,
> HBASE-18309.master.009.patch, HBASE-18309.master.010.patch,
> HBASE-18309.master.011.patch, HBASE-18309.master.012.patch,
> space_consumption_in_archive.png
>
>
> There is only one thread in LogCleaner to clean oldWALs and in our big
> cluster we find this is not enough. The number of files under oldWALs reach
> the max-directory-items limit of HDFS and cause region server crash, so we
> use multi threads for LogCleaner and the crash not happened any more.
> What's more, currently there's only one thread iterating the archive
> directory, and we could use multiple threads cleaning sub directories in
> parallel to speed it up.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
