[
https://issues.apache.org/jira/browse/HBASE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642657#comment-16642657
]
Andrew Purtell edited comment on HBASE-21000 at 10/9/18 12:59 AM:
------------------------------------------------------------------
I did a fairly quick test on a 5 node test cluster. I used PE sequentialWrite
to write 1M rows x 10 threads and monitored when compaction activity started
and stopped running on any/all servers at 10 second granularity using
getClusterStatus.
1.4.8 completes compaction in 10 seconds
1.5.0-SNAPSHOT, depending on the values of
hbase.hstore.compaction.throughput.lower.bound and
hbase.hstore.compaction.throughput.higher.bound, completes compaction in 1x to
7x the amount of time:
||Lower bound||Upper bound||Time||
|default|default| 70 sec|
|20971520 (20M)| 41943040 (40M)| 40 sec|
|26214400 (25M)| 52428800 (50M)| 30 sec|
|52428800 (50M)|104857600 (100M)| 10 sec|
This is a simple test. Production results will depend on load and schema.
However as a first pass it seems we should add these defaults to
hbase-default.xml
{{hbase.hstore.compaction.throughput.lower.bound = 52428800}}
{{hbase.hstore.compaction.throughput.higher.bound = 104857600}}
Let me put up a simple patch that does this.
was (Author: apurtell):
I did a fairly quick test on a 5 node test cluster. I used PE sequentialWrite
to write 1M rows x 10 threads and monitored when compaction activity started
and stopped running on any/all servers at 10 second granularity using
getClusterStatus.
1.4.8 completes compaction in 10 seconds
1.5.0-SNAPSHOT, depending on the values of
hbase.hstore.compaction.throughput.lower.bound and
hbase.hstore.compaction.throughput.higher.bound, completes compaction in 1x to
7x the amount of time:
||Lower bound||Upper bound||Time||
|default|default| 70 sec|
|20971520 (20M)| 41943040 (40M)| 40 sec|
|26214400 (25M)| 52428800 (50M)| 30 sec|
|52428800 (50M)|104857600 (100M)| 10 sec|
This is a simple test. Production results will depend on load and schema.
However as a first pass it seems we should add these defaults to
hbase-default.xml
{{}}{{hbase.hstore.compaction.throughput.lower.bound = }}{{52428800}}
{{}}{{hbase.hstore.compaction.throughput.higher.bound = }}{{104857600}}
Let me put up a simple patch that does this.
> Default limits for PressureAwareCompactionThroughputController are too low
> --------------------------------------------------------------------------
>
> Key: HBASE-21000
> URL: https://issues.apache.org/jira/browse/HBASE-21000
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 1.5.0
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
>
> In PressureAwareCompactionThroughputController:
> {code:java}
> /**
>
>
> * A throughput controller which uses the follow schema to limit throughput
>
>
> * <ul>
>
>
> * <li>If compaction pressure is greater than 1.0, no limitation.</li>
>
>
> * <li>In off peak hours, use a fixed throughput limitation
>
>
> * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_OFFPEAK}</li>
>
>
> * <li>In normal hours, the max throughput is tuned between
>
>
> * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND} and
>
>
> * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND}, using the
> formula "lower +
>
> * (higher - lower) * compactionPressure", where compactionPressure is
> in range [0.0, 1.0]</li>
>
> * </ul>
>
>
> */
> {code}
> The lower and upper bounds are 10MB/sec and 20MB/sec, respectively:
> {code:java}
> public static final String
> HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND =
> "hbase.hstore.compaction.throughput.higher.bound";
> private static final long
> DEFAULT_HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND =
> 20L * 1024 * 1024;
> public static final String
> HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND =
> "hbase.hstore.compaction.throughput.lower.bound";
> private static final long
> DEFAULT_HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND =
> 10L * 1024 * 1024;
> {code}
> (In contrast, in PressureAwareFlushThroughputController the lower and upper
> bounds are 10x of those limits, at 100MB/sec and 200MB/sec, respectively.)
> In fairly light load scenarios we see compaction quickly falls behind and
> write clients are backed off or failing due to RegionTooBusy exceptions.
> Although compaction throughput becomes unbounded after the store reaches the
> blocking file count, in the lead up to this the default settings do not
> provide enough bandwidth to stave off blocking. The defaults should be
> increased.
> I'm not sure what new good defaults make sense. We could start by doubling
> them to 20MB/sec and 40MB/sec respectively. Might need to be doubled again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
