[
https://issues.apache.org/jira/browse/HBASE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643792#comment-16643792
]
Andrew Purtell commented on HBASE-21000:
----------------------------------------
[~taklwu]
I used a basic test setup, 1 master+nn+rm, 5 rs+dn+nm, 1 client, all of them
c3.8xlarge instance types with datanode blocks stored on local ephemeral disks,
HBase 1.5.0-SNAPSHOT, Hadoop 2.7.7. The schema is your basic
PerformanceEvaluation schema but with ROW_INDEX_V1 block encoding and ZSTD
compression, test table presplit into 10 regions deployed evenly over the 5
RSes. The schema specifics are not germane, IMHO, because they were the same in
all tests and the relative differences in compaction completion time between
versions and settings is what is interesting. The compaction completion time is
dominated by the throughput controller sleeps when it is active.
bq. I think this change make senses especially if one has a larger instance.
and in master/branch-2 when IMC is enabled
branch-2 has different considerations, like the in memory compaction option,
and has releases with the pressure aware controller and current defaults, so
I'm not sure if this change should go in there and I plan to only commit the
branch-1 patch, then open a subtask for a forward port to branch-2 and master,
unless [~stack] would want it in right away. (I don't think he would.)
> Default limits for PressureAwareCompactionThroughputController are too low
> --------------------------------------------------------------------------
>
> Key: HBASE-21000
> URL: https://issues.apache.org/jira/browse/HBASE-21000
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 1.5.0
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-21000-branch-1.patch, HBASE-21000.patch
>
>
> In PressureAwareCompactionThroughputController:
> {code:java}
> /**
>
>
> * A throughput controller which uses the follow schema to limit throughput
>
>
> * <ul>
>
>
> * <li>If compaction pressure is greater than 1.0, no limitation.</li>
>
>
> * <li>In off peak hours, use a fixed throughput limitation
>
>
> * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_OFFPEAK}</li>
>
>
> * <li>In normal hours, the max throughput is tuned between
>
>
> * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND} and
>
>
> * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND}, using the
> formula "lower +
>
> * (higher - lower) * compactionPressure", where compactionPressure is
> in range [0.0, 1.0]</li>
>
> * </ul>
>
>
> */
> {code}
> The lower and upper bounds are 10MB/sec and 20MB/sec, respectively:
> {code:java}
> public static final String
> HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND =
> "hbase.hstore.compaction.throughput.higher.bound";
> private static final long
> DEFAULT_HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND =
> 20L * 1024 * 1024;
> public static final String
> HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND =
> "hbase.hstore.compaction.throughput.lower.bound";
> private static final long
> DEFAULT_HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND =
> 10L * 1024 * 1024;
> {code}
> (In contrast, in PressureAwareFlushThroughputController the lower and upper
> bounds are 10x of those limits, at 100MB/sec and 200MB/sec, respectively.)
> In fairly light load scenarios we see compaction quickly falls behind and
> write clients are backed off or failing due to RegionTooBusy exceptions.
> Although compaction throughput becomes unbounded after the store reaches the
> blocking file count, in the lead up to this the default settings do not
> provide enough bandwidth to stave off blocking. The defaults should be
> increased.
> I'm not sure what new good defaults make sense. We could start by doubling
> them to 20MB/sec and 40MB/sec respectively. Might need to be doubled again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
