hbase git commit: HBASE-17089 Add doc on experience running with hedged reads; ADDENDUM adding in Ashu Pachauri's experience
Repository: hbase Updated Branches: refs/heads/master c7bdb3017 -> 86df89b01 HBASE-17089 Add doc on experience running with hedged reads; ADDENDUM adding in Ashu Pachauri's experience Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/86df89b0 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/86df89b0 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/86df89b0 Branch: refs/heads/master Commit: 86df89b01608052dad4ef75abde5a3fe79447ac0 Parents: c7bdb30 Author: Michael Stack Authored: Mon Nov 14 21:06:29 2016 -0800 Committer: Michael Stack Committed: Mon Nov 14 21:06:29 2016 -0800 -- src/main/asciidoc/_chapters/performance.adoc | 31 +-- 1 file changed, 23 insertions(+), 8 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/86df89b0/src/main/asciidoc/_chapters/performance.adoc -- diff --git a/src/main/asciidoc/_chapters/performance.adoc b/src/main/asciidoc/_chapters/performance.adoc index bc8b48a..114754f 100644 --- a/src/main/asciidoc/_chapters/performance.adoc +++ b/src/main/asciidoc/_chapters/performance.adoc @@ -760,11 +760,30 @@ See the _Development Process_ section of the document link:https://issues.apache Hedged reads are a feature of HDFS, introduced in Hadoop 2.4.0 with link:https://issues.apache.org/jira/browse/HDFS-5776[HDFS-5776]. Normally, a single thread is spawned for each read request. -However, if hedged reads are enabled, the client waits some configurable amount of time, and if the read does not return, the client spawns a second read request, against a different block replica of the same data. -Whichever read returns first is used, and the other read request is discarded. -Hedged reads can be helpful for times where a rare slow read is caused by a transient error such as a failing disk or flaky network connection. +However, if hedged reads are enabled, the client waits some +configurable amount of time, and if the read does not return, +the client spawns a second read request, against a different +block replica of the same data. Whichever read returns first is +used, and the other read request is discarded. + +Hedged reads are "...very good at eliminating outlier datanodes, which +in turn makes them very good choice for latency sensitive setups. +But, if you are looking for maximizing throughput, hedged reads tend to +create load amplification as things get slower in general. In short, +the thing to watch out for is the non-graceful performance degradation +when you are running close a certain throughput threshold." (Quote from Ashu Pachauri in HBASE-17083). + +Other concerns to keep in mind while running with hedged reads enabled +include: -Because an HBase RegionServer is a HDFS client, you can enable hedged reads in HBase, by adding the following properties to the RegionServer's hbase-site.xml and tuning the values to suit your environment. +* They may lead to network congestion. See link:https://issues.apache.org/jira/browse/HBASE-17083[HBASE-17083] +* Make sure you set the thread pool large enough so as blocking on the pool does not become a bottleneck (Again see link:https://issues.apache.org/jira/browse/HBASE-17083[HBASE-17083]) + +(From Yu Li up in HBASE-17083) + +Because an HBase RegionServer is a HDFS client, you can enable hedged +reads in HBase, by adding the following properties to the RegionServer's +hbase-site.xml and tuning the values to suit your environment. .Configuration for Hedged Reads * `dfs.client.hedged.read.threadpool.size` - the number of threads dedicated to servicing hedged reads. @@ -795,10 +814,6 @@ See <> for more information. * hedgeReadOpsWin - the number of times the hedged read thread was faster than the original thread. This could indicate that a given RegionServer is having trouble servicing requests. -Concerns while running with hedged reads enabled include: - -* They may lead to network congestion. See link:https://issues.apache.org/jira/browse/HBASE-17083[HBASE-17083] -* Make sure you set the thread pool large enough so as blocking on the pool does not become a bottleneck (Again see link:https://issues.apache.org/jira/browse/HBASE-17083[HBASE-17083]) [[perf.deleting]] == Deleting from HBase
hbase git commit: HBASE-17089 Add doc on experience running with hedged reads
Repository: hbase Updated Branches: refs/heads/master c3685760f -> 6ee2dcf48 HBASE-17089 Add doc on experience running with hedged reads Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/6ee2dcf4 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/6ee2dcf4 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/6ee2dcf4 Branch: refs/heads/master Commit: 6ee2dcf480dd95877a20e33086a020eb1a19e41f Parents: c368576 Author: Michael Stack Authored: Mon Nov 14 10:27:58 2016 -0800 Committer: Michael Stack Committed: Mon Nov 14 10:28:09 2016 -0800 -- src/main/asciidoc/_chapters/performance.adoc | 12 +++- src/main/asciidoc/_chapters/protobuf.adoc| 5 +++-- 2 files changed, 14 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/6ee2dcf4/src/main/asciidoc/_chapters/performance.adoc -- diff --git a/src/main/asciidoc/_chapters/performance.adoc b/src/main/asciidoc/_chapters/performance.adoc index 5f27640..bc8b48a 100644 --- a/src/main/asciidoc/_chapters/performance.adoc +++ b/src/main/asciidoc/_chapters/performance.adoc @@ -156,6 +156,10 @@ See <> See <>. +[[perf.99th.percentile]] +=== Improving the 99th Percentile +Try link:[hedged_reads]. + [[perf.compactions.and.splits]] === Managing Compactions @@ -751,9 +755,10 @@ Most people should leave this alone. Default = 7, or can collapse to at least 1/128th of original size. See the _Development Process_ section of the document link:https://issues.apache.org/jira/secure/attachment/12444007/Bloom_Filters_in_HBase.pdf[BloomFilters in HBase] for more on what this option means. +[[hedged.reads]] === Hedged Reads -Hedged reads are a feature of HDFS, introduced in link:https://issues.apache.org/jira/browse/HDFS-5776[HDFS-5776]. +Hedged reads are a feature of HDFS, introduced in Hadoop 2.4.0 with link:https://issues.apache.org/jira/browse/HDFS-5776[HDFS-5776]. Normally, a single thread is spawned for each read request. However, if hedged reads are enabled, the client waits some configurable amount of time, and if the read does not return, the client spawns a second read request, against a different block replica of the same data. Whichever read returns first is used, and the other read request is discarded. @@ -790,6 +795,11 @@ See <> for more information. * hedgeReadOpsWin - the number of times the hedged read thread was faster than the original thread. This could indicate that a given RegionServer is having trouble servicing requests. +Concerns while running with hedged reads enabled include: + +* They may lead to network congestion. See link:https://issues.apache.org/jira/browse/HBASE-17083[HBASE-17083] +* Make sure you set the thread pool large enough so as blocking on the pool does not become a bottleneck (Again see link:https://issues.apache.org/jira/browse/HBASE-17083[HBASE-17083]) + [[perf.deleting]] == Deleting from HBase http://git-wip-us.apache.org/repos/asf/hbase/blob/6ee2dcf4/src/main/asciidoc/_chapters/protobuf.adoc -- diff --git a/src/main/asciidoc/_chapters/protobuf.adoc b/src/main/asciidoc/_chapters/protobuf.adoc index fa63127..1c2cc47 100644 --- a/src/main/asciidoc/_chapters/protobuf.adoc +++ b/src/main/asciidoc/_chapters/protobuf.adoc @@ -27,6 +27,8 @@ :icons: font :experimental: + +== Protobuf HBase uses Google's link:http://protobuf.protobufs[protobufs] wherever it persists metadata -- in the tail of hfiles or Cells written by HBase into the system hbase;meta table or when HBase writes znodes @@ -36,12 +38,11 @@ Interfaces (Services) we expose to clients, for example the `Admin` and `Client` Interfaces that the RegionServer fields, or specifying the arbitrary extensions added by developers via our xref:cp[Coprocessor Endpoint] mechanism. + In this chapter we go into detail for developers who are looking to understand better how it all works. This chapter is of particular use to those who would amend or extend HBase functionality. -== Protobuf - With protobuf, you describe serializations and services in a `.protos` file. You then feed these descriptors to a protobuf tool, the `protoc` binary, to generate classes that can marshall and unmarshall the described serializations
