[
https://issues.apache.org/jira/browse/HBASE-26304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433138#comment-17433138
]
Bryan Beaudreault commented on HBASE-26304:
-------------------------------------------
Unfortunately there isn't an easy way to achieve the 3rd option. I could
potentially add something to DFSInputStream, but we'd be stuck waiting for it
to get backported to all supported hadoop versions. I also realized that it's
very possible even without the locality healer for blocks to move, and we don't
really reflect that at all today. I ended up going with option 1 above, as the
easiest option and also most off the critical path.
I created a LocalityMetricsRefreshChore which periodically refreshes the
HDFSBlockDistribution for all stores on the server. I thought about limiting it
to only non-100% locality stores, but decided against it so that we can also
reflect locality _regressions_ due to datanodes dying or if someone ran
Balancer or Mover. For anyone monitoring or alerting on locality, it's just as
important to know if something just totally tanked locality as it is to knowing
whether it was improved by something like the healer.
I'm doing some testing, but will push a PR with this new chore probably next
week.
> Reflect out-of-band locality improvements in served requests
> ------------------------------------------------------------
>
> Key: HBASE-26304
> URL: https://issues.apache.org/jira/browse/HBASE-26304
> Project: HBase
> Issue Type: Sub-task
> Reporter: Bryan Beaudreault
> Assignee: Bryan Beaudreault
> Priority: Major
>
> Once the LocalityHealer has improved locality of a StoreFile (by moving
> blocks onto the correct host), the Reader's DFSInputStream and Region's
> localityIndex metric must be refreshed. Without refreshing the
> DFSInputStream, the improved locality will not improve latencies. In fact,
> the DFSInputStream may try to fetch blocks that have moved, resulting in a
> ReplicaNotFoundException. This is automatically retried, but the retry will
> increase long tail latencies relative to configured backoff strategy.
> See https://issues.apache.org/jira/browse/HDFS-16155 for an improvement in
> backoff strategy which can greatly mitigate latency impact of the missing
> block retry.
> Even with that mitigation, a StoreFile is often made up of many blocks.
> Without some sort of intervention, we will continue to hit
> ReplicaNotFoundException over time as clients naturally request data from
> moved blocks.
> In the original LocalityHealer design, I created a new
> RefreshHDFSBlockDistribution RPC on the RegionServer. This RPC accepts a list
> of region names and, for each region store, re-opens the underlying StoreFile
> if the locality has changed.
> I will submit a PR with that implementation, but I am also investigating
> other avenues. For example, I noticed
> https://issues.apache.org/jira/browse/HDFS-15119 which doesn't seem ideal but
> maybe can be improved as an automatic lower-level handling of block moves.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
