[
https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13882606#comment-13882606
]
Liang Xie commented on HDFS-5776:
---------------------------------
[~saint....@gmail.com]
bq. If we are too often running the requests in the current thread because the
executor has none to spare then we can up the number of pool threads (though it
requires a DN restart, a PITA)?
we don't need to restart DN/RS or sth else, we can modify/introduce a hbase
shell script to disable/enable the feature per instance or modify the thread
number or other requirements, i think it's feasible, and those works, in deed,
are the major task for supporting hedged read in HBase side:)
bq. nit: You could declare and assign in the one go rather than postpone the
assign to the constructor: HEDGED_READ_METRIC = new DFSHedgedReadMetrics();
good suggestion, let me fix it in patch v9.
[~arpitagarwal]
bq. Threads are not free. If you really want to provide a user configurable
setting for the thread count there should be a limit on the order of 64/128. I
leave the exact number to you.
Fine, let me introduce a hard code up-limit for
DFS_DFSCLIENT_HEDGED_READ_THREADPOOL_SIZE to 128.
bq. The same goes for the delay. Please add a lower bound. The exact value is
up to you. We can always revisit the value if it turns out to be a bottleneck.
Fine, let me introduce a hard code down-limit for
DFS_DFSCLIENT_HEDGED_READ_THRESHOLD_MILLIS to 1ms
bq. I think it would be best to leave a single config setting i.e. either a
boolean or a thread count, and a single method #isHedgedReadsEnabled to query
the status of the feature.
Yes, that would be perfect sometimes, but not works for HBase scenario(the
above Stack's consideration is great), since we made the pool "static", and per
client view, it's more flexible if we provide instance level disable/enable
APIs, so we can archive to use the hbase shell script to control the switch per
dfs client instance, that'll be cooler:)
bq. I still do not understand how you are guarding against multiple refetch.
Previously these counters were initialized outside any loop, now they are being
reinitialized inside a loop.
In actualGetFromOneDatanode(), the refetchToken/refetchEncryptionKey is
initialized outside the while (true) loop (see Line 993-996), when we hit
InvalidEncryptionKeyException/InvalidBlockTokenException, the refetchToken and
refetchEncryptionKey will be decreased by 1, (see refetchEncryptionKey-- and
refetchToken-- statement), if the exceptions happened again, the check
conditions will be failed definitely(see "e instanceof
InvalidEncryptionKeyException && refetchEncryptionKey > 0" and "refetchToken >
0"), so go to the else clause, that'll execute:
{code}
String msg = "Failed to connect to " + targetAddr + " for file "
+ src + " for block " + block.getBlock() + ":" + e;
DFSClient.LOG.warn("Connection failure: " + msg, e);
addToDeadNodes(chosenNode);
throw new IOException(msg);
{code}
so later, if we chooseDataNode, that dead node will be ignored. Hopefully this
time my description is more clear than before:)
bq. chooseDataNode(LocatedBlock block) function looks redundant and should be
removed.
it still be called by blockSeekTo(long) and fetchBlockByteRange(...), yes, we
can remove it, let me fix it in patch v9.
> Support 'hedged' reads in DFSClient
> -----------------------------------
>
> Key: HDFS-5776
> URL: https://issues.apache.org/jira/browse/HDFS-5776
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
> Affects Versions: 3.0.0
> Reporter: Liang Xie
> Assignee: Liang Xie
> Attachments: HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt,
> HDFS-5776-v5.txt, HDFS-5776-v6.txt, HDFS-5776-v7.txt, HDFS-5776-v8.txt,
> HDFS-5776.txt
>
>
> This is a placeholder of hdfs related stuff backport from
> https://issues.apache.org/jira/browse/HBASE-7509
> The quorum read ability should be helpful especially to optimize read outliers
> we can utilize "dfs.dfsclient.quorum.read.threshold.millis" &
> "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read
> ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we
> could export the interested metric valus into client system(e.g. HBase's
> regionserver metric).
> The core logic is in pread code path, we decide to goto the original
> fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per
> the above config items.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
