[
https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13884554#comment-13884554
]
Colin Patrick McCabe commented on HDFS-5776:
--------------------------------------------
[~arpitagarwal] : if I understand your comments correctly, you are concerned
that hedged reads may spawn too many threads. But that's why
{{dfs.client.hedged.read.threadpool.size}} exists. The {{DFSClient}} will not
create more threads than this.
We do not check other configuration settings to see if they are "reasonable."
For example, if someone wants to set {{dfs.balancer.dispatcherThreads}},
{{dfs.balancer.moverThreads}}, or {{dfs.datanode.max.transfer.threads}} to a
zillion, we don't complain. If we tried to set hard limits everywhere, people
with different needs would have to recompile hadoop to meet those needs.
Please remember that, if the client wants to, he/she can sit in a loop and call
{{new Thread(...)}}. It's not like by giving users the ability to control the
number of threads they use, we are opening up some new world of security
vulnerabilities. The ability for the client to create any number of threads
already exists. And it only inconveniences one person: the client themselves.
[~sureshms]: I agree that we should figure out the configuration issues here
rather than changing the configuration in an incompatible way later. Jing
suggested adding "an Allow-Hedged-Reads configuration" boolean. That certainly
seems to solve the problem of having different threads use different settings.
Is there any objection, besides the inelegance of having two configs rather
than one?
> Support 'hedged' reads in DFSClient
> -----------------------------------
>
> Key: HDFS-5776
> URL: https://issues.apache.org/jira/browse/HDFS-5776
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
> Affects Versions: 3.0.0
> Reporter: Liang Xie
> Assignee: Liang Xie
> Attachments: HDFS-5776-v10.txt, HDFS-5776-v2.txt, HDFS-5776-v3.txt,
> HDFS-5776-v4.txt, HDFS-5776-v5.txt, HDFS-5776-v6.txt, HDFS-5776-v7.txt,
> HDFS-5776-v8.txt, HDFS-5776-v9.txt, HDFS-5776.txt
>
>
> This is a placeholder of hdfs related stuff backport from
> https://issues.apache.org/jira/browse/HBASE-7509
> The quorum read ability should be helpful especially to optimize read outliers
> we can utilize "dfs.dfsclient.quorum.read.threshold.millis" &
> "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read
> ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we
> could export the interested metric valus into client system(e.g. HBase's
> regionserver metric).
> The core logic is in pread code path, we decide to goto the original
> fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per
> the above config items.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
