[
https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880280#comment-13880280
]
Colin Patrick McCabe commented on HDFS-5776:
--------------------------------------------
bq. One note on 1.. Is static ever a good idea for sharing resources? But your
point of being able to share amongst DFSClient instances is for sure something
we should pursue (in another JIRA?)
Unfortunately, the {{FileContext}} API creates a new {{DFSClient}} instance for
each operation that it does. (The older {{FileSystem}} API doesn't have this
problem, since the {{DistributedFileSystem}} object hangs on to the
{{DFSClient}} for a while.) This means that we do need to put this in a
static, for now, or else {{FileContext}} users will be constantly destroying
and creating thread-pools.
I have another change pending which creates the concept of a "cache context,"
where different threads can use different contexts if they like. For now,
let's use a static variable, maybe with a TODO.
bq. Related to the previous - what do you think of not exposing the
DFS_DFSCLIENT_HEDGED_READ_THREADPOOL_SIZE setting at all? Maybe we can just
expose a boolean setting to enable it. The reason I prefer not to surface such
settings is because it invites abuse (the concern is not with trusted apps like
HBase). If we do expose this setting we should at least have an internal upper
bound.
I don't see why we wouldn't expose this setting. It doesn't give the client
the ability to do anything bad it couldn't already do. You can already try to
open a zillion files at once in order to attack the {{NameNode}} /
{{DataNodes}}. Preventing denial-of-service attacks is not currently something
we try to do. And in the future, if we ever do try to prevent
denial-of-service attacks, I don't think having hedged reads makes that any
more or less difficult than it would otherwise be.
> Support 'hedged' reads in DFSClient
> -----------------------------------
>
> Key: HDFS-5776
> URL: https://issues.apache.org/jira/browse/HDFS-5776
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
> Affects Versions: 3.0.0
> Reporter: Liang Xie
> Assignee: Liang Xie
> Attachments: HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt,
> HDFS-5776-v5.txt, HDFS-5776-v6.txt, HDFS-5776.txt
>
>
> This is a placeholder of hdfs related stuff backport from
> https://issues.apache.org/jira/browse/HBASE-7509
> The quorum read ability should be helpful especially to optimize read outliers
> we can utilize "dfs.dfsclient.quorum.read.threshold.millis" &
> "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read
> ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we
> could export the interested metric valus into client system(e.g. HBase's
> regionserver metric).
> The core logic is in pread code path, we decide to goto the original
> fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per
> the above config items.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
