[
https://issues.apache.org/jira/browse/HDFS-6450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14063892#comment-14063892
]
Colin Patrick McCabe commented on HDFS-6450:
--------------------------------------------
Yeah... if the first read, done with the old block reader, doesn't finish
first, then I think we should forget about that block reader... even if it's a
{{BlockReaderLocal}} and the new one is remote. After all, slow or misbehaving
local disks are one of the main problems we're trying to cover up with hedged
reads. As you pointed out, we need to close the old block reader in this
situation.
I think from an implementation point of view, you might consider unsetting
{{DFSInputStream#blockReader}} at the beginning of the hedged read (set to null
and move the old block reader into the first future). Then have the "winning"
future set it to the block reader that it used.
There will be some performance regression just due to using threads and futures
here, instead of just doing the read from the same thread that needs the data.
Passing data between threads is slower because you might be going between CPU
cores. I don't know if there's really a good way to address this without doing
something like HDFS-6695, which is out of scope for this change. I think in
the short term, applications will have to turn on non-positional hedged reads
explicitly, and accept some small loss in throughput for a major reduction in
long-tail latency.
> Support non-positional hedged reads in HDFS
> -------------------------------------------
>
> Key: HDFS-6450
> URL: https://issues.apache.org/jira/browse/HDFS-6450
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 2.4.0
> Reporter: Colin Patrick McCabe
> Assignee: Liang Xie
> Attachments: HDFS-6450-like-pread.txt
>
>
> HDFS-5776 added support for hedged positional reads. We should also support
> hedged non-position reads (aka regular reads).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
