[
https://issues.apache.org/jira/browse/HDFS-4817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13702417#comment-13702417
]
Colin Patrick McCabe commented on HDFS-4817:
--------------------------------------------
1,2,4: fixed
3: I think {{CanSetDropBehind}} is the best we're going to get. I wish we had
a base class for all Hadoop output streams like we do for input streams. But I
think that ship has sailed already for compatibility reasons.
5: I am going to have it return false when the method is not supported. I
think most code will not check this return value, except unit tests and perhaps
high performance-conscious code. I'll also add "throws IOException" in case
any future implementations need to do that.
6, 8: ok
9: yes, I think we should have separate configs for read versus write, same as
on the datanode side. added.
10: I think it's an established convention in Hadoop to make "readahead" one
word at this point. We have {{ReadaheadPool}}, {{ReadaheadRequest}},
{{ReadaheadRequestImpl}}, {{dfs.datanode.readahead.bytes}}, etc.
> make HDFS advisory caching configurable on a per-file basis
> -----------------------------------------------------------
>
> Key: HDFS-4817
> URL: https://issues.apache.org/jira/browse/HDFS-4817
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
> Affects Versions: 3.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HDFS-4817.001.patch, HDFS-4817.002.patch,
> HDFS-4817.004.patch
>
>
> HADOOP-7753 and related JIRAs introduced some performance optimizations for
> the DataNode. One of them was readahead. When readahead is enabled, the
> DataNode starts reading the next bytes it thinks it will need in the block
> file, before the client requests them. This helps hide the latency of
> rotational media and send larger reads down to the device. Another
> optimization was "drop-behind." Using this optimization, we could remove
> files from the Linux page cache after they were no longer needed.
> Using {{dfs.datanode.drop.cache.behind.writes}} and
> {{dfs.datanode.drop.cache.behind.reads}} can improve performance
> substantially on many MapReduce jobs. In our internal benchmarks, we have
> seen speedups of 40% on certain workloads. The reason is because if we know
> the block data will not be read again any time soon, keeping it out of memory
> allows more memory to be used by the other processes on the system. See
> HADOOP-7714 for more benchmarks.
> We would like to turn on these configurations on a per-file or per-client
> basis, rather than on the DataNode as a whole. This will allow more users to
> actually make use of them. It would also be good to add unit tests for the
> drop-cache code path, to ensure that it is functioning as we expect.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
