[
https://issues.apache.org/jira/browse/HDFS-4817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13662555#comment-13662555
]
Colin Patrick McCabe commented on HDFS-4817:
--------------------------------------------
[~andrew.wang] thanks for the terasort benchmarks, those are very encouraging!
It's great to know that we can get this kind of performance with this patch
just by changing a few MR client settings.
bq. The FileSystem cache is annoying here, since it means all threads share the
same DFSClient and thus the same default CachingStrategy
Agree. I'm going to get rid of the get/set methods on the {{FileSystem}}, for
the reasons you and Todd brought up. It just doesn't make sense, given how we
cache those FileSystem objects.
bq. Cache management behavior in BlockReceiver is a bit different. Whereas
before we would sync or dropbehind in window increments, now we do it up to the
current position. Any comments on why this was changed? The difference seems
minor (4MB vs. 4MB+64KB packet).
That's not quite true. The general idea is, we try to {{sync_file_pages}} the
last few bytes we read, and we call {{FADV_DONTNEED}} from the beginning of the
file up to a position slightly behind the current position. The way this is
done changed slightly, but not the concept.
The general idea is that there isn't much point in calling {{sync_file_pages}}
twice on the same offsets, since the sync process has presumably already begun.
On the other hand, calling {{fadvise(FADV_DONTNEED)}} again and again will
tend to purge more and more bytes from the cache. The reason is because dirty
pages (those containing un-written-out-data) cannot be purged using
{{FADV_DONTNEED}}. And we can't know exactly when the pages we wrote will be
flushed to disk. But we do know that calling {{FADV_DONTNEED}} on very
recently written bytes is a waste, since they will almost certainly not have
been written out to disk. That is why it purges between 0 and
{{lastCacheManagementOffset - CACHE_WINDOW_SIZE}}, rather than simply 0 to pos.
Fixed typo in BlockReceiver comment.
bq. NativeIO.java: @VisibleForTesting on LONG_READ_THRESHOLD_BYTES too. Any
reason why these aren't configuration parameters? Then they could be final, and
still modified for tests.
I think maybe what we should do is keep {{LONG_READ_THRESHOLD_BYTES}} as-is,
but only use it if no explicit cache management policy was set. This allows
users to have control if they want on the file-level rather than the
Datanode-level, which is-- after all-- the whole point here.
bq. [API comments]
I changed the API to:
* be per-stream.
* use boxed Booleans and Longs where {{null}} represents using the (server)
defaults. I think this makes things a lot clearer than using magic numbers to
represent things.
CopyBlock, fsck: yep. We should set dropBehind in those places.
bq. Having classes named CachingStrategy and CacheStrategy is a little
confusing. Rename CacheStrategy to CachingStrategyProto per normal convention?
ok
bq. BlockPoolSliceScanner#verifyBlock, can use CachingStrategy#newDropBehind
convenience method rather than writing it out.
ok
[~saint....@gmail.com] You can now change the caching strategy on an open
file, something I believe you requested earlier. I also reduced the long read
threshold from 256kb to 16kb, so that HBase can take better advantage of these
settings. We will have to benchmark more to get the perfect balance, but this
should be at least a starting point.
> make HDFS advisory caching configurable on a per-file basis
> -----------------------------------------------------------
>
> Key: HDFS-4817
> URL: https://issues.apache.org/jira/browse/HDFS-4817
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
> Affects Versions: 3.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HDFS-4817.001.patch
>
>
> HADOOP-7753 and related JIRAs introduced some performance optimizations for
> the DataNode. One of them was readahead. When readahead is enabled, the
> DataNode starts reading the next bytes it thinks it will need in the block
> file, before the client requests them. This helps hide the latency of
> rotational media and send larger reads down to the device. Another
> optimization was "drop-behind." Using this optimization, we could remove
> files from the Linux page cache after they were no longer needed.
> Using {{dfs.datanode.drop.cache.behind.writes}} and
> {{dfs.datanode.drop.cache.behind.reads}} can improve performance
> substantially on many MapReduce jobs. In our internal benchmarks, we have
> seen speedups of 40% on certain workloads. The reason is because if we know
> the block data will not be read again any time soon, keeping it out of memory
> allows more memory to be used by the other processes on the system. See
> HADOOP-7714 for more benchmarks.
> We would like to turn on these configurations on a per-file or per-client
> basis, rather than on the DataNode as a whole. This will allow more users to
> actually make use of them. It would also be good to add unit tests for the
> drop-cache code path, to ensure that it is functioning as we expect.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
