[ https://issues.apache.org/jira/browse/HDFS-4949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707536#comment-13707536 ]
Suresh Srinivas commented on HDFS-4949: --------------------------------------- bq. However, tiered storage management schemes put an entire block into a tier. This is different than what we want to (eventually) do with caching, which is cache only part of a block. If we end up with a heavyweight scheme where the entire block has to be loaded into memory before any of it can be accessed, this may actually cause a performance regression, not an improvement. This is one point I have been mulling over as well. I agree, partially cached data does not fit with storage hierarchy. But I was not sure partial cache is being consider in the first phase of implementation in this jira (sorry I am yet to read this carefully, If i missed details). One other thing that we are considering is usage based extra replica to memory tier. Need to make some time to get that all into a doc. > Centralized cache management in HDFS > ------------------------------------ > > Key: HDFS-4949 > URL: https://issues.apache.org/jira/browse/HDFS-4949 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode > Affects Versions: 3.0.0, 2.2.0 > Reporter: Andrew Wang > Assignee: Andrew Wang > Attachments: caching-design-doc-2013-07-02.pdf > > > HDFS currently has no support for managing or exposing in-memory caches at > datanodes. This makes it harder for higher level application frameworks like > Hive, Pig, and Impala to effectively use cluster memory, because they cannot > explicitly cache important datasets or place their tasks for memory locality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira