[ https://issues.apache.org/jira/browse/HDFS-4949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707288#comment-13707288 ]
Sanjay Radia commented on HDFS-4949: ------------------------------------ I think we can treat the RAM copies as replicas - this fits into the generalized tiered-storage architecture as described in HDFS-2832 (Ram, flash, disk.) * Block reports will indicate the storage type. * NN will store the storage type in the Block map * Block locations returned by NN to client will have storage type (i.e. don't need the IsCached flag). ** NN will order the replicas locations based on closeness and speed; this will mean that the client side will automatically go to the best place (although we can have a smarter client do something different if desired.) * NN will not count Ram replicas towards the normal replica count - this is one area where the ram replicas are treated differently. * This can support a usage model where the ram replicas are at each or only some of the disk replica locations. > Centralized cache management in HDFS > ------------------------------------ > > Key: HDFS-4949 > URL: https://issues.apache.org/jira/browse/HDFS-4949 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode > Affects Versions: 3.0.0, 2.2.0 > Reporter: Andrew Wang > Assignee: Andrew Wang > Attachments: caching-design-doc-2013-07-02.pdf > > > HDFS currently has no support for managing or exposing in-memory caches at > datanodes. This makes it harder for higher level application frameworks like > Hive, Pig, and Impala to effectively use cluster memory, because they cannot > explicitly cache important datasets or place their tasks for memory locality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira