[
https://issues.apache.org/jira/browse/HDFS-4949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707288#comment-13707288
]
Sanjay Radia commented on HDFS-4949:
------------------------------------
I think we can treat the RAM copies as replicas - this fits into the
generalized tiered-storage architecture as described in HDFS-2832 (Ram, flash,
disk.)
* Block reports will indicate the storage type.
* NN will store the storage type in the Block map
* Block locations returned by NN to client will have storage type (i.e. don't
need the IsCached flag).
** NN will order the replicas locations based on closeness and speed; this will
mean that the client side will automatically go to the best place (although we
can have a smarter client do something different if desired.)
* NN will not count Ram replicas towards the normal replica count - this is one
area where the ram replicas are treated differently.
* This can support a usage model where the ram replicas are at each or only
some of the disk replica locations.
> Centralized cache management in HDFS
> ------------------------------------
>
> Key: HDFS-4949
> URL: https://issues.apache.org/jira/browse/HDFS-4949
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
> Affects Versions: 3.0.0, 2.2.0
> Reporter: Andrew Wang
> Assignee: Andrew Wang
> Attachments: caching-design-doc-2013-07-02.pdf
>
>
> HDFS currently has no support for managing or exposing in-memory caches at
> datanodes. This makes it harder for higher level application frameworks like
> Hive, Pig, and Impala to effectively use cluster memory, because they cannot
> explicitly cache important datasets or place their tasks for memory locality.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
