[
https://issues.apache.org/jira/browse/HDFS-5851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981608#comment-13981608
]
Sanjay Radia edited comment on HDFS-5851 at 4/28/14 10:23 PM:
--------------------------------------------------------------
Added comparison to Tachyon in the doc. The is also an implementation
difference that I don't cover (Tachyon I believe uses RamFs rather than a
memory that is mapped to a HDFS file -- but need to verify that).
I have reproduced the text from the updated doc here for convenience:
Recently, Spark has added an RDD implementation called Tachyon [4]. Tachyon is
outside the address space of an application and allows sharing RDDs across
applications. Both Tachyon and DDMs use memory mapped files and lazy writing to
reduce the need to recompute. Tachyon, since it is an RDD implementation,
records the computation in order to regenerate the data in case of loss whereas
DDMs relies on the application to regenerate. Tachyon and RDDs do not have a
notion of discardability, which is fundamental to DDMs where data can be
discarded when it is under memory and/or backing store pressure. DDMs are
closest to virtual memory/anti-caching in that they virtualize memory, with the
twist that data can be discarded.
was (Author: sanjay.radia):
Added comparison to Tachyon in the doc. The is also an implementation
difference that I don't cover (Tachyon I believe uses RamFs rather than a
memory that is mapped to a HDFS file -- but need to verify that).
I have reproduced the text from the updated doc here for convenience:
Recently, Spark has added an RDD implementation called Tachyon [4]. Tachyon is
outside the address space of an application and allows sharing RDDs across
applications. Both Tachyon and DDMs use memory mapped files and lazy writing to
reduce the need to recompute. Tachyon, since it is an RDD implementation,
records the computation in order to regenerate the data in case of loss whereas
DDMs relies on the application to regenerate. Tachyon and RDDs do not have a
notion of discardability, which is fundamental to DDMs where data can be
discarded when it is under memory and/or backing store pressure.
> Support memory as a storage medium
> ----------------------------------
>
> Key: HDFS-5851
> URL: https://issues.apache.org/jira/browse/HDFS-5851
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
> Affects Versions: 3.0.0
> Reporter: Arpit Agarwal
> Assignee: Arpit Agarwal
> Attachments:
> SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf,
> SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf
>
>
> Memory can be used as a storage medium for smaller/transient files for fast
> write throughput.
> More information/design will be added later.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
