[jira] [Comment Edited] (HDFS-5851) Support memory as a storage medium
[ https://issues.apache.org/jira/browse/HDFS-5851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983835#comment-13983835 ] Arpit Agarwal edited comment on HDFS-5851 at 4/29/14 7:59 PM: -- I scheduled a Google+ hangout to discuss this topic for 4/30 3-5pm PDT - [link here|https://plus.google.com/events/ckvo7ui46qihd6cfq0sqptrhogo?authkey=CMvgrcTOv9n12wE]. Let me know if you are unable to access it. I will be attending remotely as I am not located in the bay area. I have reserved a conference room at the Hortonworks Palo Alto office for anyone who wants to attend in person. Please check with [~sanjay.radia] for access to the Hortonworks office ahead of time if you plan to attend. The address is in Sanjay's comment below. _(edit: Increased the scheduled time from 1 hour - 2 hour in case we go over)_ was (Author: arpitagarwal): I scheduled a Google+ hangout for 4/30 3-4pm PDT - [link here|https://plus.google.com/events/ckvo7ui46qihd6cfq0sqptrhogo?authkey=CMvgrcTOv9n12wE]. Let me know if you are unable to access it. Support memory as a storage medium -- Key: HDFS-5851 URL: https://issues.apache.org/jira/browse/HDFS-5851 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Affects Versions: 3.0.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal Attachments: SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf, SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf Memory can be used as a storage medium for smaller/transient files for fast write throughput. More information/design will be added later. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (HDFS-5851) Support memory as a storage medium
[ https://issues.apache.org/jira/browse/HDFS-5851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13981608#comment-13981608 ] Sanjay Radia edited comment on HDFS-5851 at 4/28/14 10:23 PM: -- Added comparison to Tachyon in the doc. The is also an implementation difference that I don't cover (Tachyon I believe uses RamFs rather than a memory that is mapped to a HDFS file -- but need to verify that). I have reproduced the text from the updated doc here for convenience: Recently, Spark has added an RDD implementation called Tachyon [4]. Tachyon is outside the address space of an application and allows sharing RDDs across applications. Both Tachyon and DDMs use memory mapped files and lazy writing to reduce the need to recompute. Tachyon, since it is an RDD implementation, records the computation in order to regenerate the data in case of loss whereas DDMs relies on the application to regenerate. Tachyon and RDDs do not have a notion of discardability, which is fundamental to DDMs where data can be discarded when it is under memory and/or backing store pressure. DDMs are closest to virtual memory/anti-caching in that they virtualize memory, with the twist that data can be discarded. was (Author: sanjay.radia): Added comparison to Tachyon in the doc. The is also an implementation difference that I don't cover (Tachyon I believe uses RamFs rather than a memory that is mapped to a HDFS file -- but need to verify that). I have reproduced the text from the updated doc here for convenience: Recently, Spark has added an RDD implementation called Tachyon [4]. Tachyon is outside the address space of an application and allows sharing RDDs across applications. Both Tachyon and DDMs use memory mapped files and lazy writing to reduce the need to recompute. Tachyon, since it is an RDD implementation, records the computation in order to regenerate the data in case of loss whereas DDMs relies on the application to regenerate. Tachyon and RDDs do not have a notion of discardability, which is fundamental to DDMs where data can be discarded when it is under memory and/or backing store pressure. Support memory as a storage medium -- Key: HDFS-5851 URL: https://issues.apache.org/jira/browse/HDFS-5851 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Affects Versions: 3.0.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal Attachments: SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf, SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf Memory can be used as a storage medium for smaller/transient files for fast write throughput. More information/design will be added later. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (HDFS-5851) Support memory as a storage medium
[ https://issues.apache.org/jira/browse/HDFS-5851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13981608#comment-13981608 ] Sanjay Radia edited comment on HDFS-5851 at 4/25/14 9:10 PM: - Added comparison to Tachyon in the doc. The is also an implementation difference that I don't cover (Tachyon I believe uses RamFs rather than a memory that is mapped to a HDFS file -- but need to verify that). I have reproduced the text from the updated doc here for convenience: Recently, Spark has added an RDD implementation called Tachyon [4]. Tachyon is outside the address space of an application and allows sharing RDDs across applications. Both Tachyon and DDMs use memory mapped files and lazy writing to reduce the need to recompute. Tachyon, since it is an RDD implementation, records the computation in order to regenerate the data in case of loss whereas DDMs relies on the application to regenerate. Tachyon and RDDs do not have a notion of discardability, which is fundamental to DDMs where data can be discarded when it is under memory and/or backing store pressure. was (Author: sanjay.radia): Added comparison to Tachyon in the doc. The is also an implementation difference that I don't cover (Tachyon I believe uses RamFs rather than a memory that is mapped to a HDFS file -- but need to verify that). Support memory as a storage medium -- Key: HDFS-5851 URL: https://issues.apache.org/jira/browse/HDFS-5851 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Affects Versions: 3.0.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal Attachments: SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf, SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf Memory can be used as a storage medium for smaller/transient files for fast write throughput. More information/design will be added later. -- This message was sent by Atlassian JIRA (v6.2#6252)
