[ 
https://issues.apache.org/jira/browse/HDFS-8401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-8401:
--------------------------------
    Description: 
We propose creating a layered filesystem that can provide in-memory storage 
using existing features within HDFS. memfs will use lazy persist writes 
introduced by HDFS-6581. For reads, memfs can use the Centralized Cache 
Management feature introduced in HDFS-4949 to load hot data to memory.

Paths in memfs and hdfs will correspond 1:1 so memfs will require no additional 
metadata and it can be implemented entirely as a client-side library.

The advantage of a layered file system is that it requires little or no changes 
to existing applications. e.g. Applications can use something like {{memfs://}} 
instead of {{hdfs://}} for files targeted to memory storage. 

  was:
We propose creating a layered filesystem that can provide in-memory storage 
using existing features within HDFS. memfs will use lazy persist writes 
introduced by HDFS-6581. For reads, memfs can use the Centralized Cache 
Management feature introduced in HDFS-4949 to load hot data to memory.

Paths in memfs and hdfs will correspond 1:1 so memfs will require no additional 
metadata and it can be implemented entirely as a client-side library.

The advantage of a layered file system is that it requires little or no changes 
to existing applications. e.g. Applications can use something like {{memfs://}} 
instead of {{hdfs://}} for files targeted to memory storage. The layered file 
system idea was first proposed in HDFS-5851 by [~sanjay.radia].

More details to follow.


> Memfs - a layered file system for in-memory storage in HDFS
> -----------------------------------------------------------
>
>                 Key: HDFS-8401
>                 URL: https://issues.apache.org/jira/browse/HDFS-8401
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
>
> We propose creating a layered filesystem that can provide in-memory storage 
> using existing features within HDFS. memfs will use lazy persist writes 
> introduced by HDFS-6581. For reads, memfs can use the Centralized Cache 
> Management feature introduced in HDFS-4949 to load hot data to memory.
> Paths in memfs and hdfs will correspond 1:1 so memfs will require no 
> additional metadata and it can be implemented entirely as a client-side 
> library.
> The advantage of a layered file system is that it requires little or no 
> changes to existing applications. e.g. Applications can use something like 
> {{memfs://}} instead of {{hdfs://}} for files targeted to memory storage. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to