[jira] [Comment Edited] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17024858#comment-17024858 ] maobaolong edited comment on HDFS-15133 at 1/28/20 4:30 AM: [~brahmareddy] [~ayushtkn], After some work these day, i give some update now. - I copy the package `org.apache.hadoop.hdds.utils.db` from hadoop-ozone project to the package `org.apache.hadoop.hdds.utils.db` of the module hadoop-common of the project hadoop. In fact i think i have done the sub task HDFS-15137 - I'm doing a refactor for INodeMap, i try to abstract the `GSet map` into a concept named `InodeStore`. It have two implementation, one named HeapInodeStore, as the name, it is the origin mode, the other named RocksInodeStore, it work as rocksdb mode. In fact, i'm doing the part of https://issues.apache.org/jira/browse/HDFS-15138. But, i met some problem, - It is hard to measure the size of a rocksdb? - how to implements the method getMapIterator graceful for the RocksInodeStore. - Clear method will be implements to reset the rocksdb for the RocksInodeStore. Future Job - I will commit my work to my branch named rocks-metastore, my repo is https://github.com/maobaolong/hadoop. As now, it contains compile error now, so i will commit several days later. - Feel free to discuss with my here or in the slack hdfs channel - Feel free to take these jira task or help me to correct my mistake - Doing some minor test. This picture is the relationship of the new class mainly. !image-2020-01-28-12-30-33-015.png! was (Author: maobaolong): [~brahmareddy] [~ayushtkn], After some work these day, i give some update now. - I copy the package `org.apache.hadoop.hdds.utils.db` from hadoop-ozone project to the package `org.apache.hadoop.hdds.utils.db` of the module hadoop-common of the project hadoop. In fact i think i have done the sub task HDFS-15137 - I'm doing a refactor for INodeMap, i try to abstract the `GSet map` into a concept named `InodeStore`. It have two implementation, one named HeapInodeStore, as the name, it is the origin mode, the other named RocksInodeStore, it work as rocksdb mode. In fact, i'm doing the part of https://issues.apache.org/jira/browse/HDFS-15138. But, i met some problem, - It is hard to measure the size of a rocksdb? - how to implements the method getMapIterator graceful for the RocksInodeStore. - Clear method will be implements to reset the rocksdb for the RocksInodeStore. Future Job - I will commit my work to my branch named rocks-metastore, my repo is https://github.com/maobaolong/hadoop. As now, it contains compile error now, so i will commit several days later. - Feel free to discuss with my here or in the slack hdfs channel - Feel free to take these jira task or help me to correct my mistake - Doing some minor test. > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > Attachments: image-2020-01-28-12-30-33-015.png > > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[
https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17024869#comment-17024869
]
hemanthboyina edited comment on HDFS-15133 at 1/28/20 5:40 AM:
---
thanks for the improvements [~maobaolong]
{quote}how to implements the method getMapIterator graceful for the
RocksInodeStore.
{quote}
you can use
{code:java}
RocksDB db = RocksDB.open(new Options(), "/path");
db.newIterator();{code}
{quote} Clear method will be implements to reset the rocksdb for the
RocksInodeStore.
{quote}
you should delete all the entries in rocksdb , so we should be deleting using
DeleteFilesInRange()
some questions
1) how you have linked HeapInodeStore with RocksInodeStore ?
2) any changes in existing INodeMap tree structure ?
3) are you pushing complete INodeMap to RocksDB ?
was (Author: hemanthboyina):
thanks for the improvements [~maobaolong]
{quote}how to implements the method getMapIterator graceful for the
RocksInodeStore.
{quote}
you can use
{code:java}
RocksDB db = RocksDB.open(new Options(), "/path");
db.newIterator();{code}
one question , how you have linked HeapInodeStore with RocksInodeStore ?
> Use rocksdb to store NameNode inode and blockInfo
> -
>
> Key: HDFS-15133
> URL: https://issues.apache.org/jira/browse/HDFS-15133
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.3.0
>Reporter: maobaolong
>Priority: Major
> Attachments: image-2020-01-28-12-30-33-015.png
>
>
> Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can
> achieve the same request.
> This is ozone and alluxio way to manage meta data of master node.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17025012#comment-17025012 ] maobaolong edited comment on HDFS-15133 at 1/29/20 1:44 AM: [~ayushtkn] As far as i know, i think all of the inode stored in the INodeMap.map, isn't it? I am not very sure. was (Author: maobaolong): As far as i know, i think all of the inode stored in the INodeMap.map, isn't it? I am not very sure. > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > Attachments: image-2020-01-28-12-30-33-015.png > > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17025530#comment-17025530 ] maobaolong edited comment on HDFS-15133 at 1/29/20 2:01 AM: https://github.com/maobaolong/hadoop/tree/rocks-metastore This is a POC for my idea, it cannot works, just a prototype, but you can see my works now was (Author: maobaolong): https://github.com/maobaolong/hadoop/tree/rocks-metastore > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > Attachments: image-2020-01-28-12-30-33-015.png > > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
