[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[
https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17033184#comment-17033184
]
Ayush Saxena commented on HDFS-15133:
-
I had a quick try on this.
[~maobaolong] firstly you need to replace {{rootDir}} from {{FsDirectory}},
Since it is storing reference to all the child inodes and until there is anyone
holding reference to the inode, the inode will be in heap only. You can replace
it with storing root's Inode id, and create a method as {{getRootDir()}} which
fetches the root from the rootID from the DB.
Secondly, when you will be serializing the inode to store in DB it will loose
its traversal behavior, once you add root, then its child and so on, you can't
traverse to the last child by just fetching the root Inode. So, you need to
change the way present way of fetching the inodes. That shall be trickiest part
here IMO.
Thirdly once a inode is added to the DB, if you get the inode from the DB and
do any modifications, as adding blocks, changing replica states and other
stuffs, you need to again put it back in the DB and replace with the existing
one. For this you can create a method in {{FSDirectory}} to update the iNodes.
I tried that way only,
Whenever BlockReports or IBR are coming, and changing the block state, that
also you need to update in the inode, Fetch the inode id from the Block Info,
then get the corresponding Inode and update the new blockInfo there.
Off all I think writing file would be still doable stuff, For reading you need
to change the present read mechanism, due to lack of chaining through root, may
be something using the {{InodeId}}
> Use rocksdb to store NameNode inode and blockInfo
> -
>
> Key: HDFS-15133
> URL: https://issues.apache.org/jira/browse/HDFS-15133
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.3.0
>Reporter: maobaolong
>Priority: Major
> Attachments: image-2020-01-28-12-30-33-015.png
>
>
> Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can
> achieve the same request.
> This is ozone and alluxio way to manage meta data of master node.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[
https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17033022#comment-17033022
]
hemanthboyina commented on HDFS-15133:
--
{quote}as in the existing HDFS code we have a tree structure of INode's in Heap
which will be put in INodeMap and used as reference
{quote}
Each inode contains reference to parent and its children. So putting the INode
into DB doesn't solve our issue , as inode contains its children inodes as a
referrence ,if you add a child its parent needs to be updated . this doesn't
happen if we put inode in rocksdb
> Use rocksdb to store NameNode inode and blockInfo
> -
>
> Key: HDFS-15133
> URL: https://issues.apache.org/jira/browse/HDFS-15133
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.3.0
>Reporter: maobaolong
>Priority: Major
> Attachments: image-2020-01-28-12-30-33-015.png
>
>
> Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can
> achieve the same request.
> This is ozone and alluxio way to manage meta data of master node.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[
https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17033008#comment-17033008
]
Ayush Saxena commented on HDFS-15133:
-
bq. IMO Storing the INode into RocksDB doesn't satisfy our problem scenario ,
as in the existing HDFS code we have a tree structure of INode's in Heap which
will be put in INodeMap and used as reference
Are you talking about {{INodeDirectory rootDir;}} in FSDirectory? If so,
Presently it stores root as {{InodeDirectory}} we can change to just the inode
Id of the root and whenever the root is required we can fetch it using the
root's inode Id from the inode map. The rootDir is also put into InodeMap.
[~maobaolong] regarding the whole of Inodes in the DB, I have a doubt on this
if you put all of the nodes into DB, if you fetch a entry from DB and then do
modifications on it. I think we need to replace the updated Inode in the DB,
which was not done when the Inode object stayed in heap.
Each inode contains reference to parent and its children, When in heap, its all
references, if you add one child it can be chained from the top most Inode, now
to add a INode, do we need to update all the parent entries?
> Use rocksdb to store NameNode inode and blockInfo
> -
>
> Key: HDFS-15133
> URL: https://issues.apache.org/jira/browse/HDFS-15133
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.3.0
>Reporter: maobaolong
>Priority: Major
> Attachments: image-2020-01-28-12-30-33-015.png
>
>
> Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can
> achieve the same request.
> This is ozone and alluxio way to manage meta data of master node.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[
https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17032990#comment-17032990
]
hemanthboyina commented on HDFS-15133:
--
thanks for the work [~maobaolong]
{quote}We propose to implement the HDFS namespace on top of a key-value (KV)
store
{quote}
IMO Storing the INode into RocksDB doesn't satisfy our problem scenario , as in
the existing HDFS code we have a tree structure of INode's in Heap which will
be put in INodeMap and used as reference
To solve our problem scenario we are proposing to have a light weight INode
which will be having variables ID and name only , so the size of light weight
INode object will be very less comparing to existing INode
After this modification , light weight INode object will be in Heap and
Complete INode object in RocksDB
I have created HDFS-15152 to implement this
> Use rocksdb to store NameNode inode and blockInfo
> -
>
> Key: HDFS-15133
> URL: https://issues.apache.org/jira/browse/HDFS-15133
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.3.0
>Reporter: maobaolong
>Priority: Major
> Attachments: image-2020-01-28-12-30-33-015.png
>
>
> Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can
> achieve the same request.
> This is ozone and alluxio way to manage meta data of master node.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17032130#comment-17032130 ] maobaolong commented on HDFS-15133: --- There are some work todo. - how to measure the size of the inodeTable? for the heapStore, the size is easy to get, but how to get the size of the rocksdb table? - I want to use protobuf to serialisation and deserialisation, so there are a big work to write serval message of proto, and a PBHelper is needed, serval Codec of the sub class of INode should be implemented. If everything goes well, we will see the result soon. > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > Attachments: image-2020-01-28-12-30-33-015.png > > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17032122#comment-17032122 ] maobaolong commented on HDFS-15133: --- [~hemanthboyina] Thank you very much for your comments. 1) I don't begin to do this work, we can use protobuf to do serialisation and deserialisation 2) Indeed, we need to add our inodeTable. 3) Yeah, so careless i am. Thank you twice for your carefully review, i have push a PR, look forward to your review again. https://github.com/maobaolong/hadoop/pull/1 > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > Attachments: image-2020-01-28-12-30-33-015.png > > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[
https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17031475#comment-17031475
]
hemanthboyina commented on HDFS-15133:
--
thanks for the work [~maobaolong]
i have gone through your commit .
some questions and suggestions
1)
{quote}// TODO(maobaolong): addCodec for INode and INodeWithAdditionalFields
and all of their sub class.
{quote}
How are you planning to addCodec for INode Objects ?
I think we need to use serialisation and deserialisation of INodeDirectory and
INodeFile
2)
{code:java}
public RocksInodeStore(INodeDirectory rootDir) throws IOException {
DBStore dbStore = DBStoreBuilder.newBuilder(new
OzoneConfiguration()).setName("inode.db")
.setPath(Paths.get("./"))
.build(); {code}
we have to add our INodeTable as
{code:java}
DBStore dbStore = DBStoreBuilder.newBuilder(new
OzoneConfiguration()).setName("inode.db")
.setPath(Paths.get("./")).addTable("inodeTable")
.build();{code}
3) While Intialising RocksINodeStore , First we have to put Root INodeDirectory
into the INodeTable .
> Use rocksdb to store NameNode inode and blockInfo
> -
>
> Key: HDFS-15133
> URL: https://issues.apache.org/jira/browse/HDFS-15133
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.3.0
>Reporter: maobaolong
>Priority: Major
> Attachments: image-2020-01-28-12-30-33-015.png
>
>
> Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can
> achieve the same request.
> This is ozone and alluxio way to manage meta data of master node.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17028299#comment-17028299 ] maobaolong commented on HDFS-15133: --- [~brahmareddy] Ok, i've make the inodeStore to be configurable, and default value is HeapInodeStore. New commit has been put to my repository https://github.com/maobaolong/hadoop/tree/rocks-metastore > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > Attachments: image-2020-01-28-12-30-33-015.png > > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17025536#comment-17025536 ] Brahma Reddy Battula commented on HDFS-15133: - As discussed earlier, make this configurable to aviod the confusion. Once POC test is done, Please publish the result. > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > Attachments: image-2020-01-28-12-30-33-015.png > > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17025530#comment-17025530 ] maobaolong commented on HDFS-15133: --- https://github.com/maobaolong/hadoop/tree/rocks-metastore > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > Attachments: image-2020-01-28-12-30-33-015.png > > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17025531#comment-17025531 ] maobaolong commented on HDFS-15133: --- https://github.com/maobaolong/hadoop/tree/rocks-metastore > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > Attachments: image-2020-01-28-12-30-33-015.png > > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17025528#comment-17025528 ] maobaolong commented on HDFS-15133: --- [~hemanthboyina] Thank you for your question. Indeed, if the RocksInodeStore works perfect, it is no need to have a HeapInodeStore, but everything has two sides, although rocksInodeStore mode can enlarge the maximum of inode, and we will try to use cache to improve the performance, but i think in some scenarios, HeapInodeStore can work well, and i think some company use the origin heap mode well, they don't have to try the rocksInodeStore, so we must provide a way to let them run the namenode in the origin way. HeapInodeStore is the origin way to manage inode. > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > Attachments: image-2020-01-28-12-30-33-015.png > > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17025039#comment-17025039 ] hemanthboyina commented on HDFS-15133: -- thanks for the reply [~maobaolong] I think you are using HeapInodeStore and RocksInodeStore to store the INode object . What is the benefit of this ? As you are still storing the entire INode object in HeapInodeStore.Map . IMO we are not getting any benefit in terms of heap used . please correct me if i am missing something > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > Attachments: image-2020-01-28-12-30-33-015.png > > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17025012#comment-17025012 ] maobaolong commented on HDFS-15133: --- As far as i know, i think all of the inode stored in the INodeMap.map, isn't it? I am not very sure. > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > Attachments: image-2020-01-28-12-30-33-015.png > > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17025011#comment-17025011 ] maobaolong commented on HDFS-15133: --- [~hemanthboyina] Thank you for your help. - I found the TypedTable.iterator of HDDS can call RocksDB. newIterator() Indirectly. - Yeah, i agree with "delete all the entries in rocksdb , so we should be deleting using DeleteFilesInRange()" Answer your question 1) HeapInodeStore and RocksInodeStore are the two independent implementation of InodeStore, so there are no link between HeapInodeStore with RocksInodeStore. 2) No changes INodeMap tree structure, i honor the origin structure. 3) I think InodeMap is a manage class, the true structure for the store inode is the member InodeMap.map, I think all inode in this container, so i try to put all of the inode into the rocksdb by replace the InodeMap.map into a inodestore. Finally I am not sure i explain exactly, so please see my commit diff of rocks-metastore branch, my repo is https://github.com/maobaolong/hadoop. > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > Attachments: image-2020-01-28-12-30-33-015.png > > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17024876#comment-17024876 ] Ayush Saxena commented on HDFS-15133: - [~maobaolong] are you just trying to push just the inodeMap to the DB? Pushing just the inodeMap wouldn't be of much use. Ideally we need to push all the inodes with all its fields into the db, so as to ensure storing max objects. > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > Attachments: image-2020-01-28-12-30-33-015.png > > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[
https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17024869#comment-17024869
]
hemanthboyina commented on HDFS-15133:
--
thanks for the improvements [~maobaolong]
{quote}how to implements the method getMapIterator graceful for the
RocksInodeStore.
{quote}
you can use
{code:java}
RocksDB db = RocksDB.open(new Options(), "/path");
db.newIterator();{code}
one question , how you have linked HeapInodeStore with RocksInodeStore ?
> Use rocksdb to store NameNode inode and blockInfo
> -
>
> Key: HDFS-15133
> URL: https://issues.apache.org/jira/browse/HDFS-15133
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.3.0
>Reporter: maobaolong
>Priority: Major
> Attachments: image-2020-01-28-12-30-33-015.png
>
>
> Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can
> achieve the same request.
> This is ozone and alluxio way to manage meta data of master node.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17024858#comment-17024858 ] maobaolong commented on HDFS-15133: --- [~brahmareddy] [~ayushtkn], After some work these day, i give some update now. - I copy the package `org.apache.hadoop.hdds.utils.db` from hadoop-ozone project to the package `org.apache.hadoop.hdds.utils.db` of the module hadoop-common of the project hadoop. In fact i think i have done the sub task HDFS-15137 - I'm doing a refactor for INodeMap, i try to abstract the `GSet map` into a concept named `InodeStore`. It have two implementation, one named HeapInodeStore, as the name, it is the origin mode, the other named RocksInodeStore, it work as rocksdb mode. In fact, i'm doing the part of https://issues.apache.org/jira/browse/HDFS-15138. But, i met some problem, - It is hard to measure the size of a rocksdb? - how to implements the method getMapIterator graceful for the RocksInodeStore. - Clear method will be implements to reset the rocksdb for the RocksInodeStore. Future Job - I will commit my work to my branch named rocks-metastore, my repo is https://github.com/maobaolong/hadoop. As now, it contains compile error now, so i will commit several days later. - Feel free to discuss with my here or in the slack hdfs channel - Feel free to take these jira task or help me to correct my mistake - Doing some minor test. > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17021272#comment-17021272 ] Ayush Saxena commented on HDFS-15133: - Thanx [~maobaolong] for the initiative, seems interesting, do you mind sharing either some design doc or may be just action plan for this. > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17021264#comment-17021264 ] Brahma Reddy Battula commented on HDFS-15133: - [~maobaolong] thanks for proposing this, would like to hear POC reults if you've done on this. thanks. > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17020765#comment-17020765 ] Surendra Singh Lilhore commented on HDFS-15133: --- bq. The RDBStore and TypedTable can be responsible for the kv store manager, so we can starts all work by the moving the code of RDBStore related to hadoop-common, so that ozone and hdfs or yarn and other component can use this wonderful feature without any more effort. [~maobaolong], good idea (y). > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17020727#comment-17020727 ] maobaolong commented on HDFS-15133: --- [~iwasakims] [~hemanthboyina] After review the HDFS-5389 and HDFS-8286, i saw the similar design, but they are all aimed to invent a new approach to achieve the goal. This jira mainly hope to reuse the approach of how to manage racksdb which implemented in HDDS. The RDBStore and TypedTable can be responsible for the kv store manager, so we can starts all work by the moving the code of RDBStore related to hadoop-common, so that ozone and hdfs or yarn and other component can use this wonderful feature without any more effort. > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17019880#comment-17019880 ] hemanthboyina commented on HDFS-15133: -- I think +HDFS-8286+ is also relates to this Jira . > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17019862#comment-17019862 ] Masatake Iwasaki commented on HDFS-15133: - Is this reviving HDFS-5389? > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
