[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940016#comment-15940016 ]
ramkrishna.s.vasudevan edited comment on HBASE-16438 at 3/24/17 8:54 AM: ------------------------------------------------------------------------- I have a way to solve this problem. LEt's discuss before I put up the patch. Most of the other RB comments are fixed. -> Now since we need to know if the chunk is from pool or not - the Chunk will have a boolean indicating whther the chunk was created for the pool. Say we have isFromPool() will return true for those chunks. -> Every chunk will have an AtomicInteger ref count. -> When the MSLAB does a copyToChunkCell - where we know that the cell has to have a chunk(comes out of chunkCreator) we do an increment of the refCount. -> Now in the MemstoreImpl when we do getCellSet().add() ( we need to have a new API in CellSet which actually returns the cell that was already there in the CSLM which is returned by CSLM.put() returns. Now we only have cellSet#add() which return boolean). -> On this returned cell (which is the actual duplicate cell) we get the chunkId from the Cell. remember we now have a BbChunkCell which can give the chunkid from the 0th offset. -> Use this chunkId to actually do a decrement of the reference count of this chunk. For this we need a decrementChunkRefCount in MSLAB interface. I think it is valid because MSLAB impl is nothing but Chunks. -> Now on doing this decrementChunkRefCount , we could check if the result is now 0 and if so just remove that chunk from the chunkCreator map. So by this way we are making sure that the reference to the chunk is released immediately. -> Things to note is that in case the chunk is from Pool this increment/decrement will not have any impact. This will impact only when we have ondemand chunks. -> There is an atomic ref count operation happening now which may add on to the write path overhead. May be need to see the impact. but remember this is going to happen only if there are lot of duplicates like in HBASE-16195. In a normal case this should not be a problem because the CSLM#put() is going to return a null as there is no duplicate and so there are no such problems. And infact in such a case the GC issue mentioned in HBASE-16195 will not happen as all the chunks are needed till the MSLAB is closed. Thoughts!!! was (Author: ram_krish): I have a way to solve this problem. LEt's discuss before I put up the patch. Most of the other RB comments are fixed. -> Now since we need to if the chunk is from pool or not - the Chunk will have a boolean indicating whther the chunk was created for the pool. Say we have isFromPool() will return true for those chunks. -> Every chunk will have an AtomicInteger ref count. -> When the MSLAB does a copyToChunkCell - where we know that the cell has to have a chunk(comes out of chunkCreator) we do an increment of the refCount. -> Now in the MemstoreImpl when we do getCellSet().add() ( we need to have a new API in CellSet which actually returns the cell that was already there in the CSLM which is returned by CSLM.put() returns. Now we only have cellSet#add() which return boolean). -> On this returned cell (which is the actual duplicate cell) we get the chunkId from the Cell. remember we now have a BbChunkCell which can give the chunkid frm the 0th offset. -> Use this chunkId to actually do a decrement of the reference count of this chunk. For this we need a decrementChunkRefCount in MSLAB interface. I think it is valid because MSLAB impl is nothing but Chunks. -> Now on doing this decrementChunkRefCount , we could check if the result is now 0 and if so just remove that chunk from the chunkCreator map. So by this way we are making sure that the reference to the chunk is released immediately. -> Things to note is that in case the chunk is from Pool this increment/decrement will not have any impact. This will impact only when we have ondemand chunks. -> There is an atomic ref count operation happening now which may add on to the write path overhead. May be need to see the impact. but remember this is going to happen only if there are lot of duplicates like in HBASE-16195. In a normal case this should not be a problem because the CSLM#put() is going to return a null as there is no duplicate and so there are no such problems. And infact in such a case the GC issue mentioned in HBASE-16195 will not happen as all the chunks are needed till the MSLAB is closed. Thoughts!!! > Create a cell type so that chunk id is embedded in it > ----------------------------------------------------- > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task > Affects Versions: 2.0.0 > Reporter: ramkrishna.s.vasudevan > Assignee: ramkrishna.s.vasudevan > Attachments: HBASE-16438_1.patch, > HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, > MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, > MemstoreChunkCell_trunk.patch > > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.15#6346)