[jira] [Commented] (HBASE-11646) Handle the MOB in compaction
[ https://issues.apache.org/jira/browse/HBASE-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637837#comment-14637837 ] Hudson commented on HBASE-11646: FAILURE: Integrated in HBase-TRUNK #6672 (See [https://builds.apache.org/job/HBase-TRUNK/6672/]) HBASE-11646 Handle the MOB in compaction (Jingcheng Du) (jmhsieh: rev 7dbd828a90e36c3b94fcadf659ce047a1ddbce0e) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HMobStore.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobUtils.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMobCompaction.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/StripeCompactor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobCompactor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DefaultCompactor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobStoreEngine.java > Handle the MOB in compaction > > > Key: HBASE-11646 > URL: https://issues.apache.org/jira/browse/HBASE-11646 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: hbase-11339 >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Fix For: hbase-11339 > > Attachments: HBASE-11646-V2.diff, HBASE-11646-V3.diff, > HBASE-11646-V4.diff, HBASE-11646-V5.diff, HBASE-11646-V6.diff, > HBASE-11646-V7.diff, HBASE-11646-V9.diff, HBASE-11646.diff, > hbase-11646-v10.patch, hbase-11646-v8.patch > > > In the updated MOB design however, admins can set CF level thresholds that > would force cell values > the threshold to use the MOB write path instead of > the traditional path. There are two cases where mobs need to interact with > this threshold > 1) How do we handle the case when the threshold size is changed? > 2) Today, you can bulkload hfiles that contain MOBs. These cells will work > as normal inside hbase. Unfortunately the cells with MOBs in them will never > benefit form the MOB write path. > The proposal here is to modify compaction in mob enabled cf's such that the > threshold value is honored with compactions. This handles case #1 -- > elements that should be moved out of the normal hfiles get 'compacted' into > refs and mob hfiles, and values that should be pulled into the cf get derefed > and written out wholy in the compaction. For case #2, we can maintain the > same behavior and compaction would move data into the mob writepath/lifecycle. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11646) Handle the MOB in compaction
[ https://issues.apache.org/jira/browse/HBASE-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138179#comment-14138179 ] Jonathan Hsieh commented on HBASE-11646: Did a minor tweak and ran {code} mvn clean test -Dtest=Test*Compact* {code} It passed. Commimtted to hbase-11339 branch. thanks jingcheng and thanks for the reviews ram and anoop. > Handle the MOB in compaction > > > Key: HBASE-11646 > URL: https://issues.apache.org/jira/browse/HBASE-11646 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: hbase-11339 >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Fix For: hbase-11339 > > Attachments: HBASE-11646-V2.diff, HBASE-11646-V3.diff, > HBASE-11646-V4.diff, HBASE-11646-V5.diff, HBASE-11646-V6.diff, > HBASE-11646-V7.diff, HBASE-11646.diff > > > In the updated MOB design however, admins can set CF level thresholds that > would force cell values > the threshold to use the MOB write path instead of > the traditional path. There are two cases where mobs need to interact with > this threshold > 1) How do we handle the case when the threshold size is changed? > 2) Today, you can bulkload hfiles that contain MOBs. These cells will work > as normal inside hbase. Unfortunately the cells with MOBs in them will never > benefit form the MOB write path. > The proposal here is to modify compaction in mob enabled cf's such that the > threshold value is honored with compactions. This handles case #1 -- > elements that should be moved out of the normal hfiles get 'compacted' into > refs and mob hfiles, and values that should be pulled into the cf get derefed > and written out wholy in the compaction. For case #2, we can maintain the > same behavior and compaction would move data into the mob writepath/lifecycle. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11646) Handle the MOB in compaction
[ https://issues.apache.org/jira/browse/HBASE-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138147#comment-14138147 ] Jonathan Hsieh commented on HBASE-11646: I +1'ed in review board. Will commit to hbase-11339 pending clean test run. > Handle the MOB in compaction > > > Key: HBASE-11646 > URL: https://issues.apache.org/jira/browse/HBASE-11646 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: hbase-11339 >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Fix For: hbase-11339 > > Attachments: HBASE-11646-V2.diff, HBASE-11646-V3.diff, > HBASE-11646-V4.diff, HBASE-11646-V5.diff, HBASE-11646-V6.diff, > HBASE-11646-V7.diff, HBASE-11646.diff > > > In the updated MOB design however, admins can set CF level thresholds that > would force cell values > the threshold to use the MOB write path instead of > the traditional path. There are two cases where mobs need to interact with > this threshold > 1) How do we handle the case when the threshold size is changed? > 2) Today, you can bulkload hfiles that contain MOBs. These cells will work > as normal inside hbase. Unfortunately the cells with MOBs in them will never > benefit form the MOB write path. > The proposal here is to modify compaction in mob enabled cf's such that the > threshold value is honored with compactions. This handles case #1 -- > elements that should be moved out of the normal hfiles get 'compacted' into > refs and mob hfiles, and values that should be pulled into the cf get derefed > and written out wholy in the compaction. For case #2, we can maintain the > same behavior and compaction would move data into the mob writepath/lifecycle. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11646) Handle the MOB in compaction
[ https://issues.apache.org/jira/browse/HBASE-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14137287#comment-14137287 ] Anoop Sam John commented on HBASE-11646: +1 > Handle the MOB in compaction > > > Key: HBASE-11646 > URL: https://issues.apache.org/jira/browse/HBASE-11646 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: hbase-11339 >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Fix For: hbase-11339 > > Attachments: HBASE-11646-V2.diff, HBASE-11646-V3.diff, > HBASE-11646-V4.diff, HBASE-11646-V5.diff, HBASE-11646-V6.diff, > HBASE-11646-V7.diff, HBASE-11646.diff > > > In the updated MOB design however, admins can set CF level thresholds that > would force cell values > the threshold to use the MOB write path instead of > the traditional path. There are two cases where mobs need to interact with > this threshold > 1) How do we handle the case when the threshold size is changed? > 2) Today, you can bulkload hfiles that contain MOBs. These cells will work > as normal inside hbase. Unfortunately the cells with MOBs in them will never > benefit form the MOB write path. > The proposal here is to modify compaction in mob enabled cf's such that the > threshold value is honored with compactions. This handles case #1 -- > elements that should be moved out of the normal hfiles get 'compacted' into > refs and mob hfiles, and values that should be pulled into the cf get derefed > and written out wholy in the compaction. For case #2, we can maintain the > same behavior and compaction would move data into the mob writepath/lifecycle. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11646) Handle the MOB in compaction
[ https://issues.apache.org/jira/browse/HBASE-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14137012#comment-14137012 ] Hadoop QA commented on HBASE-11646: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12669376/HBASE-11646-V7.diff against trunk revision . ATTACHMENT ID: 12669376 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified tests. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10942//console This message is automatically generated. > Handle the MOB in compaction > > > Key: HBASE-11646 > URL: https://issues.apache.org/jira/browse/HBASE-11646 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: hbase-11339 >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Fix For: hbase-11339 > > Attachments: HBASE-11646-V2.diff, HBASE-11646-V3.diff, > HBASE-11646-V4.diff, HBASE-11646-V5.diff, HBASE-11646-V6.diff, > HBASE-11646-V7.diff, HBASE-11646.diff > > > In the updated MOB design however, admins can set CF level thresholds that > would force cell values > the threshold to use the MOB write path instead of > the traditional path. There are two cases where mobs need to interact with > this threshold > 1) How do we handle the case when the threshold size is changed? > 2) Today, you can bulkload hfiles that contain MOBs. These cells will work > as normal inside hbase. Unfortunately the cells with MOBs in them will never > benefit form the MOB write path. > The proposal here is to modify compaction in mob enabled cf's such that the > threshold value is honored with compactions. This handles case #1 -- > elements that should be moved out of the normal hfiles get 'compacted' into > refs and mob hfiles, and values that should be pulled into the cf get derefed > and written out wholy in the compaction. For case #2, we can maintain the > same behavior and compaction would move data into the mob writepath/lifecycle. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11646) Handle the MOB in compaction
[ https://issues.apache.org/jira/browse/HBASE-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131370#comment-14131370 ] Hadoop QA commented on HBASE-11646: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12668311/HBASE-11646-V6.diff against trunk revision . ATTACHMENT ID: 12668311 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified tests. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10859//console This message is automatically generated. > Handle the MOB in compaction > > > Key: HBASE-11646 > URL: https://issues.apache.org/jira/browse/HBASE-11646 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: hbase-11339 >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Fix For: hbase-11339 > > Attachments: HBASE-11646-V2.diff, HBASE-11646-V3.diff, > HBASE-11646-V4.diff, HBASE-11646-V5.diff, HBASE-11646-V6.diff, > HBASE-11646.diff > > > In the updated MOB design however, admins can set CF level thresholds that > would force cell values > the threshold to use the MOB write path instead of > the traditional path. There are two cases where mobs need to interact with > this threshold > 1) How do we handle the case when the threshold size is changed? > 2) Today, you can bulkload hfiles that contain MOBs. These cells will work > as normal inside hbase. Unfortunately the cells with MOBs in them will never > benefit form the MOB write path. > The proposal here is to modify compaction in mob enabled cf's such that the > threshold value is honored with compactions. This handles case #1 -- > elements that should be moved out of the normal hfiles get 'compacted' into > refs and mob hfiles, and values that should be pulled into the cf get derefed > and written out wholy in the compaction. For case #2, we can maintain the > same behavior and compaction would move data into the mob writepath/lifecycle. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11646) Handle the MOB in compaction
[ https://issues.apache.org/jira/browse/HBASE-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14102497#comment-14102497 ] Jonathan Hsieh commented on HBASE-11646: note -- the patch is being reviewed here https://reviews.apache.org/r/24736/ > Handle the MOB in compaction > > > Key: HBASE-11646 > URL: https://issues.apache.org/jira/browse/HBASE-11646 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Attachments: HBASE-11646.diff > > > In the updated MOB design however, admins can set CF level thresholds that > would force cell values > the threshold to use the MOB write path instead of > the traditional path. There are two cases where mobs need to interact with > this threshold > 1) How do we handle the case when the threshold size is changed? > 2) Today, you can bulkload hfiles that contain MOBs. These cells will work > as normal inside hbase. Unfortunately the cells with MOBs in them will never > benefit form the MOB write path. > The proposal here is to modify compaction in mob enabled cf's such that the > threshold value is honored with compactions. This handles case #1 -- > elements that should be moved out of the normal hfiles get 'compacted' into > refs and mob hfiles, and values that should be pulled into the cf get derefed > and written out wholy in the compaction. For case #2, we can maintain the > same behavior and compaction would move data into the mob writepath/lifecycle. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11646) Handle the MOB in compaction
[ https://issues.apache.org/jira/browse/HBASE-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087287#comment-14087287 ] Jingcheng Du commented on HBASE-11646: -- Thanks. I don't think I have. I agree with the bi-direction move. > Handle the MOB in compaction > > > Key: HBASE-11646 > URL: https://issues.apache.org/jira/browse/HBASE-11646 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Jingcheng Du >Assignee: Jingcheng Du > > In the updated MOB design however, admins can set CF level thresholds that > would force cell values > the threshold to use the MOB write path instead of > the traditional path. There are two cases where mobs need to interact with > this threshold > 1) How do we handle the case when the threshold size is changed? > 2) Today, you can bulkload hfiles that contain MOBs. These cells will work > as normal inside hbase. Unfortunately the cells with MOBs in them will never > benefit form the MOB write path. > The proposal here is to modify compaction in mob enabled cf's such that the > threshold value is honored with compactions. This handles case #1 -- > elements that should be moved out of the normal hfiles get 'compacted' into > refs and mob hfiles, and values that should be pulled into the cf get derefed > and written out wholy in the compaction. For case #2, we can maintain the > same behavior and compaction would move data into the mob writepath/lifecycle. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11646) Handle the MOB in compaction
[ https://issues.apache.org/jira/browse/HBASE-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087205#comment-14087205 ] Jonathan Hsieh commented on HBASE-11646: The most important thing is that hbase serves the correct data regardless of whether we move data into or out of the mob write path. My gut says we'd want to put a new copy of the mob them back into normal hfiles if the threshold is changed -- having it only go one way and being irreversible seems inconsistent. Can you come up with a case where we wouldn't want or expect the copy back into hfiles behavior? > Handle the MOB in compaction > > > Key: HBASE-11646 > URL: https://issues.apache.org/jira/browse/HBASE-11646 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Jingcheng Du >Assignee: Jingcheng Du > > In the updated MOB design however, admins can set CF level thresholds that > would force cell values > the threshold to use the MOB write path instead of > the traditional path. There are two cases where mobs need to interact with > this threshold > 1) How do we handle the case when the threshold size is changed? > 2) Today, you can bulkload hfiles that contain MOBs. These cells will work > as normal inside hbase. Unfortunately the cells with MOBs in them will never > benefit form the MOB write path. > The proposal here is to modify compaction in mob enabled cf's such that the > threshold value is honored with compactions. This handles case #1 -- > elements that should be moved out of the normal hfiles get 'compacted' into > refs and mob hfiles, and values that should be pulled into the cf get derefed > and written out wholy in the compaction. For case #2, we can maintain the > same behavior and compaction would move data into the mob writepath/lifecycle. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11646) Handle the MOB in compaction
[ https://issues.apache.org/jira/browse/HBASE-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087140#comment-14087140 ] Jingcheng Du commented on HBASE-11646: -- [~jmhsieh] Thanks Jon for the descriptions. for the case#1, if a threshold is set larger, do we need to save the mob data back to the HBase? Or just leave them in the mob files? > Handle the MOB in compaction > > > Key: HBASE-11646 > URL: https://issues.apache.org/jira/browse/HBASE-11646 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Jingcheng Du >Assignee: Jingcheng Du > > In the updated MOB design however, admins can set CF level thresholds that > would force cell values > the threshold to use the MOB write path instead of > the traditional path. There are two cases where mobs need to interact with > this threshold > 1) How do we handle the case when the threshold size is changed? > 2) Today, you can bulkload hfiles that contain MOBs. These cells will work > as normal inside hbase. Unfortunately the cells with MOBs in them will never > benefit form the MOB write path. > The proposal here is to modify compaction in mob enabled cf's such that the > threshold value is honored with compactions. This handles case #1 -- > elements that should be moved out of the normal hfiles get 'compacted' into > refs and mob hfiles, and values that should be pulled into the cf get derefed > and written out wholy in the compaction. For case #2, we can maintain the > same behavior and compaction would move data into the mob writepath/lifecycle. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11646) Handle the MOB in compaction
[ https://issues.apache.org/jira/browse/HBASE-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14086330#comment-14086330 ] Jonathan Hsieh commented on HBASE-11646: reorganized the description to make it easier to follow > Handle the MOB in compaction > > > Key: HBASE-11646 > URL: https://issues.apache.org/jira/browse/HBASE-11646 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Jingcheng Du >Assignee: Jingcheng Du > > For those MOB Cells loaded by the bulk load, they're saved in HBase. We need > handle them in HBase compaction to write them to the MOB files. -- This message was sent by Atlassian JIRA (v6.2#6252)