[jira] [Commented] (HBASE-11644) External MOB compaction tools
[ https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14637844#comment-14637844 ] Hudson commented on HBASE-11644: FAILURE: Integrated in HBase-TRUNK #6672 (See [https://builds.apache.org/job/HBase-TRUNK/6672/]) HBASE-11644 External MOB compaction tools (Jingcheng Du) (jmhsieh: rev 84e957c875ae971578a5b147775445368ea26188) * hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobConstants.java * hbase-server/src/test/java/org/apache/hadoop/hbase/mob/mapreduce/TestMobSweepJob.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepMapper.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMobStoreScanner.java * hbase-server/src/test/java/org/apache/hadoop/hbase/mob/mapreduce/TestMobSweeper.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobUtils.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/MemStoreWrapper.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMobCompaction.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/Sweeper.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mob/ExpiredMobFileCleaner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MobReferenceOnlyFilter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/ExpiredMobFileCleanerChore.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepJob.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepJobNodeTracker.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobZookeeper.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HMobStore.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepReducer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestExpiredMobFileCleaner.java * hbase-server/src/test/java/org/apache/hadoop/hbase/mob/mapreduce/TestMobSweepMapper.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/MobFilePathHashPartitioner.java * hbase-common/src/main/resources/hbase-default.xml * hbase-server/src/test/java/org/apache/hadoop/hbase/mob/mapreduce/TestMobSweepReducer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobCompactor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java External MOB compaction tools - Key: HBASE-11644 URL: https://issues.apache.org/jira/browse/HBASE-11644 Project: HBase Issue Type: Sub-task Components: Compaction, master Affects Versions: hbase-11339 Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-11644-Sep-15.diff, HBASE-11644-Sep-16.diff, HBASE-11644-Sep-16.diff, HBASE-11644-Sep-18.diff, HBASE-11644-Sep-19-V2.patch, HBASE-11644-Sep-19.diff, HBASE-11644.diff, HBASE-11646-0918-bad.patch From the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11644) External MOB compaction tools
[
https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14140231#comment-14140231
]
Hadoop QA commented on HBASE-11644:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12669957/HBASE-11644-Sep-19.diff
against trunk revision .
ATTACHMENT ID: 12669957
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 19 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10992//console
This message is automatically generated.
External MOB compaction tools
-
Key: HBASE-11644
URL: https://issues.apache.org/jira/browse/HBASE-11644
Project: HBase
Issue Type: Sub-task
Components: Compaction, master
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Fix For: hbase-11339
Attachments: HBASE-11644-Sep-15.diff, HBASE-11644-Sep-16.diff,
HBASE-11644-Sep-16.diff, HBASE-11644-Sep-18.diff, HBASE-11644-Sep-19.diff,
HBASE-11644.diff, HBASE-11646-0918-bad.patch
From the design doc, mob files are not involved in the normal HBase
compaction process. This means deleted mobs would still take up space and
that we never really merge mob files that accrue over time. Currently, MOBs
depend on two external tools:
1) A TTL cleaner that removes mobs that have passed their TTL or exceeded
minVersions.
2) A 'sweep tool' cleaner that remove mobs that have had their references
deleted and merges small files into larger ones.
Today the tools are triggered by admins. The longer term goal would be to
integrate them into hbase such that by default mobs are cleaned. The tools
will be preserved however so that advanced admins can disable automatic
cleanups and manually trigger these compaction like operaitons. #1 would
likely be a chore in the master while #2 requires some design work to
integrate into hbase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11644) External MOB compaction tools
[ https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14140235#comment-14140235 ] Jingcheng Du commented on HBASE-11644: -- And use mvn clean test -Dtest=TestMob* to pass the testing in the patch. External MOB compaction tools - Key: HBASE-11644 URL: https://issues.apache.org/jira/browse/HBASE-11644 Project: HBase Issue Type: Sub-task Components: Compaction, master Affects Versions: hbase-11339 Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-11644-Sep-15.diff, HBASE-11644-Sep-16.diff, HBASE-11644-Sep-16.diff, HBASE-11644-Sep-18.diff, HBASE-11644-Sep-19.diff, HBASE-11644.diff, HBASE-11646-0918-bad.patch From the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11644) External MOB compaction tools
[
https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14140259#comment-14140259
]
Hadoop QA commented on HBASE-11644:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12669962/HBASE-11644-Sep-19-V2.patch
against trunk revision .
ATTACHMENT ID: 12669962
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 19 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10993//console
This message is automatically generated.
External MOB compaction tools
-
Key: HBASE-11644
URL: https://issues.apache.org/jira/browse/HBASE-11644
Project: HBase
Issue Type: Sub-task
Components: Compaction, master
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Fix For: hbase-11339
Attachments: HBASE-11644-Sep-15.diff, HBASE-11644-Sep-16.diff,
HBASE-11644-Sep-16.diff, HBASE-11644-Sep-18.diff,
HBASE-11644-Sep-19-V2.patch, HBASE-11644-Sep-19.diff, HBASE-11644.diff,
HBASE-11646-0918-bad.patch
From the design doc, mob files are not involved in the normal HBase
compaction process. This means deleted mobs would still take up space and
that we never really merge mob files that accrue over time. Currently, MOBs
depend on two external tools:
1) A TTL cleaner that removes mobs that have passed their TTL or exceeded
minVersions.
2) A 'sweep tool' cleaner that remove mobs that have had their references
deleted and merges small files into larger ones.
Today the tools are triggered by admins. The longer term goal would be to
integrate them into hbase such that by default mobs are cleaned. The tools
will be preserved however so that advanced admins can disable automatic
cleanups and manually trigger these compaction like operaitons. #1 would
likely be a chore in the master while #2 requires some design work to
integrate into hbase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11644) External MOB compaction tools
[ https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14140274#comment-14140274 ] Jonathan Hsieh commented on HBASE-11644: the sept19 v2 version compiles and seems to pass for with the 'mvn clean test -Dtest=test\*Mob\*' External MOB compaction tools - Key: HBASE-11644 URL: https://issues.apache.org/jira/browse/HBASE-11644 Project: HBase Issue Type: Sub-task Components: Compaction, master Affects Versions: hbase-11339 Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-11644-Sep-15.diff, HBASE-11644-Sep-16.diff, HBASE-11644-Sep-16.diff, HBASE-11644-Sep-18.diff, HBASE-11644-Sep-19-V2.patch, HBASE-11644-Sep-19.diff, HBASE-11644.diff, HBASE-11646-0918-bad.patch From the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11644) External MOB compaction tools
[
https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14138835#comment-14138835
]
Hadoop QA commented on HBASE-11644:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12669710/HBASE-11644-Sep-18.diff
against trunk revision .
ATTACHMENT ID: 12669710
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 16 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10967//console
This message is automatically generated.
External MOB compaction tools
-
Key: HBASE-11644
URL: https://issues.apache.org/jira/browse/HBASE-11644
Project: HBase
Issue Type: Sub-task
Components: Compaction, master
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Fix For: hbase-11339
Attachments: HBASE-11644-Sep-15.diff, HBASE-11644-Sep-16.diff,
HBASE-11644-Sep-16.diff, HBASE-11644-Sep-18.diff, HBASE-11644.diff
From the design doc, mob files are not involved in the normal HBase
compaction process. This means deleted mobs would still take up space and
that we never really merge mob files that accrue over time. Currently, MOBs
depend on two external tools:
1) A TTL cleaner that removes mobs that have passed their TTL or exceeded
minVersions.
2) A 'sweep tool' cleaner that remove mobs that have had their references
deleted and merges small files into larger ones.
Today the tools are triggered by admins. The longer term goal would be to
integrate them into hbase such that by default mobs are cleaned. The tools
will be preserved however so that advanced admins can disable automatic
cleanups and manually trigger these compaction like operaitons. #1 would
likely be a chore in the master while #2 requires some design work to
integrate into hbase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11644) External MOB compaction tools
[ https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14139839#comment-14139839 ] Jonathan Hsieh commented on HBASE-11644: hey [~jingchengdu], this latest patch has problems when applied to the hbase-11339 branch -- significantly more than HBASE-11646 had. Most of them are around TableName vs String in the admin flush argument. (this might also be due to patches being applied in a different order). I took a stab at working through most of them, but now several of the mob tests fail. (mvn clean test -Dtest=TestMob*) Can you rebase and please post patches that actually apply against the target hbase-11339 branch properly? I've attached the quick version I put together as hbase-11644-0918-bad.patch if you want to work from that. External MOB compaction tools - Key: HBASE-11644 URL: https://issues.apache.org/jira/browse/HBASE-11644 Project: HBase Issue Type: Sub-task Components: Compaction, master Affects Versions: hbase-11339 Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-11644-Sep-15.diff, HBASE-11644-Sep-16.diff, HBASE-11644-Sep-16.diff, HBASE-11644-Sep-18.diff, HBASE-11644.diff From the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11644) External MOB compaction tools
[
https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14139848#comment-14139848
]
Hadoop QA commented on HBASE-11644:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12669884/HBASE-11646-0918-bad.patch
against trunk revision .
ATTACHMENT ID: 12669884
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 27 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10983//console
This message is automatically generated.
External MOB compaction tools
-
Key: HBASE-11644
URL: https://issues.apache.org/jira/browse/HBASE-11644
Project: HBase
Issue Type: Sub-task
Components: Compaction, master
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Fix For: hbase-11339
Attachments: HBASE-11644-Sep-15.diff, HBASE-11644-Sep-16.diff,
HBASE-11644-Sep-16.diff, HBASE-11644-Sep-18.diff, HBASE-11644.diff,
HBASE-11646-0918-bad.patch
From the design doc, mob files are not involved in the normal HBase
compaction process. This means deleted mobs would still take up space and
that we never really merge mob files that accrue over time. Currently, MOBs
depend on two external tools:
1) A TTL cleaner that removes mobs that have passed their TTL or exceeded
minVersions.
2) A 'sweep tool' cleaner that remove mobs that have had their references
deleted and merges small files into larger ones.
Today the tools are triggered by admins. The longer term goal would be to
integrate them into hbase such that by default mobs are cleaned. The tools
will be preserved however so that advanced admins can disable automatic
cleanups and manually trigger these compaction like operaitons. #1 would
likely be a chore in the master while #2 requires some design work to
integrate into hbase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11644) External MOB compaction tools
[
https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135037#comment-14135037
]
Hadoop QA commented on HBASE-11644:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12668985/HBASE-11644-Sep-16.diff
against trunk revision .
ATTACHMENT ID: 12668985
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 16 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10917//console
This message is automatically generated.
External MOB compaction tools
-
Key: HBASE-11644
URL: https://issues.apache.org/jira/browse/HBASE-11644
Project: HBase
Issue Type: Sub-task
Components: Compaction, master
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Fix For: hbase-11339
Attachments: HBASE-11644-Sep-15.diff, HBASE-11644-Sep-16.diff,
HBASE-11644-Sep-16.diff, HBASE-11644.diff
From the design doc, mob files are not involved in the normal HBase
compaction process. This means deleted mobs would still take up space and
that we never really merge mob files that accrue over time. Currently, MOBs
depend on two external tools:
1) A TTL cleaner that removes mobs that have passed their TTL or exceeded
minVersions.
2) A 'sweep tool' cleaner that remove mobs that have had their references
deleted and merges small files into larger ones.
Today the tools are triggered by admins. The longer term goal would be to
integrate them into hbase such that by default mobs are cleaned. The tools
will be preserved however so that advanced admins can disable automatic
cleanups and manually trigger these compaction like operaitons. #1 would
likely be a chore in the master while #2 requires some design work to
integrate into hbase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11644) External MOB compaction tools
[
https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14133787#comment-14133787
]
Hadoop QA commented on HBASE-11644:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12668748/HBASE-11644-Sep-15.diff
against trunk revision .
ATTACHMENT ID: 12668748
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 16 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10888//console
This message is automatically generated.
External MOB compaction tools
-
Key: HBASE-11644
URL: https://issues.apache.org/jira/browse/HBASE-11644
Project: HBase
Issue Type: Sub-task
Components: Compaction, master
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Fix For: hbase-11339
Attachments: HBASE-11644-Sep-15.diff, HBASE-11644.diff
From the design doc, mob files are not involved in the normal HBase
compaction process. This means deleted mobs would still take up space and
that we never really merge mob files that accrue over time. Currently, MOBs
depend on two external tools:
1) A TTL cleaner that removes mobs that have passed their TTL or exceeded
minVersions.
2) A 'sweep tool' cleaner that remove mobs that have had their references
deleted and merges small files into larger ones.
Today the tools are triggered by admins. The longer term goal would be to
integrate them into hbase such that by default mobs are cleaned. The tools
will be preserved however so that advanced admins can disable automatic
cleanups and manually trigger these compaction like operaitons. #1 would
likely be a chore in the master while #2 requires some design work to
integrate into hbase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11644) External MOB compaction tools
[
https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14134879#comment-14134879
]
Hadoop QA commented on HBASE-11644:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12668958/HBASE-11644-Sep-16.diff
against trunk revision .
ATTACHMENT ID: 12668958
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 16 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10910//console
This message is automatically generated.
External MOB compaction tools
-
Key: HBASE-11644
URL: https://issues.apache.org/jira/browse/HBASE-11644
Project: HBase
Issue Type: Sub-task
Components: Compaction, master
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Fix For: hbase-11339
Attachments: HBASE-11644-Sep-15.diff, HBASE-11644-Sep-16.diff,
HBASE-11644.diff
From the design doc, mob files are not involved in the normal HBase
compaction process. This means deleted mobs would still take up space and
that we never really merge mob files that accrue over time. Currently, MOBs
depend on two external tools:
1) A TTL cleaner that removes mobs that have passed their TTL or exceeded
minVersions.
2) A 'sweep tool' cleaner that remove mobs that have had their references
deleted and merges small files into larger ones.
Today the tools are triggered by admins. The longer term goal would be to
integrate them into hbase such that by default mobs are cleaned. The tools
will be preserved however so that advanced admins can disable automatic
cleanups and manually trigger these compaction like operaitons. #1 would
likely be a chore in the master while #2 requires some design work to
integrate into hbase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11644) External MOB compaction tools
[
https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128425#comment-14128425
]
Hadoop QA commented on HBASE-11644:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12662941/HBASE-11644.diff
against trunk revision .
ATTACHMENT ID: 12662941
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10812//console
This message is automatically generated.
External MOB compaction tools
-
Key: HBASE-11644
URL: https://issues.apache.org/jira/browse/HBASE-11644
Project: HBase
Issue Type: Sub-task
Components: Compaction, master
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Fix For: hbase-11339
Attachments: HBASE-11644.diff
From the design doc, mob files are not involved in the normal HBase
compaction process. This means deleted mobs would still take up space and
that we never really merge mob files that accrue over time. Currently, MOBs
depend on two external tools:
1) A TTL cleaner that removes mobs that have passed their TTL or exceeded
minVersions.
2) A 'sweep tool' cleaner that remove mobs that have had their references
deleted and merges small files into larger ones.
Today the tools are triggered by admins. The longer term goal would be to
integrate them into hbase such that by default mobs are cleaned. The tools
will be preserved however so that advanced admins can disable automatic
cleanups and manually trigger these compaction like operaitons. #1 would
likely be a chore in the master while #2 requires some design work to
integrate into hbase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11644) External MOB compaction tools
[ https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122352#comment-14122352 ] Jingcheng Du commented on HBASE-11644: -- Now the latest patch is uploaded to RB, you could find it by https://reviews.apache.org/r/23676/. External MOB compaction tools - Key: HBASE-11644 URL: https://issues.apache.org/jira/browse/HBASE-11644 Project: HBase Issue Type: Sub-task Components: Compaction, master Reporter: Jingcheng Du Assignee: Jingcheng Du Attachments: HBASE-11644.diff From the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11644) External MOB compaction tools
[ https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117592#comment-14117592 ] Anoop Sam John commented on HBASE-11644: Can u put patch on RB. External MOB compaction tools - Key: HBASE-11644 URL: https://issues.apache.org/jira/browse/HBASE-11644 Project: HBase Issue Type: Sub-task Components: Compaction, master Reporter: Jingcheng Du Assignee: Jingcheng Du Attachments: HBASE-11644.diff From the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11644) External MOB compaction tools
[ https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14086324#comment-14086324 ] Jonathan Hsieh commented on HBASE-11644: I reorganized the description to make it easier to follow. External MOB compaction tools - Key: HBASE-11644 URL: https://issues.apache.org/jira/browse/HBASE-11644 Project: HBase Issue Type: Sub-task Components: Compaction, master Reporter: Jingcheng Du Assignee: Jingcheng Du The MOB files are involved in the HBase compaction. It means there's no chance to delete and merge the MOB files. The external tools do this, one is a cleaner to clean the MOB files that are expired (by TTL and minVersions), the other one is a sweep tool to clean the deleted Cells in HBase and merge small files into bigger ones. These tools are triggered by users. Besides, the cleaner could be a chore in HMaster. -- This message was sent by Atlassian JIRA (v6.2#6252)
