[jira] [Updated] (HBASE-11644) External MOB compaction tools
[ https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-11644: - Attachment: HBASE-11644-Sep-19.diff Update the patch based on the patch in handle mob in compaction. External MOB compaction tools - Key: HBASE-11644 URL: https://issues.apache.org/jira/browse/HBASE-11644 Project: HBase Issue Type: Sub-task Components: Compaction, master Affects Versions: hbase-11339 Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-11644-Sep-15.diff, HBASE-11644-Sep-16.diff, HBASE-11644-Sep-16.diff, HBASE-11644-Sep-18.diff, HBASE-11644-Sep-19.diff, HBASE-11644.diff, HBASE-11646-0918-bad.patch From the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11644) External MOB compaction tools
[ https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-11644: - Attachment: HBASE-11644-Sep-19-V2.patch Generate the patch(HBASE-11644-Sep-19-V2.patch) based on the latest code in hbase-11339. Thanks Anoop for the help! External MOB compaction tools - Key: HBASE-11644 URL: https://issues.apache.org/jira/browse/HBASE-11644 Project: HBase Issue Type: Sub-task Components: Compaction, master Affects Versions: hbase-11339 Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-11644-Sep-15.diff, HBASE-11644-Sep-16.diff, HBASE-11644-Sep-16.diff, HBASE-11644-Sep-18.diff, HBASE-11644-Sep-19-V2.patch, HBASE-11644-Sep-19.diff, HBASE-11644.diff, HBASE-11646-0918-bad.patch From the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11644) External MOB compaction tools
[ https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-11644: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) thanks jingcheng, and thans for reviews anoop. I've committed to the hbase-11339 branch. External MOB compaction tools - Key: HBASE-11644 URL: https://issues.apache.org/jira/browse/HBASE-11644 Project: HBase Issue Type: Sub-task Components: Compaction, master Affects Versions: hbase-11339 Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-11644-Sep-15.diff, HBASE-11644-Sep-16.diff, HBASE-11644-Sep-16.diff, HBASE-11644-Sep-18.diff, HBASE-11644-Sep-19-V2.patch, HBASE-11644-Sep-19.diff, HBASE-11644.diff, HBASE-11646-0918-bad.patch From the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11644) External MOB compaction tools
[ https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-11644: - Attachment: HBASE-11644-Sep-18.diff Update the patch according to Anoop's comments in RB. 1. Several nits. 2. Move the SweepJobNodeTracker to setup() instead of in map(). External MOB compaction tools - Key: HBASE-11644 URL: https://issues.apache.org/jira/browse/HBASE-11644 Project: HBase Issue Type: Sub-task Components: Compaction, master Affects Versions: hbase-11339 Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-11644-Sep-15.diff, HBASE-11644-Sep-16.diff, HBASE-11644-Sep-16.diff, HBASE-11644-Sep-18.diff, HBASE-11644.diff From the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11644) External MOB compaction tools
[ https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-11644: --- Attachment: HBASE-11646-0918-bad.patch External MOB compaction tools - Key: HBASE-11644 URL: https://issues.apache.org/jira/browse/HBASE-11644 Project: HBase Issue Type: Sub-task Components: Compaction, master Affects Versions: hbase-11339 Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-11644-Sep-15.diff, HBASE-11644-Sep-16.diff, HBASE-11644-Sep-16.diff, HBASE-11644-Sep-18.diff, HBASE-11644.diff, HBASE-11646-0918-bad.patch From the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11644) External MOB compaction tools
[ https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-11644: - Attachment: HBASE-11644-Sep-16.diff Upload the latest patch( Sep-16), add a necessary comment to the code. External MOB compaction tools - Key: HBASE-11644 URL: https://issues.apache.org/jira/browse/HBASE-11644 Project: HBase Issue Type: Sub-task Components: Compaction, master Affects Versions: hbase-11339 Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-11644-Sep-15.diff, HBASE-11644-Sep-16.diff, HBASE-11644-Sep-16.diff, HBASE-11644.diff From the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11644) External MOB compaction tools
[ https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-11644: - Attachment: HBASE-11644-Sep-15.diff Update the patch according to Jon, Anoop and Ram's comments. External MOB compaction tools - Key: HBASE-11644 URL: https://issues.apache.org/jira/browse/HBASE-11644 Project: HBase Issue Type: Sub-task Components: Compaction, master Affects Versions: hbase-11339 Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-11644-Sep-15.diff, HBASE-11644.diff From the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11644) External MOB compaction tools
[ https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-11644: - Attachment: HBASE-11644-Sep-16.diff Update the patch (Sep-16) according to Anoop's comments in RB. Fix some typos in the patch. External MOB compaction tools - Key: HBASE-11644 URL: https://issues.apache.org/jira/browse/HBASE-11644 Project: HBase Issue Type: Sub-task Components: Compaction, master Affects Versions: hbase-11339 Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-11644-Sep-15.diff, HBASE-11644-Sep-16.diff, HBASE-11644.diff From the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11644) External MOB compaction tools
[ https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-11644: --- Affects Version/s: hbase-11339 Fix Version/s: hbase-11339 External MOB compaction tools - Key: HBASE-11644 URL: https://issues.apache.org/jira/browse/HBASE-11644 Project: HBase Issue Type: Sub-task Components: Compaction, master Affects Versions: hbase-11339 Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-11644.diff From the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11644) External MOB compaction tools
[ https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-11644: --- Status: Patch Available (was: Open) External MOB compaction tools - Key: HBASE-11644 URL: https://issues.apache.org/jira/browse/HBASE-11644 Project: HBase Issue Type: Sub-task Components: Compaction, master Reporter: Jingcheng Du Assignee: Jingcheng Du Attachments: HBASE-11644.diff From the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11644) External MOB compaction tools
[ https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-11644: - Attachment: HBASE-11644.diff In this patch, there're two tools, one is a file cleaner which cleans the expired mob files by TTL, the other is a sweep tool which merges the small files to bigger ones. The cleaner is run by HMaster periodically, and the sweep tool is triggered by users. External MOB compaction tools - Key: HBASE-11644 URL: https://issues.apache.org/jira/browse/HBASE-11644 Project: HBase Issue Type: Sub-task Components: Compaction, master Reporter: Jingcheng Du Assignee: Jingcheng Du Attachments: HBASE-11644.diff From the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11644) External MOB compaction tools
[ https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-11644: --- Summary: External MOB compaction tools (was: External tools) External MOB compaction tools - Key: HBASE-11644 URL: https://issues.apache.org/jira/browse/HBASE-11644 Project: HBase Issue Type: Sub-task Components: Compaction, master Reporter: Jingcheng Du Assignee: Jingcheng Du The MOB files are involved in the HBase compaction. It means there's no chance to delete and merge the MOB files. The external tools do this, one is a cleaner to clean the MOB files that are expired (by TTL and minVersions), the other one is a sweep tool to clean the deleted Cells in HBase and merge small files into bigger ones. These tools are triggered by users. Besides, the cleaner could be a chore in HMaster. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11644) External MOB compaction tools
[ https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-11644: --- Description: Form the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. was:The MOB files are involved in the HBase compaction. It means there's no chance to delete and merge the MOB files. The external tools do this, one is a cleaner to clean the MOB files that are expired (by TTL and minVersions), the other one is a sweep tool to clean the deleted Cells in HBase and merge small files into bigger ones. These tools are triggered by users. Besides, the cleaner could be a chore in HMaster. External MOB compaction tools - Key: HBASE-11644 URL: https://issues.apache.org/jira/browse/HBASE-11644 Project: HBase Issue Type: Sub-task Components: Compaction, master Reporter: Jingcheng Du Assignee: Jingcheng Du Form the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11644) External MOB compaction tools
[ https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-11644: --- Description: From the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. was: Form the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. External MOB compaction tools - Key: HBASE-11644 URL: https://issues.apache.org/jira/browse/HBASE-11644 Project: HBase Issue Type: Sub-task Components: Compaction, master Reporter: Jingcheng Du Assignee: Jingcheng Du From the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. -- This message was sent by Atlassian JIRA (v6.2#6252)
