[jira] [Commented] (HBASE-12712) skipLargeFiles in minor compact but not in major compact
[ https://issues.apache.org/jira/browse/HBASE-12712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15072353#comment-15072353 ] Junegunn Choi commented on HBASE-12712: --- We're having a related issue. In our case, it's not the number of versions, but TTL of the column family. We expected old (and large) storefiles to be removed from the system but they are not because skipLargeFile excludes them and thus major compaction is never triggered for them. It seems trivial to make the method take TTL into account, i.e. do not skip storefiles whose minimum timestamps are older than TTL. However, I'm not completely sure if it's the right way to do it as one may argue that the current implementation is not "wrong" and "hbase.hstore.compaction.max.size" simply has priority over TTL. Also it does not fix the problem [~mopishv0] is having. > skipLargeFiles in minor compact but not in major compact > > > Key: HBASE-12712 > URL: https://issues.apache.org/jira/browse/HBASE-12712 > Project: HBase > Issue Type: New Feature > Components: Compaction >Affects Versions: 0.98.6 >Reporter: Liu Junhong > Labels: beginner > Fix For: 0.98.6 > > Attachments: compact.diff > > Original Estimate: 72h > Remaining Estimate: 72h > > Here is my case. After repeatedly minor compaction, the size of storefile is > very large. Compaction with large storefile will waste much bandwidth, so i > use the “hbase.hstore.compaction.max.size” to skip this case. But after use > this config, i find that major compaction will be skipped forever when i read > the source code and the deletes and muti-versions data my waste storage. So i > had to modify the code. > Now i'm try to submit my patch.But my patch is not perfect. I think there > should be an other config to determine if the large size storefile should > join major compaction in HColumnDescriptor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12712) skipLargeFiles in minor compact but not in major compact
[ https://issues.apache.org/jira/browse/HBASE-12712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260304#comment-14260304 ] stack commented on HBASE-12712: --- bq. In my case, i need major compaction can be triggered when forceMajor is setted false. You need a major compaction even though majorCompaction is false? Or do you mean it is a minor compaction but all files have been selected? skipLargeFiles in minor compact but not in major compact Key: HBASE-12712 URL: https://issues.apache.org/jira/browse/HBASE-12712 Project: HBase Issue Type: New Feature Components: Compaction Affects Versions: 0.98.6 Reporter: Liu Junhong Labels: newbie, patch Fix For: 0.98.6 Attachments: compact.diff Original Estimate: 72h Remaining Estimate: 72h Here is my case. After repeatedly minor compaction, the size of storefile is very large. Compaction with large storefile will waste much bandwidth, so i use the “hbase.hstore.compaction.max.size” to skip this case. But after use this config, i find that major compaction will be skipped forever when i read the source code and the deletes and muti-versions data my waste storage. So i had to modify the code. Now i'm try to submit my patch.But my patch is not perfect. I think there should be an other config to determine if the large size storefile should join major compaction in HColumnDescriptor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12712) skipLargeFiles in minor compact but not in major compact
[ https://issues.apache.org/jira/browse/HBASE-12712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260837#comment-14260837 ] Liu Junhong commented on HBASE-12712: - I need a major compaction triggered by compactionChecker even though i set hbase.hstore.compaction.max.size. In this case majorCompaction should be true when the function isMajorCompaction returns true. skipLargeFiles in minor compact but not in major compact Key: HBASE-12712 URL: https://issues.apache.org/jira/browse/HBASE-12712 Project: HBase Issue Type: New Feature Components: Compaction Affects Versions: 0.98.6 Reporter: Liu Junhong Labels: newbie, patch Fix For: 0.98.6 Attachments: compact.diff Original Estimate: 72h Remaining Estimate: 72h Here is my case. After repeatedly minor compaction, the size of storefile is very large. Compaction with large storefile will waste much bandwidth, so i use the “hbase.hstore.compaction.max.size” to skip this case. But after use this config, i find that major compaction will be skipped forever when i read the source code and the deletes and muti-versions data my waste storage. So i had to modify the code. Now i'm try to submit my patch.But my patch is not perfect. I think there should be an other config to determine if the large size storefile should join major compaction in HColumnDescriptor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12712) skipLargeFiles in minor compact but not in major compact
[
https://issues.apache.org/jira/browse/HBASE-12712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251919#comment-14251919
]
Ted Yu commented on HBASE-12712:
In master branch, we have:
{code}
if (!(forceMajor isAllFiles)) {
candidateSelection = skipLargeFiles(candidateSelection);
isAllFiles = candidateFiles.size() == candidateSelection.size();
}
{code}
In 0.98, we have:
{code}
if (!forceMajor) {
candidateSelection = skipLargeFiles(candidateSelection);
}
{code}
Can you try 0.98.8 to see if the problem is still there ?
skipLargeFiles in minor compact but not in major compact
Key: HBASE-12712
URL: https://issues.apache.org/jira/browse/HBASE-12712
Project: HBase
Issue Type: New Feature
Components: Compaction
Affects Versions: 0.98.6
Reporter: Liu Junhong
Labels: newbie, patch
Fix For: 0.98.6
Attachments: compact.diff
Original Estimate: 72h
Remaining Estimate: 72h
Here is my case. After repeatedly minor compaction, the size of storefile is
very large. Compaction with large storefile will waste much bandwidth, so i
use the “hbase.hstore.compaction.max.size” to skip this case. But after use
this config, i find that major compaction will be skipped forever when i read
the source code and the deletes and muti-versions data my waste storage. So i
had to modify the code.
Now i'm try to submit my patch.But my patch is not perfect. I think there
should be an other config to determine if the large size storefile should
join major compaction in HColumnDescriptor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12712) skipLargeFiles in minor compact but not in major compact
[ https://issues.apache.org/jira/browse/HBASE-12712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252999#comment-14252999 ] Liu Junhong commented on HBASE-12712: - The value of forceMajor will be only setted to true when trigger major compaction manually. In my case we need trigger major compaction by CompactionChecker. The master branch can solve this case. Thank you, i'll close this issue. skipLargeFiles in minor compact but not in major compact Key: HBASE-12712 URL: https://issues.apache.org/jira/browse/HBASE-12712 Project: HBase Issue Type: New Feature Components: Compaction Affects Versions: 0.98.6 Reporter: Liu Junhong Labels: newbie, patch Fix For: 0.98.6 Attachments: compact.diff Original Estimate: 72h Remaining Estimate: 72h Here is my case. After repeatedly minor compaction, the size of storefile is very large. Compaction with large storefile will waste much bandwidth, so i use the “hbase.hstore.compaction.max.size” to skip this case. But after use this config, i find that major compaction will be skipped forever when i read the source code and the deletes and muti-versions data my waste storage. So i had to modify the code. Now i'm try to submit my patch.But my patch is not perfect. I think there should be an other config to determine if the large size storefile should join major compaction in HColumnDescriptor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12712) skipLargeFiles in minor compact but not in major compact
[ https://issues.apache.org/jira/browse/HBASE-12712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14253004#comment-14253004 ] Liu Junhong commented on HBASE-12712: - Sorry i misunderstanding the code in master branch. It will skipLargeFiles too when forceMajor is setted false. In my case, i need major compaction can be triggered when forceMajor is setted false. Is my case a widespread requirement? skipLargeFiles in minor compact but not in major compact Key: HBASE-12712 URL: https://issues.apache.org/jira/browse/HBASE-12712 Project: HBase Issue Type: New Feature Components: Compaction Affects Versions: 0.98.6 Reporter: Liu Junhong Labels: newbie, patch Fix For: 0.98.6 Attachments: compact.diff Original Estimate: 72h Remaining Estimate: 72h Here is my case. After repeatedly minor compaction, the size of storefile is very large. Compaction with large storefile will waste much bandwidth, so i use the “hbase.hstore.compaction.max.size” to skip this case. But after use this config, i find that major compaction will be skipped forever when i read the source code and the deletes and muti-versions data my waste storage. So i had to modify the code. Now i'm try to submit my patch.But my patch is not perfect. I think there should be an other config to determine if the large size storefile should join major compaction in HColumnDescriptor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12712) skipLargeFiles in minor compact but not in major compact
[
https://issues.apache.org/jira/browse/HBASE-12712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251295#comment-14251295
]
Hadoop QA commented on HBASE-12712:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12687977/compact.diff
against master branch at commit 83e4bfaf73e1c7db16835b20c4f996adde30054a.
ATTACHMENT ID: 12687977
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12131//console
This message is automatically generated.
skipLargeFiles in minor compact but not in major compact
Key: HBASE-12712
URL: https://issues.apache.org/jira/browse/HBASE-12712
Project: HBase
Issue Type: New Feature
Components: Compaction
Affects Versions: 0.98.6
Reporter: Liu Junhong
Labels: newbie, patch
Fix For: 0.98.6
Attachments: compact.diff
Original Estimate: 72h
Remaining Estimate: 72h
Here is my case. After repeatedly minor compaction, the size of storefile is
very large. Compaction with large storefile will waste much bandwidth, so i
use the “hbase.hstore.compaction.max.size” to skip this case. But after use
this config, i find that major compaction will be skipped forever when i read
the source code and the deletes and muti-versions data my waste storage. So i
had to modify the code.
Now i'm try to submit my patch.But my patch is not perfect. I think there
should be an other config to determine if the large size storefile should
join major compaction in HColumnDescriptor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
