[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017909#comment-16017909 ] huaxiang sun commented on HBASE-18083: -- Hi [~carp84], it is interesting, HBASE-17215 is not enough to solve this new issue. Last time, we found tcp nagle issue with 0.98 based releases, which caused about 120 ms delay for continuous deletes. How fast is the cleaner thread finish one run? Thanks. > Make large/small file clean thread number configurable in HFileCleaner > -- > > Key: HBASE-18083 > URL: https://issues.apache.org/jira/browse/HBASE-18083 > Project: HBase > Issue Type: Bug >Reporter: Yu Li >Assignee: Yu Li > > Currently we have only one thread for both large and small file cleaning, but > when write pressure is huge we might need more cleaner threads, so we need to > make the thread number configurable. > We observed more than 1.8PB data in archive directory online due to business > access rate change, and this proposal is one of the necessary changes > required. > Default value of the configurations would still be left to 1 to keep low > pressure to NN for normal case. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16018477#comment-16018477 ] Yu Li commented on HBASE-18083: --- bq. Last time, we found tcp nagle issue with 0.98 based releases, which caused about 120 ms delay for continuous deletes Any JIRA opened for this one? bq. How fast is the cleaner thread finish one run Pretty fast in our case, and no obvious change in NN metrics after we opened 25 threads for each (large and small file deleting) > Make large/small file clean thread number configurable in HFileCleaner > -- > > Key: HBASE-18083 > URL: https://issues.apache.org/jira/browse/HBASE-18083 > Project: HBase > Issue Type: Bug >Reporter: Yu Li >Assignee: Yu Li > > Currently we have only one thread for both large and small file cleaning, but > when write pressure is huge we might need more cleaner threads, so we need to > make the thread number configurable. > We observed more than 1.8PB data in archive directory online due to business > access rate change, and this proposal is one of the necessary changes > required. > Default value of the configurations would still be left to 1 to keep low > pressure to NN for normal case. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[
https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16018838#comment-16018838
]
huaxiang sun commented on HBASE-18083:
--
Thanks [~carp84]. I believe it is HADOOP-2232.
{bq}
Pretty fast in our case, and no obvious change in NN metrics after we opened 25
threads for each (large and small file deleting)
{bq}
With 50 threads, it is fast enough to finish one loop. I think I got the
picture together with HBASE-18084, looking forward to your patch.
> Make large/small file clean thread number configurable in HFileCleaner
> --
>
> Key: HBASE-18083
> URL: https://issues.apache.org/jira/browse/HBASE-18083
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
>
> Currently we have only one thread for both large and small file cleaning, but
> when write pressure is huge we might need more cleaner threads, so we need to
> make the thread number configurable.
> We observed more than 1.8PB data in archive directory online due to business
> access rate change, and this proposal is one of the necessary changes
> required.
> Default value of the configurations would still be left to 1 to keep low
> pressure to NN for normal case.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[
https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072213#comment-16072213
]
Chia-Ping Tsai commented on HBASE-18083:
{code}
private boolean running;
private long deletedLargeFiles = 0L;
private long deletedSmallFiles = 0L;
{code}
Do we need to declare them as volatile or Atomic object? They may be used
concurrently.
{code}
List leftOverTasks = new ArrayList<>();
for (HFileDeleteTask task : largeFileQueue) {
leftOverTasks.add(task);
}
for (HFileDeleteTask task : smallFileQueue) {
leftOverTasks.add(task);
}
{code}
Would you please set the initial capacity for leftOverTasks? Is the addAll good
to go in?
If all configs used by HFileCleaner aren't changed, is HFileCleaner still reset
when onConfigurationChange is called?
> Make large/small file clean thread number configurable in HFileCleaner
> --
>
> Key: HBASE-18083
> URL: https://issues.apache.org/jira/browse/HBASE-18083
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 3.0.0
>
> Attachments: HBASE-18083.patch
>
>
> Currently we have only one thread for both large and small file cleaning, but
> when write pressure is huge we might need more cleaner threads, so we need to
> make the thread number configurable.
> We observed more than 1.8PB data in archive directory online due to business
> access rate change, and this proposal is one of the necessary changes
> required.
> Default value of the configurations would still be left to 1 to keep low
> pressure to NN for normal case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[
https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072227#comment-16072227
]
Hadoop QA commented on HBASE-18083:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
25s{color} | {color:red} hbase-server in master has 10 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 9s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha3. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}110m 56s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}152m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.procedure.TestMasterProcedureWalLease |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Issue | HBASE-18083 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12875464/HBASE-18083.patch |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 7133aa63336b 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ba54af3 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/7474/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/7474/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/7474/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/7474/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> Make large/small file clean thread number configurable in HFileCleaner
>
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[
https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073316#comment-16073316
]
Yu Li commented on HBASE-18083:
---
Thanks for the review [~chia7712]
bq. Do we need to declare them as volatile or Atomic object?
Yes, let me make them Atomic although these fields only used for testing now.
bq. Would you please set the initial capacity for leftOverTasks?
Ok, let's do this to reduce capacity grow actions
bq. Is the addAll good to go in?
Not sure since we need to add both largeFileQueue and smallFileQueue, if we use
addAll we will need to use {{addAll(int index, Collection c)}} and
there would be element shifts, maybe less efficient?
> Make large/small file clean thread number configurable in HFileCleaner
> --
>
> Key: HBASE-18083
> URL: https://issues.apache.org/jira/browse/HBASE-18083
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 3.0.0
>
> Attachments: HBASE-18083.patch
>
>
> Currently we have only one thread for both large and small file cleaning, but
> when write pressure is huge we might need more cleaner threads, so we need to
> make the thread number configurable.
> We observed more than 1.8PB data in archive directory online due to business
> access rate change, and this proposal is one of the necessary changes
> required.
> Default value of the configurations would still be left to 1 to keep low
> pressure to NN for normal case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[
https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073629#comment-16073629
]
Hadoop QA commented on HBASE-18083:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m
26s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
0s{color} | {color:red} hbase-server in master has 10 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
34m 38s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha3. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}179m 8s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}246m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.wal.TestAsyncLogRolling |
| | hadoop.hbase.master.procedure.TestMasterProcedureWalLease |
| | hadoop.hbase.regionserver.TestRegionReplicaFailover |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.10.1 Server=1.10.1 Image:yetus/hbase:757bf37 |
| JIRA Issue | HBASE-18083 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12875588/HBASE-18083.v2.patch |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 23e06c388a16 3.19.0-25-generic #26~14.04.1-Ubuntu SMP Fri Jul
24 21:16:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 193a980 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC3 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/7493/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/7493/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/7493/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/7493/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16075836#comment-16075836 ] Yu Li commented on HBASE-18083: --- The failed UT cases are irrelative to change here. [~chia7712] does the v2 patch look good to you? Thanks. > Make large/small file clean thread number configurable in HFileCleaner > -- > > Key: HBASE-18083 > URL: https://issues.apache.org/jira/browse/HBASE-18083 > Project: HBase > Issue Type: Bug >Reporter: Yu Li >Assignee: Yu Li > Fix For: 3.0.0 > > Attachments: HBASE-18083.patch, HBASE-18083.v2.patch > > > Currently we have only one thread for both large and small file cleaning, but > when write pressure is huge we might need more cleaner threads, so we need to > make the thread number configurable. > We observed more than 1.8PB data in archive directory online due to business > access rate change, and this proposal is one of the necessary changes > required. > Default value of the configurations would still be left to 1 to keep low > pressure to NN for normal case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[
https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16075840#comment-16075840
]
Chia-Ping Tsai commented on HBASE-18083:
v2 LGTM. I had a question shown below.
{noformat}
If all configs used by HFileCleaner aren't changed, is HFileCleaner still reset
when onConfigurationChange is called?
{noformat}
Is the cost of reset is low?
> Make large/small file clean thread number configurable in HFileCleaner
> --
>
> Key: HBASE-18083
> URL: https://issues.apache.org/jira/browse/HBASE-18083
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 3.0.0
>
> Attachments: HBASE-18083.patch, HBASE-18083.v2.patch
>
>
> Currently we have only one thread for both large and small file cleaning, but
> when write pressure is huge we might need more cleaner threads, so we need to
> make the thread number configurable.
> We observed more than 1.8PB data in archive directory online due to business
> access rate change, and this proposal is one of the necessary changes
> required.
> Default value of the configurations would still be left to 1 to keep low
> pressure to NN for normal case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076019#comment-16076019 ] Yu Li commented on HBASE-18083: --- Oops, somehow neglected this question... Makes sense to reduce the cost when no changes made, let me update the patch > Make large/small file clean thread number configurable in HFileCleaner > -- > > Key: HBASE-18083 > URL: https://issues.apache.org/jira/browse/HBASE-18083 > Project: HBase > Issue Type: Bug >Reporter: Yu Li >Assignee: Yu Li > Fix For: 3.0.0 > > Attachments: HBASE-18083.patch, HBASE-18083.v2.patch > > > Currently we have only one thread for both large and small file cleaning, but > when write pressure is huge we might need more cleaner threads, so we need to > make the thread number configurable. > We observed more than 1.8PB data in archive directory online due to business > access rate change, and this proposal is one of the necessary changes > required. > Default value of the configurations would still be left to 1 to keep low > pressure to NN for normal case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[
https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076027#comment-16076027
]
Yu Li commented on HBASE-18083:
---
Below is what the logs look like in UT:
{noformat}
2017-07-06 14:31:00,908 DEBUG [Time-limited test] cleaner.HFileCleaner(437):
Updating throttle point, from 524288 to 1024
2017-07-06 14:31:00,908 DEBUG [Time-limited test] cleaner.HFileCleaner(444):
Updating largeQueueInitSize, from 512 to 1024
2017-07-06 14:31:00,908 DEBUG [Time-limited test] cleaner.HFileCleaner(452):
Updating smallQueueInitSize, from 512 to 1024
2017-07-06 14:31:00,908 DEBUG [Time-limited test] cleaner.HFileCleaner(460):
Updating largeFileDeleteThreadNumber, from 1 to 2
2017-07-06 14:31:00,908 DEBUG [Time-limited test] cleaner.HFileCleaner(468):
Updating smallFileDeleteThreadNumber, from 1 to 4
2017-07-06 14:31:00,908 DEBUG [Time-limited test] cleaner.HFileCleaner(295):
Stopping file delete threads
...
2017-07-06 14:31:00,912 DEBUG [Time-limited test] cleaner.HFileCleaner(404):
Update configuration triggered but nothing changed for this cleaner
...
{noformat}
> Make large/small file clean thread number configurable in HFileCleaner
> --
>
> Key: HBASE-18083
> URL: https://issues.apache.org/jira/browse/HBASE-18083
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 3.0.0
>
> Attachments: HBASE-18083.patch, HBASE-18083.v2.patch,
> HBASE-18083.v3.patch
>
>
> Currently we have only one thread for both large and small file cleaning, but
> when write pressure is huge we might need more cleaner threads, so we need to
> make the thread number configurable.
> We observed more than 1.8PB data in archive directory online due to business
> access rate change, and this proposal is one of the necessary changes
> required.
> Default value of the configurations would still be left to 1 to keep low
> pressure to NN for normal case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076058#comment-16076058 ] Ashish Singhi commented on HBASE-18083: --- v3 lgtm > Make large/small file clean thread number configurable in HFileCleaner > -- > > Key: HBASE-18083 > URL: https://issues.apache.org/jira/browse/HBASE-18083 > Project: HBase > Issue Type: Bug >Reporter: Yu Li >Assignee: Yu Li > Fix For: 3.0.0 > > Attachments: HBASE-18083.patch, HBASE-18083.v2.patch, > HBASE-18083.v3.patch > > > Currently we have only one thread for both large and small file cleaning, but > when write pressure is huge we might need more cleaner threads, so we need to > make the thread number configurable. > We observed more than 1.8PB data in archive directory online due to business > access rate change, and this proposal is one of the necessary changes > required. > Default value of the configurations would still be left to 1 to keep low > pressure to NN for normal case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076098#comment-16076098 ] Chia-Ping Tsai commented on HBASE-18083: +1 on v3 > Make large/small file clean thread number configurable in HFileCleaner > -- > > Key: HBASE-18083 > URL: https://issues.apache.org/jira/browse/HBASE-18083 > Project: HBase > Issue Type: Bug >Reporter: Yu Li >Assignee: Yu Li > Fix For: 3.0.0 > > Attachments: HBASE-18083.patch, HBASE-18083.v2.patch, > HBASE-18083.v3.patch > > > Currently we have only one thread for both large and small file cleaning, but > when write pressure is huge we might need more cleaner threads, so we need to > make the thread number configurable. > We observed more than 1.8PB data in archive directory online due to business > access rate change, and this proposal is one of the necessary changes > required. > Default value of the configurations would still be left to 1 to keep low > pressure to NN for normal case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076132#comment-16076132 ] Yu Li commented on HBASE-18083: --- Thanks for review [~ashish singhi] and [~chia7712], will commit soon if HadoopQA looks good. > Make large/small file clean thread number configurable in HFileCleaner > -- > > Key: HBASE-18083 > URL: https://issues.apache.org/jira/browse/HBASE-18083 > Project: HBase > Issue Type: Bug >Reporter: Yu Li >Assignee: Yu Li > Fix For: 3.0.0 > > Attachments: HBASE-18083.patch, HBASE-18083.v2.patch, > HBASE-18083.v3.patch > > > Currently we have only one thread for both large and small file cleaning, but > when write pressure is huge we might need more cleaner threads, so we need to > make the thread number configurable. > We observed more than 1.8PB data in archive directory online due to business > access rate change, and this proposal is one of the necessary changes > required. > Default value of the configurations would still be left to 1 to keep low > pressure to NN for normal case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[
https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076227#comment-16076227
]
Hadoop QA commented on HBASE-18083:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
45s{color} | {color:red} hbase-server in master has 10 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 7s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha3. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}110m
12s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}153m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:757bf37 |
| JIRA Issue | HBASE-18083 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12875863/HBASE-18083.v3.patch |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 7abae7ed31d0 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 50bb045 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC3 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/7537/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/7537/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/7537/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> Make large/small file clean thread number configurable in HFileCleaner
> --
>
> Key: HBASE-18083
> URL: https://issues.apache.org/jira/browse/HBASE-18083
> Project: HBase
> I
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[
https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076269#comment-16076269
]
Ted Yu commented on HBASE-18083:
{code}
72public final static String LARGE_HFILE_DELETE_THREAD_NUMBER =
73"hbase.regionserver.hfilecleaner.large.threadnumber";
{code}
threadnumber tends to refer to the thread number for a particular thread.
We have existing config such as:
{code}
public static final String REGION_SERVER_HANDLER_COUNT =
"hbase.regionserver.handler.count";
{code}
I think 'thread.count' would be more informative as to the purpose of the
config.
> Make large/small file clean thread number configurable in HFileCleaner
> --
>
> Key: HBASE-18083
> URL: https://issues.apache.org/jira/browse/HBASE-18083
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 3.0.0
>
> Attachments: HBASE-18083.patch, HBASE-18083.v2.patch,
> HBASE-18083.v3.patch
>
>
> Currently we have only one thread for both large and small file cleaning, but
> when write pressure is huge we might need more cleaner threads, so we need to
> make the thread number configurable.
> We observed more than 1.8PB data in archive directory online due to business
> access rate change, and this proposal is one of the necessary changes
> required.
> Default value of the configurations would still be left to 1 to keep low
> pressure to NN for normal case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076280#comment-16076280 ] Yu Li commented on HBASE-18083: --- bq. I think 'thread.count' would be more informative as to the purpose of the config Ok, mind if I apply the change when commit sir? [~tedyu] Thanks. > Make large/small file clean thread number configurable in HFileCleaner > -- > > Key: HBASE-18083 > URL: https://issues.apache.org/jira/browse/HBASE-18083 > Project: HBase > Issue Type: Bug >Reporter: Yu Li >Assignee: Yu Li > Fix For: 3.0.0 > > Attachments: HBASE-18083.patch, HBASE-18083.v2.patch, > HBASE-18083.v3.patch > > > Currently we have only one thread for both large and small file cleaning, but > when write pressure is huge we might need more cleaner threads, so we need to > make the thread number configurable. > We observed more than 1.8PB data in archive directory online due to business > access rate change, and this proposal is one of the necessary changes > required. > Default value of the configurations would still be left to 1 to keep low > pressure to NN for normal case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076283#comment-16076283 ] Ted Yu commented on HBASE-18083: Should be fine. > Make large/small file clean thread number configurable in HFileCleaner > -- > > Key: HBASE-18083 > URL: https://issues.apache.org/jira/browse/HBASE-18083 > Project: HBase > Issue Type: Bug >Reporter: Yu Li >Assignee: Yu Li > Fix For: 3.0.0 > > Attachments: HBASE-18083.patch, HBASE-18083.v2.patch, > HBASE-18083.v3.patch > > > Currently we have only one thread for both large and small file cleaning, but > when write pressure is huge we might need more cleaner threads, so we need to > make the thread number configurable. > We observed more than 1.8PB data in archive directory online due to business > access rate change, and this proposal is one of the necessary changes > required. > Default value of the configurations would still be left to 1 to keep low > pressure to NN for normal case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076291#comment-16076291 ] Yu Li commented on HBASE-18083: --- It seems to me the last HadoopQA should be "+1 overall" but it gave -1, is this by design or something went wrong? Could you take a look here? [~busbey] [~stack] Thanks. > Make large/small file clean thread number configurable in HFileCleaner > -- > > Key: HBASE-18083 > URL: https://issues.apache.org/jira/browse/HBASE-18083 > Project: HBase > Issue Type: Bug >Reporter: Yu Li >Assignee: Yu Li > Fix For: 3.0.0 > > Attachments: HBASE-18083.patch, HBASE-18083.v2.patch, > HBASE-18083.v3.patch > > > Currently we have only one thread for both large and small file cleaning, but > when write pressure is huge we might need more cleaner threads, so we need to > make the thread number configurable. > We observed more than 1.8PB data in archive directory online due to business > access rate change, and this proposal is one of the necessary changes > required. > Default value of the configurations would still be left to 1 to keep low > pressure to NN for normal case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076320#comment-16076320 ] Ashish Singhi commented on HBASE-18083: --- There are findbugs warnings reported by QA so it gave -1. May be not related to this patch you can also cross check once. > Make large/small file clean thread number configurable in HFileCleaner > -- > > Key: HBASE-18083 > URL: https://issues.apache.org/jira/browse/HBASE-18083 > Project: HBase > Issue Type: Bug >Reporter: Yu Li >Assignee: Yu Li > Fix For: 3.0.0 > > Attachments: HBASE-18083.patch, HBASE-18083.v2.patch, > HBASE-18083.v3.patch > > > Currently we have only one thread for both large and small file cleaning, but > when write pressure is huge we might need more cleaner threads, so we need to > make the thread number configurable. > We observed more than 1.8PB data in archive directory online due to business > access rate change, and this proposal is one of the necessary changes > required. > Default value of the configurations would still be left to 1 to keep low > pressure to NN for normal case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076325#comment-16076325 ] Chia-Ping Tsai commented on HBASE-18083: These warnings are traced by HBASE-18266. > Make large/small file clean thread number configurable in HFileCleaner > -- > > Key: HBASE-18083 > URL: https://issues.apache.org/jira/browse/HBASE-18083 > Project: HBase > Issue Type: Bug >Reporter: Yu Li >Assignee: Yu Li > Fix For: 3.0.0 > > Attachments: HBASE-18083.patch, HBASE-18083.v2.patch, > HBASE-18083.v3.patch > > > Currently we have only one thread for both large and small file cleaning, but > when write pressure is huge we might need more cleaner threads, so we need to > make the thread number configurable. > We observed more than 1.8PB data in archive directory online due to business > access rate change, and this proposal is one of the necessary changes > required. > Default value of the configurations would still be left to 1 to keep low > pressure to NN for normal case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076326#comment-16076326 ] ramkrishna.s.vasudevan commented on HBASE-18083: OOO for personal reasons. No access to official emails during this period. > Make large/small file clean thread number configurable in HFileCleaner > -- > > Key: HBASE-18083 > URL: https://issues.apache.org/jira/browse/HBASE-18083 > Project: HBase > Issue Type: Bug >Reporter: Yu Li >Assignee: Yu Li > Fix For: 3.0.0 > > Attachments: HBASE-18083.patch, HBASE-18083.v2.patch, > HBASE-18083.v3.patch > > > Currently we have only one thread for both large and small file cleaning, but > when write pressure is huge we might need more cleaner threads, so we need to > make the thread number configurable. > We observed more than 1.8PB data in archive directory online due to business > access rate change, and this proposal is one of the necessary changes > required. > Default value of the configurations would still be left to 1 to keep low > pressure to NN for normal case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076828#comment-16076828 ] Anoop Sam John commented on HBASE-18083: Pls add release notes while commit. > Make large/small file clean thread number configurable in HFileCleaner > -- > > Key: HBASE-18083 > URL: https://issues.apache.org/jira/browse/HBASE-18083 > Project: HBase > Issue Type: Bug >Reporter: Yu Li >Assignee: Yu Li > Fix For: 3.0.0 > > Attachments: HBASE-18083.patch, HBASE-18083.v2.patch, > HBASE-18083.v3.patch > > > Currently we have only one thread for both large and small file cleaning, but > when write pressure is huge we might need more cleaner threads, so we need to > make the thread number configurable. > We observed more than 1.8PB data in archive directory online due to business > access rate change, and this proposal is one of the necessary changes > required. > Default value of the configurations would still be left to 1 to keep low > pressure to NN for normal case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077644#comment-16077644 ] Yu Li commented on HBASE-18083: --- IIRC, HadoopQA will report "overall 1" if no new warnings introduced by patch, no matter whether there exists warnings in current code base, but it seems the rule has changed... [~ashish singhi] [~chia7712] > Make large/small file clean thread number configurable in HFileCleaner > -- > > Key: HBASE-18083 > URL: https://issues.apache.org/jira/browse/HBASE-18083 > Project: HBase > Issue Type: Bug >Reporter: Yu Li >Assignee: Yu Li > Fix For: 3.0.0 > > Attachments: HBASE-18083.patch, HBASE-18083.v2.patch, > HBASE-18083.v3.patch > > > Currently we have only one thread for both large and small file cleaning, but > when write pressure is huge we might need more cleaner threads, so we need to > make the thread number configurable. > We observed more than 1.8PB data in archive directory online due to business > access rate change, and this proposal is one of the necessary changes > required. > Default value of the configurations would still be left to 1 to keep low > pressure to NN for normal case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077671#comment-16077671 ] stack commented on HBASE-18083: --- Yes please [~carp84] > Make large/small file clean thread number configurable in HFileCleaner > -- > > Key: HBASE-18083 > URL: https://issues.apache.org/jira/browse/HBASE-18083 > Project: HBase > Issue Type: Bug >Reporter: Yu Li >Assignee: Yu Li > Fix For: 3.0.0 > > Attachments: HBASE-18083.patch, HBASE-18083.v2.patch, > HBASE-18083.v3.patch > > > Currently we have only one thread for both large and small file cleaning, but > when write pressure is huge we might need more cleaner threads, so we need to > make the thread number configurable. > We observed more than 1.8PB data in archive directory online due to business > access rate change, and this proposal is one of the necessary changes > required. > Default value of the configurations would still be left to 1 to keep low > pressure to NN for normal case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078002#comment-16078002 ] Hudson commented on HBASE-18083: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3331 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3331/]) HBASE-18083 Make large/small file clean thread number configurable in (liyu: rev 4fe73857679ecba89a7edd3c17d9f92e4c0e2164) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java > Make large/small file clean thread number configurable in HFileCleaner > -- > > Key: HBASE-18083 > URL: https://issues.apache.org/jira/browse/HBASE-18083 > Project: HBase > Issue Type: Bug >Reporter: Yu Li >Assignee: Yu Li > Fix For: 2.0.0, 3.0.0 > > Attachments: HBASE-18083.patch, HBASE-18083.v2.patch, > HBASE-18083.v3.patch > > > Currently we have only one thread for both large and small file cleaning, but > when write pressure is huge we might need more cleaner threads, so we need to > make the thread number configurable. > We observed more than 1.8PB data in archive directory online due to business > access rate change, and this proposal is one of the necessary changes > required. > Default value of the configurations would still be left to 1 to keep low > pressure to NN for normal case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16081797#comment-16081797 ] Hudson commented on HBASE-18083: FAILURE: Integrated in Jenkins build HBase-2.0 #151 (See [https://builds.apache.org/job/HBase-2.0/151/]) HBASE-18083 Make large/small file clean thread number configurable in (liyu: rev 89d2adfe92b2e775a0a3761257846e2d4ed7d141) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java > Make large/small file clean thread number configurable in HFileCleaner > -- > > Key: HBASE-18083 > URL: https://issues.apache.org/jira/browse/HBASE-18083 > Project: HBase > Issue Type: Bug >Reporter: Yu Li >Assignee: Yu Li > Fix For: 2.0.0, 3.0.0 > > Attachments: HBASE-18083.patch, HBASE-18083.v2.patch, > HBASE-18083.v3.patch > > > Currently we have only one thread for both large and small file cleaning, but > when write pressure is huge we might need more cleaner threads, so we need to > make the thread number configurable. > We observed more than 1.8PB data in archive directory online due to business > access rate change, and this proposal is one of the necessary changes > required. > Default value of the configurations would still be left to 1 to keep low > pressure to NN for normal case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090264#comment-16090264 ] Hudson commented on HBASE-18083: FAILURE: Integrated in Jenkins build HBASE-14070.HLC #69 (See [https://builds.apache.org/job/HBASE-14070.HLC/69/]) HBASE-18083 Make large/small file clean thread number configurable in (liyu: rev 4fe73857679ecba89a7edd3c17d9f92e4c0e2164) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java > Make large/small file clean thread number configurable in HFileCleaner > -- > > Key: HBASE-18083 > URL: https://issues.apache.org/jira/browse/HBASE-18083 > Project: HBase > Issue Type: Bug >Reporter: Yu Li >Assignee: Yu Li > Fix For: 2.0.0, 3.0.0 > > Attachments: HBASE-18083.patch, HBASE-18083.v2.patch, > HBASE-18083.v3.patch > > > Currently we have only one thread for both large and small file cleaning, but > when write pressure is huge we might need more cleaner threads, so we need to > make the thread number configurable. > We observed more than 1.8PB data in archive directory online due to business > access rate change, and this proposal is one of the necessary changes > required. > Default value of the configurations would still be left to 1 to keep low > pressure to NN for normal case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16720783#comment-16720783 ] Hudson commented on HBASE-18083: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-20559 Backport HBASE-18083 (Make large/small file clean thread (apurtell: rev 2434162594f22df0bba94bc40186bd5628501c8e) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java > Make large/small file clean thread number configurable in HFileCleaner > -- > > Key: HBASE-18083 > URL: https://issues.apache.org/jira/browse/HBASE-18083 > Project: HBase > Issue Type: Improvement >Reporter: Yu Li >Assignee: Yu Li >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-18083.patch, HBASE-18083.v2.patch, > HBASE-18083.v3.patch > > > Currently we have only one thread for both large and small file cleaning, but > when write pressure is huge we might need more cleaner threads, so we need to > make the thread number configurable. > We observed more than 1.8PB data in archive directory online due to business > access rate change, and this proposal is one of the necessary changes > required. > Default value of the configurations would still be left to 1 to keep low > pressure to NN for normal case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[
https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16721173#comment-16721173
]
Hudson commented on HBASE-18083:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Make large/small file clean thread number configurable in HFileCleaner
> --
>
> Key: HBASE-18083
> URL: https://issues.apache.org/jira/browse/HBASE-18083
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-18083.patch, HBASE-18083.v2.patch,
> HBASE-18083.v3.patch
>
>
> Currently we have only one thread for both large and small file cleaning, but
> when write pressure is huge we might need more cleaner threads, so we need to
> make the thread number configurable.
> We observed more than 1.8PB data in archive directory online due to business
> access rate change, and this proposal is one of the necessary changes
> required.
> Default value of the configurations would still be left to 1 to keep low
> pressure to NN for normal case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[
https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16552483#comment-16552483
]
Hudson commented on HBASE-18083:
Results for branch branch-1
[build #389 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/389/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/389//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/389//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/389//console].
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Make large/small file clean thread number configurable in HFileCleaner
> --
>
> Key: HBASE-18083
> URL: https://issues.apache.org/jira/browse/HBASE-18083
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-18083.patch, HBASE-18083.v2.patch,
> HBASE-18083.v3.patch
>
>
> Currently we have only one thread for both large and small file cleaning, but
> when write pressure is huge we might need more cleaner threads, so we need to
> make the thread number configurable.
> We observed more than 1.8PB data in archive directory online due to business
> access rate change, and this proposal is one of the necessary changes
> required.
> Default value of the configurations would still be left to 1 to keep low
> pressure to NN for normal case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[
https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16553351#comment-16553351
]
Hudson commented on HBASE-18083:
Results for branch branch-1.4
[build #394 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/394/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/394//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/394//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/394//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Make large/small file clean thread number configurable in HFileCleaner
> --
>
> Key: HBASE-18083
> URL: https://issues.apache.org/jira/browse/HBASE-18083
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-18083.patch, HBASE-18083.v2.patch,
> HBASE-18083.v3.patch
>
>
> Currently we have only one thread for both large and small file cleaning, but
> when write pressure is huge we might need more cleaner threads, so we need to
> make the thread number configurable.
> We observed more than 1.8PB data in archive directory online due to business
> access rate change, and this proposal is one of the necessary changes
> required.
> Default value of the configurations would still be left to 1 to keep low
> pressure to NN for normal case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
