[GitHub] [hbase] Apache-HBase commented on pull request #3784: HBASE-26392 Update ClassSize.BYTE_BUFFER for JDK17
Apache-HBase commented on pull request #3784: URL: https://github.com/apache/hbase/pull/3784#issuecomment-951569007 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 44s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 58s | master passed | | +1 :green_heart: | compile | 1m 4s | master passed | | +1 :green_heart: | checkstyle | 0m 34s | master passed | | +1 :green_heart: | spotbugs | 0m 59s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 35s | the patch passed | | +1 :green_heart: | compile | 1m 7s | the patch passed | | +1 :green_heart: | javac | 1m 7s | the patch passed | | +1 :green_heart: | checkstyle | 0m 29s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 29m 11s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 1m 11s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 19s | The patch does not generate ASF License warnings. | | | | 62m 11s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3784/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3784 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux a4e10eb55882 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7af5277859 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-common U: hbase-common | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3784/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3784: HBASE-26392 Update ClassSize.BYTE_BUFFER for JDK17
Apache-HBase commented on pull request #3784: URL: https://github.com/apache/hbase/pull/3784#issuecomment-951557696 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 6m 3s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 1s | master passed | | +1 :green_heart: | compile | 0m 27s | master passed | | +1 :green_heart: | shadedjars | 9m 2s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 2s | the patch passed | | +1 :green_heart: | compile | 0m 27s | the patch passed | | +1 :green_heart: | javac | 0m 27s | the patch passed | | +1 :green_heart: | shadedjars | 9m 6s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 38s | hbase-common in the patch passed. | | | | 39m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3784/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3784 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 34e319a4552b 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7af5277859 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3784/4/testReport/ | | Max. process+thread count | 201 (vs. ulimit of 3) | | modules | C: hbase-common U: hbase-common | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3784/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3784: HBASE-26392 Update ClassSize.BYTE_BUFFER for JDK17
Apache-HBase commented on pull request #3784: URL: https://github.com/apache/hbase/pull/3784#issuecomment-951554674 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 50s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 10s | master passed | | +1 :green_heart: | compile | 0m 24s | master passed | | +1 :green_heart: | shadedjars | 8m 19s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 51s | the patch passed | | +1 :green_heart: | compile | 0m 26s | the patch passed | | +1 :green_heart: | javac | 0m 26s | the patch passed | | +1 :green_heart: | shadedjars | 8m 13s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 51s | hbase-common in the patch passed. | | | | 32m 58s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3784/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3784 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e49ad8ce6873 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7af5277859 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3784/4/testReport/ | | Max. process+thread count | 338 (vs. ulimit of 3) | | modules | C: hbase-common U: hbase-common | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3784/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3795: HBASE-26396 Remove duplicate thread creation during migrating rsgroup
Apache-HBase commented on pull request #3795: URL: https://github.com/apache/hbase/pull/3795#issuecomment-951546927 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 6m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 36s | master passed | | +1 :green_heart: | compile | 1m 5s | master passed | | +1 :green_heart: | shadedjars | 9m 6s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 17s | the patch passed | | +1 :green_heart: | compile | 1m 2s | the patch passed | | +1 :green_heart: | javac | 1m 2s | the patch passed | | +1 :green_heart: | shadedjars | 9m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 212m 1s | hbase-server in the patch passed. | | | | 251m 2s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3795/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3795 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 7d6f32036659 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7af5277859 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3795/1/testReport/ | | Max. process+thread count | 2552 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3795/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] YutSean commented on a change in pull request #3784: HBASE-26392 Update ClassSize.BYTE_BUFFER for JDK17
YutSean commented on a change in pull request #3784:
URL: https://github.com/apache/hbase/pull/3784#discussion_r736128832
##
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/util/JVM.java
##
@@ -60,6 +60,9 @@
private static final String JVMVersion = System.getProperty("java.version");
+ private static final Float JREVersion =
+Float.parseFloat(System.getProperty("java.specification.version"));
Review comment:
OK, have added comments and rename the variable. I think it is easier to
read and use. (Version 8 is represented by 1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-26384) Segment already flushed to hfile may still be remained in CompactingMemStore
[
https://issues.apache.org/jira/browse/HBASE-26384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17434091#comment-17434091
]
chenglei commented on HBASE-26384:
--
[~zhangduo], Would you please help me have review at your convenience? It is
just making the branch-2 in line with master and adding UTs for the problem ,
thank you very much.
> Segment already flushed to hfile may still be remained in CompactingMemStore
> -
>
> Key: HBASE-26384
> URL: https://issues.apache.org/jira/browse/HBASE-26384
> Project: HBase
> Issue Type: Bug
> Components: in-memory-compaction
>Affects Versions: 3.0.0-alpha-1, 2.4.8
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Labels: branch-2
>
> When {{CompactingMemStore}} prepares flushing,
> {{CompactingMemStore.snapshot}} invokes following
> {{CompactingMemStore.pushPipelineToSnapshot}} method to get {{Snapshot}},
> following line 570 and line 575 uses {{CompactionPipeline#version}} to track
> whether the Segments in {{CompactionPipeline#pipeline}} has changed since it
> gets {{VersionedSegmentsList}} in line 570 before emptying
> {{CompactionPipeline#pipeline}} in line 575.
> {code:java}
> 565private void pushPipelineToSnapshot() {
> 566int iterationsCnt = 0;
> 567boolean done = false;
> 568while (!done) {
> 569 iterationsCnt++;
> 570 VersionedSegmentsList segments =
> pipeline.getVersionedList();
> 571 pushToSnapshot(segments.getStoreSegments());
> 572 // swap can return false in case the pipeline was updated
> by ongoing compaction
> 573 // and the version increase, the chance of it happenning is
> very low
> 574 // In Swap: don't close segments (they are in snapshot now)
> and don't update the region size
> 575done = pipeline.swap(segments, null, false, false);
> ...
> }
>{code}
> However, when {{CompactingMemStore#inMemoryCompaction}} executes
> {{CompactionPipeline#flattenOneSegment}}, it does not change
> {{CompactionPipeline#version}} , if there is an {{in memeory compaction}}
> which executes {{CompactingMemStore#flattenOneSegment}} between above line
> 570 and line 575, the {{CompactionPipeline#version}} not change, but the
> {{Segment}} in {{CompactionPipeline}} has changed. Because
> {{CompactionPipeline#version}} not change, {{pipeline.swap}} in above line
> 575 could think it is safe to invoke following
> {{CompactionPipeline#swapSuffix}} method to remove {{Segment}} in
> {{CompactionPipeline}} , but the {{Segment}} in {{CompactionPipeline}} has
> changed because of {{CompactingMemStore#flattenOneSegment}} , so the
> {{Segment}} not removed in following line 295 and still remaining in
> {{CompactionPipeline}}.
> {code:java}
> 293 private void swapSuffix(List suffix,
> ImmutableSegment segment,
> 294 boolean closeSegmentsInSuffix) {
> 295 pipeline.removeAll(suffix);
> 296 if(segment != null) pipeline.addLast(segment);
>
> {code}
> However {{CompactingMemStore.snapshot}} think it is successful and continues
> to flush the {{Segment}} got by {{CompactingMemStore.snapshot}} as normal,
> but the {{Segment}} with the same cells still be left in
> {{CompactingMemStore}}. Leaving {{Segment}} which already flushed in
> {{MemStore}} is dangerous: if a Major Compaction before the left {{Segment}}
> flushing, there may be data erroneous.
> My Fix in the PR is as following:
> # Increasing the {{CompactionPipeline#version}} in
> {{CompactingMemStore#flattenOneSegment}} .
>Branch-2 has this problem but master not， because the branch-2 patch for
> HBASE-18375 omitting this.
> # For {{CompactionPipeline#swapSuffix}} , explicitly checking that the
> {{Segment}} in {{suffix}} input parameter is same as the {{Segment}} in
> {{pipeline}} one by one from
>the last element to the first element of {{suffix}} ， I think explicitly
> throwing Exception is better than hiding error and causing subtle problem.
> I made separate PRs for master and branch-2 so the code for master and
> brach-2 could consistent and master could also has UTs for this problem.
> [PR#3777|https://github.com/apache/hbase/pull/3777] is for master and
> [PR#3779|https://github.com/apache/hbase/pull/3779] is for branch-2.The
> difference between them is patch for brach-2 including following code in
> {{CompactionPipeline.replaceAtIndex}} which not included in branch-2 patch
> for HBASE-18375:
> {code:java}
> // the version increment is indeed needed, because the swap uses
> removeAll() method of the
> // linked-list that compares the objects to find what to remove.
[jira] [Comment Edited] (HBASE-26384) Segment already flushed to hfile may still be remained in CompactingMemStore
[
https://issues.apache.org/jira/browse/HBASE-26384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17434091#comment-17434091
]
chenglei edited comment on HBASE-26384 at 10/26/21, 3:31 AM:
-
[~zhangduo], Would you please help me have a review at your convenience? It is
just making the branch-2 in line with master and adding UTs for the problem ,
thank you very much.
was (Author: comnetwork):
[~zhangduo], Would you please help me have review at your convenience? It is
just making the branch-2 in line with master and adding UTs for the problem ,
thank you very much.
> Segment already flushed to hfile may still be remained in CompactingMemStore
> -
>
> Key: HBASE-26384
> URL: https://issues.apache.org/jira/browse/HBASE-26384
> Project: HBase
> Issue Type: Bug
> Components: in-memory-compaction
>Affects Versions: 3.0.0-alpha-1, 2.4.8
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Labels: branch-2
>
> When {{CompactingMemStore}} prepares flushing,
> {{CompactingMemStore.snapshot}} invokes following
> {{CompactingMemStore.pushPipelineToSnapshot}} method to get {{Snapshot}},
> following line 570 and line 575 uses {{CompactionPipeline#version}} to track
> whether the Segments in {{CompactionPipeline#pipeline}} has changed since it
> gets {{VersionedSegmentsList}} in line 570 before emptying
> {{CompactionPipeline#pipeline}} in line 575.
> {code:java}
> 565private void pushPipelineToSnapshot() {
> 566int iterationsCnt = 0;
> 567boolean done = false;
> 568while (!done) {
> 569 iterationsCnt++;
> 570 VersionedSegmentsList segments =
> pipeline.getVersionedList();
> 571 pushToSnapshot(segments.getStoreSegments());
> 572 // swap can return false in case the pipeline was updated
> by ongoing compaction
> 573 // and the version increase, the chance of it happenning is
> very low
> 574 // In Swap: don't close segments (they are in snapshot now)
> and don't update the region size
> 575done = pipeline.swap(segments, null, false, false);
> ...
> }
>{code}
> However, when {{CompactingMemStore#inMemoryCompaction}} executes
> {{CompactionPipeline#flattenOneSegment}}, it does not change
> {{CompactionPipeline#version}} , if there is an {{in memeory compaction}}
> which executes {{CompactingMemStore#flattenOneSegment}} between above line
> 570 and line 575, the {{CompactionPipeline#version}} not change, but the
> {{Segment}} in {{CompactionPipeline}} has changed. Because
> {{CompactionPipeline#version}} not change, {{pipeline.swap}} in above line
> 575 could think it is safe to invoke following
> {{CompactionPipeline#swapSuffix}} method to remove {{Segment}} in
> {{CompactionPipeline}} , but the {{Segment}} in {{CompactionPipeline}} has
> changed because of {{CompactingMemStore#flattenOneSegment}} , so the
> {{Segment}} not removed in following line 295 and still remaining in
> {{CompactionPipeline}}.
> {code:java}
> 293 private void swapSuffix(List suffix,
> ImmutableSegment segment,
> 294 boolean closeSegmentsInSuffix) {
> 295 pipeline.removeAll(suffix);
> 296 if(segment != null) pipeline.addLast(segment);
>
> {code}
> However {{CompactingMemStore.snapshot}} think it is successful and continues
> to flush the {{Segment}} got by {{CompactingMemStore.snapshot}} as normal,
> but the {{Segment}} with the same cells still be left in
> {{CompactingMemStore}}. Leaving {{Segment}} which already flushed in
> {{MemStore}} is dangerous: if a Major Compaction before the left {{Segment}}
> flushing, there may be data erroneous.
> My Fix in the PR is as following:
> # Increasing the {{CompactionPipeline#version}} in
> {{CompactingMemStore#flattenOneSegment}} .
>Branch-2 has this problem but master not， because the branch-2 patch for
> HBASE-18375 omitting this.
> # For {{CompactionPipeline#swapSuffix}} , explicitly checking that the
> {{Segment}} in {{suffix}} input parameter is same as the {{Segment}} in
> {{pipeline}} one by one from
>the last element to the first element of {{suffix}} ， I think explicitly
> throwing Exception is better than hiding error and causing subtle problem.
> I made separate PRs for master and branch-2 so the code for master and
> brach-2 could consistent and master could also has UTs for this problem.
> [PR#3777|https://github.com/apache/hbase/pull/3777] is for master and
> [PR#3779|https://github.com/apache/hbase/pull/3779] is for branch-2.The
> difference between them is patch for brach-2 including following code in
>
[jira] [Commented] (HBASE-26390) Upload src tarball to nightlies for nightly jobs
[
https://issues.apache.org/jira/browse/HBASE-26390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17434089#comment-17434089
]
Hudson commented on HBASE-26390:
Results for branch branch-2.4
[build #222 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/222/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/222/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/222/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/222/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/222/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Upload src tarball to nightlies for nightly jobs
>
>
> Key: HBASE-26390
> URL: https://issues.apache.org/jira/browse/HBASE-26390
> Project: HBase
> Issue Type: Sub-task
> Components: jenkins, scripts
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 1.7.2, 2.4.9
>
>
> The src tarball is around 30-40MB, so it is also a bit space consuming.
> Let’s upload it to nightlies and link it from the Jenkins artifacts.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on a change in pull request #3784: HBASE-26392 Update ClassSize.BYTE_BUFFER for JDK17
Apache9 commented on a change in pull request #3784:
URL: https://github.com/apache/hbase/pull/3784#discussion_r736109066
##
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/util/JVM.java
##
@@ -60,6 +60,9 @@
private static final String JVMVersion = System.getProperty("java.version");
+ private static final Float JREVersion =
+Float.parseFloat(System.getProperty("java.specification.version"));
Review comment:
OK. But maybe we'd better unify them to the same pattern? For example,
we check whether the returned property contains a dot, if so, we just parse the
last character as a int, otherwise parse the whole String. And for the field
name, better name it as JVM_RELEASE_VERSION or JVM_SPEC_VERSION? And also add
some comments to explain the meaning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3795: HBASE-26396 Remove duplicate thread creation during migrating rsgroup
Apache-HBase commented on pull request #3795: URL: https://github.com/apache/hbase/pull/3795#issuecomment-951514268 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 50s | master passed | | +1 :green_heart: | compile | 1m 13s | master passed | | +1 :green_heart: | shadedjars | 8m 22s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 28s | the patch passed | | +1 :green_heart: | compile | 1m 14s | the patch passed | | +1 :green_heart: | javac | 1m 14s | the patch passed | | +1 :green_heart: | shadedjars | 8m 12s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 137m 11s | hbase-server in the patch passed. | | | | 169m 43s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3795/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3795 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e9006d9fd89d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7af5277859 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3795/1/testReport/ | | Max. process+thread count | 3706 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3795/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] YutSean commented on pull request #3776: HBASE-26325 Backport the HBASE-26305 to branch-1. Move NavigableSet add operation to writer thread in BucketCache
YutSean commented on pull request #3776: URL: https://github.com/apache/hbase/pull/3776#issuecomment-951514324 From the results, it seems that the test JVM crashed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] YutSean commented on a change in pull request #3784: HBASE-26392 Update ClassSize.BYTE_BUFFER for JDK17
YutSean commented on a change in pull request #3784:
URL: https://github.com/apache/hbase/pull/3784#discussion_r736106026
##
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/util/JVM.java
##
@@ -60,6 +60,9 @@
private static final String JVMVersion = System.getProperty("java.version");
+ private static final Float JREVersion =
+Float.parseFloat(System.getProperty("java.specification.version"));
Review comment:
Only version > 9, will return an Integer friendly String.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] YutSean commented on a change in pull request #3784: HBASE-26392 Update ClassSize.BYTE_BUFFER for JDK17
YutSean commented on a change in pull request #3784:
URL: https://github.com/apache/hbase/pull/3784#discussion_r736105791
##
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/util/JVM.java
##
@@ -60,6 +60,9 @@
private static final String JVMVersion = System.getProperty("java.version");
+ private static final Float JREVersion =
+Float.parseFloat(System.getProperty("java.specification.version"));
Review comment:
If the java is 7 or 8, the String will be "1.7" or "1.8". In this case,
if we parse it to Integer, it will raise an exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] comnetwork commented on pull request #3794: HBASE-26394 Cache in RSRpcServices.executeProcedures does not take ef…
comnetwork commented on pull request #3794: URL: https://github.com/apache/hbase/pull/3794#issuecomment-951497469 @sunhelly, @Apache9 , thank you very much for review, checkstyle error already fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #3784: HBASE-26392 Update ClassSize.BYTE_BUFFER for JDK17
Apache9 commented on a change in pull request #3784:
URL: https://github.com/apache/hbase/pull/3784#discussion_r736075454
##
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/util/JVM.java
##
@@ -60,6 +60,9 @@
private static final String JVMVersion = System.getProperty("java.version");
+ private static final Float JREVersion =
+Float.parseFloat(System.getProperty("java.specification.version"));
Review comment:
OK, got it. So why use Float here? I think it is just a integer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3795: HBASE-26396 Remove duplicate thread creation during migrating rsgroup
Apache-HBase commented on pull request #3795: URL: https://github.com/apache/hbase/pull/3795#issuecomment-951463808 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 5m 16s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 11s | master passed | | +1 :green_heart: | compile | 3m 9s | master passed | | +1 :green_heart: | checkstyle | 1m 4s | master passed | | +1 :green_heart: | spotbugs | 2m 8s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 49s | the patch passed | | +1 :green_heart: | compile | 3m 6s | the patch passed | | -0 :warning: | javac | 3m 6s | hbase-server generated 1 new + 192 unchanged - 1 fixed = 193 total (was 193) | | +1 :green_heart: | checkstyle | 1m 2s | the patch passed | | +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 19m 16s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 17s | The patch does not generate ASF License warnings. | | | | 53m 43s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3795/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3795 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 10faf01da08e 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7af5277859 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3795/1/artifact/yetus-general-check/output/diff-compile-javac-hbase-server.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3795/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-26304) Reflect out-of-band locality improvements in served requests
[ https://issues.apache.org/jira/browse/HBASE-26304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17433138#comment-17433138 ] Bryan Beaudreault edited comment on HBASE-26304 at 10/26/21, 12:34 AM: --- Unfortunately there isn't an easy way to achieve the 3rd option. I could potentially add something to DFSInputStream, but we'd be stuck waiting for it to get backported to all supported hadoop versions. I also realized that it's very possible even without the locality healer for blocks to move, and we don't really reflect that at all today. I ended up going with option 1 above, as the easiest option and also most off the critical path. I created a LocalityMetricsRefreshChore which periodically refreshes the HDFSBlockDistribution for all stores on the server, if their locality is less than 100%. I also updated the Balancer's RegionLocationFinder so that when it's given a new ClusterMetrics (at start of each balance and periodically), it automatically clears the locality cache for any region whose locality has changed since the last time. It seems like any change in locality is a pretty clear indicator that the cache is stale, so this is a relatively elegant way to ensure the balancer's cache is not an issue as we improve locality. I'm doing some testing, but will push a PR with this new chore probably next week. was (Author: bbeaudreault): Unfortunately there isn't an easy way to achieve the 3rd option. I could potentially add something to DFSInputStream, but we'd be stuck waiting for it to get backported to all supported hadoop versions. I also realized that it's very possible even without the locality healer for blocks to move, and we don't really reflect that at all today. I ended up going with option 1 above, as the easiest option and also most off the critical path. I created a LocalityMetricsRefreshChore which periodically refreshes the HDFSBlockDistribution for all stores on the server. I thought about limiting it to only non-100% locality stores, but decided against it so that we can also reflect locality _regressions_ due to datanodes dying or if someone ran Balancer or Mover. For anyone monitoring or alerting on locality, it's just as important to know if something just totally tanked locality as it is to knowing whether it was improved by something like the healer. I'm doing some testing, but will push a PR with this new chore probably next week. > Reflect out-of-band locality improvements in served requests > > > Key: HBASE-26304 > URL: https://issues.apache.org/jira/browse/HBASE-26304 > Project: HBase > Issue Type: Sub-task >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Major > > Once the LocalityHealer has improved locality of a StoreFile (by moving > blocks onto the correct host), the Reader's DFSInputStream and Region's > localityIndex metric must be refreshed. Without refreshing the > DFSInputStream, the improved locality will not improve latencies. In fact, > the DFSInputStream may try to fetch blocks that have moved, resulting in a > ReplicaNotFoundException. This is automatically retried, but the retry will > increase long tail latencies relative to configured backoff strategy. > See https://issues.apache.org/jira/browse/HDFS-16155 for an improvement in > backoff strategy which can greatly mitigate latency impact of the missing > block retry. > Even with that mitigation, a StoreFile is often made up of many blocks. > Without some sort of intervention, we will continue to hit > ReplicaNotFoundException over time as clients naturally request data from > moved blocks. > In the original LocalityHealer design, I created a new > RefreshHDFSBlockDistribution RPC on the RegionServer. This RPC accepts a list > of region names and, for each region store, re-opens the underlying StoreFile > if the locality has changed. > I will submit a PR with that implementation, but I am also investigating > other avenues. For example, I noticed > https://issues.apache.org/jira/browse/HDFS-15119 which doesn't seem ideal but > maybe can be improved as an automatic lower-level handling of block moves. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26396) Remove duplicate thread creation during migrating rsgroup
Zhuoyue Huang created HBASE-26396: - Summary: Remove duplicate thread creation during migrating rsgroup Key: HBASE-26396 URL: https://issues.apache.org/jira/browse/HBASE-26396 Project: HBase Issue Type: Bug Components: master, rsgroup Affects Versions: 3.0.0-alpha-2 Reporter: Zhuoyue Huang Assignee: Zhuoyue Huang Fix For: 3.0.0-alpha-2 There is a thread that migrate the table rs group info from RSGroupInfo into the table descriptor. The thread is created when RSGroupManager is initialized and is created again when RSGroupStartupWorker is started. I think this is a bug. Since this thread will not exit until all table rsgroups are migrated, there is no reason to need two threads to run together? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] YutSean commented on a change in pull request #3784: HBASE-26392 Update ClassSize.BYTE_BUFFER for JDK17
YutSean commented on a change in pull request #3784:
URL: https://github.com/apache/hbase/pull/3784#discussion_r736047488
##
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/util/JVM.java
##
@@ -60,6 +60,9 @@
private static final String JVMVersion = System.getProperty("java.version");
+ private static final Float JREVersion =
+Float.parseFloat(System.getProperty("java.specification.version"));
Review comment:
The above JVMVersion is a String like "1.8.0_302", which is not easy to
be changed to a numeric variable.
System.getProperty("java.specification.version") will only return String like
"1", "8", "9", "10", "11", which can be easily parsed to a Float.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-26047) [JDK17] Track JDK17 unit test failures
[
https://issues.apache.org/jira/browse/HBASE-26047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17434035#comment-17434035
]
Huaxiang Sun commented on HBASE-26047:
--
[~xytss123], I linked HBASE-26392.
> [JDK17] Track JDK17 unit test failures
> --
>
> Key: HBASE-26047
> URL: https://issues.apache.org/jira/browse/HBASE-26047
> Project: HBase
> Issue Type: Sub-task
>Reporter: Wei-Chiu Chuang
>Priority: Major
>
> As of now, there are still two failed unit tests after exporting JDK internal
> modules and the modifier access hack.
> {noformat}
> [ERROR] Tests run: 7, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.217
> s <<< FAILURE! - in org.apache.hadoop.hbase.io.TestHeapSize
> [ERROR] org.apache.hadoop.hbase.io.TestHeapSize.testSizes Time elapsed:
> 0.041 s <<< FAILURE!
> java.lang.AssertionError: expected:<160> but was:<152>
> at
> org.apache.hadoop.hbase.io.TestHeapSize.testSizes(TestHeapSize.java:335)
> [ERROR] org.apache.hadoop.hbase.io.TestHeapSize.testNativeSizes Time
> elapsed: 0.01 s <<< FAILURE!
> java.lang.AssertionError: expected:<72> but was:<64>
> at
> org.apache.hadoop.hbase.io.TestHeapSize.testNativeSizes(TestHeapSize.java:134)
> [INFO] Running org.apache.hadoop.hbase.io.Tes
> [ERROR] Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.697
> s <<< FAILURE! - in org.apache.hadoop.hbase.ipc.TestBufferChain
> [ERROR] org.apache.hadoop.hbase.ipc.TestBufferChain.testWithSpy Time
> elapsed: 0.537 s <<< ERROR!
> java.lang.NullPointerException: Cannot enter synchronized block because
> "this.closeLock" is null
> at
> org.apache.hadoop.hbase.ipc.TestBufferChain.testWithSpy(TestBufferChain.java:119)
> {noformat}
> It appears that JDK17 makes the heap size estimate different than before. Not
> sure why.
> TestBufferChain.testWithSpy failure might be because of yet another
> unexported module.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] huaxiangsun merged pull request #3789: HBASE-26327 Replicas cohosted on a rack shouldn't keep triggering bal…
huaxiangsun merged pull request #3789: URL: https://github.com/apache/hbase/pull/3789 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on pull request #3788: HBASE-26327 Replicas cohosted on a rack shouldn't keep triggering bal…
huaxiangsun commented on pull request #3788: URL: https://github.com/apache/hbase/pull/3788#issuecomment-951390188 Got it, merged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun merged pull request #3788: HBASE-26327 Replicas cohosted on a rack shouldn't keep triggering bal…
huaxiangsun merged pull request #3788: URL: https://github.com/apache/hbase/pull/3788 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-26373) Release 2.3.7
[ https://issues.apache.org/jira/browse/HBASE-26373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-26373. -- Resolution: Done > Release 2.3.7 > - > > Key: HBASE-26373 > URL: https://issues.apache.org/jira/browse/HBASE-26373 > Project: HBase > Issue Type: Task > Components: community >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26381) Send announce email
[ https://issues.apache.org/jira/browse/HBASE-26381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-26381. -- Resolution: Done > Send announce email > --- > > Key: HBASE-26381 > URL: https://issues.apache.org/jira/browse/HBASE-26381 > Project: HBase > Issue Type: Sub-task >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] clarax commented on pull request #3788: HBASE-26327 Replicas cohosted on a rack shouldn't keep triggering bal…
clarax commented on pull request #3788: URL: https://github.com/apache/hbase/pull/3788#issuecomment-951299695 > @clarax, can you check what is the check failure? Thanks. It is TEST-org.apache.hadoop.hbase.replication.TestMasterReplication.xml.[failed-to-read] This a known issue on the pipe line @Apache9 reported before. we can kick off a new run too. @huaxiangsun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on pull request #3788: HBASE-26327 Replicas cohosted on a rack shouldn't keep triggering bal…
huaxiangsun commented on pull request #3788: URL: https://github.com/apache/hbase/pull/3788#issuecomment-951195400 @clarax, can you check what is the check failure? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on pull request #3729: HBASE-26327 Replicas cohosted on a rack shouldn't keep triggering Bal…
huaxiangsun commented on pull request #3729: URL: https://github.com/apache/hbase/pull/3729#issuecomment-951192158 > If the weight is set to 0, it won't be considered for evaluation. However the 1 out 4 generator will still waste time generating candidates for rack balance. Therefor I have #3732. Your review would be helpful too. @huaxiangsun Thanks @clarax, let me review the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun merged pull request #3729: HBASE-26327 Replicas cohosted on a rack shouldn't keep triggering Bal…
huaxiangsun merged pull request #3729: URL: https://github.com/apache/hbase/pull/3729 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #3770: HBASE-26380 Add 2.3.7 to downloads page
ndimiduk commented on pull request #3770: URL: https://github.com/apache/hbase/pull/3770#issuecomment-951123578 Thanks @Apache9 ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] BukrosSzabolcs commented on a change in pull request #3786: HBASE-26271: Cleanup the broken store files under data directory
BukrosSzabolcs commented on a change in pull request #3786:
URL: https://github.com/apache/hbase/pull/3786#discussion_r735748629
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
##
@@ -348,8 +353,14 @@ private InternalScanner
postCompactScannerOpen(CompactionRequestImpl request, Sc
smallestReadPoint = Math.min(fd.minSeqIdToKeep, smallestReadPoint);
cleanSeqId = true;
}
+ if (writer != null){
+LOG.warn("Writer exists when it should not: " +
getCompactionTargets().stream()
+ .map(n -> n.toString())
+ .collect(Collectors.joining(", ", "{ ", " }")));
+writer = null;
Review comment:
Setting the writer to null without any warning felt wrong. It's there to
make sure if we run into a corner case where the writer was not reset we at
least get a warning and could later investigate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] BukrosSzabolcs commented on a change in pull request #3786: HBASE-26271: Cleanup the broken store files under data directory
BukrosSzabolcs commented on a change in pull request #3786:
URL: https://github.com/apache/hbase/pull/3786#discussion_r735746314
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
##
@@ -1895,6 +1900,22 @@ private void initializeThreads() {
this.storefileRefresher = new
StorefileRefresherChore(storefileRefreshPeriod,
onlyMetaRefresh, this, this);
}
+
+int fileBasedStoreFileCleanerPeriod = conf.getInt(
+ FileBasedStoreFileCleaner.FILEBASED_STOREFILE_CLEANER_PERIOD,
+ FileBasedStoreFileCleaner.DEFAULT_FILEBASED_STOREFILE_CLEANER_PERIOD);
+int fileBasedStoreFileCleanerDelay = conf.getInt(
+ FileBasedStoreFileCleaner.FILEBASED_STOREFILE_CLEANER_DELAY,
+ FileBasedStoreFileCleaner.DEFAULT_FILEBASED_STOREFILE_CLEANER_DELAY);
+double fileBasedStoreFileCleanerDelayJitter = conf.getDouble(
+ FileBasedStoreFileCleaner.FILEBASED_STOREFILE_CLEANER_DELAY_JITTER,
+
FileBasedStoreFileCleaner.DEFAULT_FILEBASED_STOREFILE_CLEANER_DELAY_JITTER);
+double jitterRate = (RandomUtils.nextDouble() - 0.5D) *
fileBasedStoreFileCleanerDelayJitter;
+long jitterValue = Math.round(fileBasedStoreFileCleanerDelay * jitterRate);
Review comment:
I only put it there because StorefileRefresherChore have done the same.
But I could move it if you would like to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3794: HBASE-26394 Cache in RSRpcServices.executeProcedures does not take ef…
Apache-HBase commented on pull request #3794: URL: https://github.com/apache/hbase/pull/3794#issuecomment-951066529 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 53s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 45s | master passed | | +1 :green_heart: | compile | 1m 11s | master passed | | +1 :green_heart: | shadedjars | 9m 23s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 27s | the patch passed | | +1 :green_heart: | compile | 1m 9s | the patch passed | | +1 :green_heart: | javac | 1m 9s | the patch passed | | +1 :green_heart: | shadedjars | 9m 20s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 215m 30s | hbase-server in the patch passed. | | | | 250m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3794/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3794 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 94ac957fed18 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 84a527bdcc | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3794/1/testReport/ | | Max. process+thread count | 2452 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3794/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26390) Upload src tarball to nightlies for nightly jobs
[ https://issues.apache.org/jira/browse/HBASE-26390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26390: -- Fix Version/s: 2.4.9 1.7.2 3.0.0-alpha-2 2.5.0 > Upload src tarball to nightlies for nightly jobs > > > Key: HBASE-26390 > URL: https://issues.apache.org/jira/browse/HBASE-26390 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-2, 1.7.2, 2.4.9 > > > The src tarball is around 30-40MB, so it is also a bit space consuming. > Let’s upload it to nightlies and link it from the Jenkins artifacts. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] BukrosSzabolcs commented on a change in pull request #3786: HBASE-26271: Cleanup the broken store files under data directory
BukrosSzabolcs commented on a change in pull request #3786:
URL: https://github.com/apache/hbase/pull/3786#discussion_r735736298
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
##
@@ -537,4 +547,17 @@ protected InternalScanner createScanner(HStore store,
ScanInfo scanInfo,
return new StoreScanner(store, scanInfo, scanners, smallestReadPoint,
earliestPutTs,
dropDeletesFromRow, dropDeletesToRow);
}
+
+ public List getCompactionTargets(){
+if (writer == null){
Review comment:
You are right, there is a slight gap between setting the writer to null
and replacing the storefile list. Thanks for pointing it out.
I would prefer to put the writer reset call to `HStore` in `doCompaction`,
just after the `replaceStoreFiles` call. It would have the downside that
StoreEngine has to expose the reset method, but I would argue moving it any
lower on the call chain would just involve additional complications ( being
called during `replayCompactionMarker`, handling different StoreEngine
implementations )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (HBASE-26390) Upload src tarball to nightlies for nightly jobs
[ https://issues.apache.org/jira/browse/HBASE-26390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26390. --- Hadoop Flags: Reviewed Resolution: Fixed Pushed to branch-2.4+ and branch-1. Thanks [~haxiaolin] for reviewing. > Upload src tarball to nightlies for nightly jobs > > > Key: HBASE-26390 > URL: https://issues.apache.org/jira/browse/HBASE-26390 > Project: HBase > Issue Type: Sub-task > Components: jenkins, scripts >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-2, 1.7.2, 2.4.9 > > > The src tarball is around 30-40MB, so it is also a bit space consuming. > Let’s upload it to nightlies and link it from the Jenkins artifacts. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-26390) Upload src tarball to nightlies for nightly jobs
[ https://issues.apache.org/jira/browse/HBASE-26390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26390: -- Component/s: scripts jenkins > Upload src tarball to nightlies for nightly jobs > > > Key: HBASE-26390 > URL: https://issues.apache.org/jira/browse/HBASE-26390 > Project: HBase > Issue Type: Sub-task > Components: jenkins, scripts >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-2, 1.7.2, 2.4.9 > > > The src tarball is around 30-40MB, so it is also a bit space consuming. > Let’s upload it to nightlies and link it from the Jenkins artifacts. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 merged pull request #3793: HBASE-26390 Upload src tarball to nightlies for nightly jobs
Apache9 merged pull request #3793: URL: https://github.com/apache/hbase/pull/3793 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] BukrosSzabolcs commented on a change in pull request #3786: HBASE-26271: Cleanup the broken store files under data directory
BukrosSzabolcs commented on a change in pull request #3786:
URL: https://github.com/apache/hbase/pull/3786#discussion_r735718786
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FileBasedStoreFileCleaner.java
##
@@ -0,0 +1,191 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.io.HFileLink;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.ipc.RemoteException;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * This Chore, every time it runs, will clear the unsused HFiles in the data
+ * folder.
+ */
+@InterfaceAudience.Private public class FileBasedStoreFileCleaner extends
ScheduledChore {
Review comment:
Ok, I'm going with BrokenStoreFileCleaner then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] BukrosSzabolcs commented on a change in pull request #3786: HBASE-26271: Cleanup the broken store files under data directory
BukrosSzabolcs commented on a change in pull request #3786: URL: https://github.com/apache/hbase/pull/3786#discussion_r735718289 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTrackerBase.java ## @@ -176,7 +176,7 @@ public final StoreFileWriter createWriter(CreateStoreFileWriterParams params) th * Whether the implementation of this tracker requires you to write to temp directory first, i.e, * does not allow broken store files under the actual data directory. */ - protected abstract boolean requireWritingToTmpDirFirst(); + public abstract boolean requireWritingToTmpDirFirst(); Review comment: Thanks I forgot to clean this up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #3784: HBASE-26392 Update ClassSize.BYTE_BUFFER for JDK17
Apache9 commented on a change in pull request #3784:
URL: https://github.com/apache/hbase/pull/3784#discussion_r735713973
##
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/util/JVM.java
##
@@ -60,6 +60,9 @@
private static final String JVMVersion = System.getProperty("java.version");
+ private static final Float JREVersion =
+Float.parseFloat(System.getProperty("java.specification.version"));
Review comment:
We can not use the above 'java.version' to test whether it is JDK17?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #3794: HBASE-26394 Cache in RSRpcServices.executeProcedures does not take ef…
Apache9 commented on pull request #3794: URL: https://github.com/apache/hbase/pull/3794#issuecomment-951039652 Please fix the checkstyle issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-26395) Add 1.4.14 to download page
Duo Zhang created HBASE-26395: - Summary: Add 1.4.14 to download page Key: HBASE-26395 URL: https://issues.apache.org/jira/browse/HBASE-26395 Project: HBase Issue Type: Sub-task Components: website Reporter: Duo Zhang Assignee: Duo Zhang Fix For: 3.0.0-alpha-2 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HBASE-26395) Add 1.4.14 to download page
[ https://issues.apache.org/jira/browse/HBASE-26395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-26395 started by Duo Zhang. - > Add 1.4.14 to download page > --- > > Key: HBASE-26395 > URL: https://issues.apache.org/jira/browse/HBASE-26395 > Project: HBase > Issue Type: Sub-task > Components: website >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26358) Put up 1.4.14RC0
[ https://issues.apache.org/jira/browse/HBASE-26358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26358. --- Resolution: Fixed Done. > Put up 1.4.14RC0 > > > Key: HBASE-26358 > URL: https://issues.apache.org/jira/browse/HBASE-26358 > Project: HBase > Issue Type: Sub-task > Components: community >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-26358) Put up 1.4.14RC0
[ https://issues.apache.org/jira/browse/HBASE-26358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26358: -- Component/s: community > Put up 1.4.14RC0 > > > Key: HBASE-26358 > URL: https://issues.apache.org/jira/browse/HBASE-26358 > Project: HBase > Issue Type: Sub-task > Components: community >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3794: HBASE-26394 Cache in RSRpcServices.executeProcedures does not take ef…
Apache-HBase commented on pull request #3794: URL: https://github.com/apache/hbase/pull/3794#issuecomment-950991418 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 53s | master passed | | +1 :green_heart: | compile | 1m 14s | master passed | | +1 :green_heart: | shadedjars | 8m 17s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 24s | the patch passed | | +1 :green_heart: | compile | 1m 12s | the patch passed | | +1 :green_heart: | javac | 1m 12s | the patch passed | | +1 :green_heart: | shadedjars | 8m 12s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 137m 19s | hbase-server in the patch passed. | | | | 169m 42s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3794/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3794 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 7414fe52468d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 84a527bdcc | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3794/1/testReport/ | | Max. process+thread count | 3615 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3794/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #3786: HBASE-26271: Cleanup the broken store files under data directory
wchevreuil commented on a change in pull request #3786:
URL: https://github.com/apache/hbase/pull/3786#discussion_r735493337
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
##
@@ -1895,6 +1900,22 @@ private void initializeThreads() {
this.storefileRefresher = new
StorefileRefresherChore(storefileRefreshPeriod,
onlyMetaRefresh, this, this);
}
+
+int fileBasedStoreFileCleanerPeriod = conf.getInt(
+ FileBasedStoreFileCleaner.FILEBASED_STOREFILE_CLEANER_PERIOD,
+ FileBasedStoreFileCleaner.DEFAULT_FILEBASED_STOREFILE_CLEANER_PERIOD);
+int fileBasedStoreFileCleanerDelay = conf.getInt(
+ FileBasedStoreFileCleaner.FILEBASED_STOREFILE_CLEANER_DELAY,
+ FileBasedStoreFileCleaner.DEFAULT_FILEBASED_STOREFILE_CLEANER_DELAY);
+double fileBasedStoreFileCleanerDelayJitter = conf.getDouble(
+ FileBasedStoreFileCleaner.FILEBASED_STOREFILE_CLEANER_DELAY_JITTER,
+
FileBasedStoreFileCleaner.DEFAULT_FILEBASED_STOREFILE_CLEANER_DELAY_JITTER);
+double jitterRate = (RandomUtils.nextDouble() - 0.5D) *
fileBasedStoreFileCleanerDelayJitter;
+long jitterValue = Math.round(fileBasedStoreFileCleanerDelay * jitterRate);
Review comment:
Can we move all these properties initialisation to the Cleaner
constructor itself?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
##
@@ -348,8 +353,14 @@ private InternalScanner
postCompactScannerOpen(CompactionRequestImpl request, Sc
smallestReadPoint = Math.min(fd.minSeqIdToKeep, smallestReadPoint);
cleanSeqId = true;
}
+ if (writer != null){
+LOG.warn("Writer exists when it should not: " +
getCompactionTargets().stream()
+ .map(n -> n.toString())
+ .collect(Collectors.joining(", ", "{ ", " }")));
+writer = null;
Review comment:
Do we need this? We are setting the writer in the next line...
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FileBasedStoreFileCleaner.java
##
@@ -0,0 +1,191 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.io.HFileLink;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.ipc.RemoteException;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * This Chore, every time it runs, will clear the unsused HFiles in the data
+ * folder.
+ */
+@InterfaceAudience.Private public class FileBasedStoreFileCleaner extends
ScheduledChore {
+ private static final Logger LOG =
LoggerFactory.getLogger(FileBasedStoreFileCleaner.class);
+ public static final String FILEBASED_STOREFILE_CLEANER_ENABLED =
+ "hbase.region.filebased.storefilecleaner.enabled";
+ public static final boolean DEFAULT_FILEBASED_STOREFILE_CLEANER_ENABLED =
false;
+ public static final String FILEBASED_STOREFILE_CLEANER_TTL =
+ "hbase.region.filebased.storefilecleaner.ttl";
+ public static final long DEFAULT_FILEBASED_STOREFILE_CLEANER_TTL = 1000 * 60
* 60 * 12; //12h
+ public static final String FILEBASED_STOREFILE_CLEANER_DELAY =
+ "hbase.region.filebased.storefilecleaner.delay";
+ public static final int DEFAULT_FILEBASED_STOREFILE_CLEANER_DELAY = 1000 *
60 * 60 * 2; //2h
+ public static final String FILEBASED_STOREFILE_CLEANER_DELAY_JITTER =
+ "hbase.region.filebased.storefilecleaner.delay.jitter";
+ public static final double DEFAULT_FILEBASED_STOREFILE_CLEANER_DELAY_JITTER
= 0.25D;
+ public static final String FILEBASED_STOREFILE_CLEANER_PERIOD =
+
[GitHub] [hbase] Apache-HBase commented on pull request #3794: HBASE-26394 Cache in RSRpcServices.executeProcedures does not take ef…
Apache-HBase commented on pull request #3794: URL: https://github.com/apache/hbase/pull/3794#issuecomment-950885479 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 56s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 41s | master passed | | +1 :green_heart: | compile | 3m 36s | master passed | | +1 :green_heart: | checkstyle | 1m 12s | master passed | | +1 :green_heart: | spotbugs | 2m 18s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 19s | the patch passed | | +1 :green_heart: | compile | 3m 32s | the patch passed | | +1 :green_heart: | javac | 3m 32s | the patch passed | | -0 :warning: | checkstyle | 1m 9s | hbase-server: The patch generated 1 new + 30 unchanged - 0 fixed = 31 total (was 30) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 23m 37s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 53s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 59m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3794/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3794 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux a94c173d3e39 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 84a527bdcc | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3794/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3794/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-26394) Cache in RSRpcServices.executeProcedures does not take effect
[
https://issues.apache.org/jira/browse/HBASE-26394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenglei reassigned HBASE-26394:
Assignee: chenglei
> Cache in RSRpcServices.executeProcedures does not take effect
> -
>
> Key: HBASE-26394
> URL: https://issues.apache.org/jira/browse/HBASE-26394
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.4.8
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
>
> In {{RSRpcServices.executeProcedures}} , we use tdCache to avoid reading
> from the {{TableDescritor}} every time(usually it will read from the file
> system), just as following line 3923 :
> {code:java}
> 3914 public ExecuteProceduresResponse executeProcedures(RpcController
> controller,
> 3915 ExecuteProceduresRequest request) throws ServiceException {
> 3916 try {
> 3917 checkOpen();
> 3918 throwOnWrongStartCode(request);
> 3919
> regionServer.getRegionServerCoprocessorHost().preExecuteProcedures();
> 3920 if (request.getOpenRegionCount() > 0) {
> 3921// Avoid reading from the TableDescritor every time(usually it
> will read from the file
> 3922// system)
> 3923Map tdCache = new HashMap<>();
> 3924request.getOpenRegionList().forEach(req ->
> executeOpenRegionProcedures(req, tdCache));
> 3925 }
>..
> {code}
> but for {{RSRpcServices.executeOpenRegionProcedures}} in above line 3924,
> there is no place to fill the {{tdCache}}, so the {{tdCache}} does not take
> effect in fact.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-26394) Cache in RSRpcServices.executeProcedures does not take effect
[
https://issues.apache.org/jira/browse/HBASE-26394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenglei updated HBASE-26394:
-
Status: Patch Available (was: Open)
> Cache in RSRpcServices.executeProcedures does not take effect
> -
>
> Key: HBASE-26394
> URL: https://issues.apache.org/jira/browse/HBASE-26394
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.8, 3.0.0-alpha-1
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
>
> In {{RSRpcServices.executeProcedures}} , we use tdCache to avoid reading
> from the {{TableDescritor}} every time(usually it will read from the file
> system), just as following line 3923 :
> {code:java}
> 3914 public ExecuteProceduresResponse executeProcedures(RpcController
> controller,
> 3915 ExecuteProceduresRequest request) throws ServiceException {
> 3916 try {
> 3917 checkOpen();
> 3918 throwOnWrongStartCode(request);
> 3919
> regionServer.getRegionServerCoprocessorHost().preExecuteProcedures();
> 3920 if (request.getOpenRegionCount() > 0) {
> 3921// Avoid reading from the TableDescritor every time(usually it
> will read from the file
> 3922// system)
> 3923Map tdCache = new HashMap<>();
> 3924request.getOpenRegionList().forEach(req ->
> executeOpenRegionProcedures(req, tdCache));
> 3925 }
>..
> {code}
> but for {{RSRpcServices.executeOpenRegionProcedures}} in above line 3924,
> there is no place to fill the {{tdCache}}, so the {{tdCache}} does not take
> effect in fact.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] comnetwork opened a new pull request #3794: HBASE-26394 Cache in RSRpcServices.executeProcedures does not take ef…
comnetwork opened a new pull request #3794: URL: https://github.com/apache/hbase/pull/3794 …fect -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26394) Cache in RSRpcServices.executeProcedures does not take effect
[
https://issues.apache.org/jira/browse/HBASE-26394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenglei updated HBASE-26394:
-
Description:
In {{RSRpcServices.executeProcedures}} , we use tdCache to avoid reading from
the {{TableDescritor}} every time(usually it will read from the file system),
just as following line 3923 :
{code:java}
3914 public ExecuteProceduresResponse executeProcedures(RpcController
controller,
3915 ExecuteProceduresRequest request) throws ServiceException {
3916 try {
3917 checkOpen();
3918 throwOnWrongStartCode(request);
3919 regionServer.getRegionServerCoprocessorHost().preExecuteProcedures();
3920 if (request.getOpenRegionCount() > 0) {
3921// Avoid reading from the TableDescritor every time(usually it will
read from the file
3922// system)
3923Map tdCache = new HashMap<>();
3924request.getOpenRegionList().forEach(req ->
executeOpenRegionProcedures(req, tdCache));
3925 }
..
{code}
but for {{RSRpcServices.executeOpenRegionProcedures}} in above line 3924, there
is no place to fill the {{tdCache}}, so the {{tdCache}} does not take effect in
fact.
was:In
> Cache in RSRpcServices.executeProcedures does not take effect
> -
>
> Key: HBASE-26394
> URL: https://issues.apache.org/jira/browse/HBASE-26394
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.4.8
>Reporter: chenglei
>Priority: Major
>
> In {{RSRpcServices.executeProcedures}} , we use tdCache to avoid reading
> from the {{TableDescritor}} every time(usually it will read from the file
> system), just as following line 3923 :
> {code:java}
> 3914 public ExecuteProceduresResponse executeProcedures(RpcController
> controller,
> 3915 ExecuteProceduresRequest request) throws ServiceException {
> 3916 try {
> 3917 checkOpen();
> 3918 throwOnWrongStartCode(request);
> 3919
> regionServer.getRegionServerCoprocessorHost().preExecuteProcedures();
> 3920 if (request.getOpenRegionCount() > 0) {
> 3921// Avoid reading from the TableDescritor every time(usually it
> will read from the file
> 3922// system)
> 3923Map tdCache = new HashMap<>();
> 3924request.getOpenRegionList().forEach(req ->
> executeOpenRegionProcedures(req, tdCache));
> 3925 }
>..
> {code}
> but for {{RSRpcServices.executeOpenRegionProcedures}} in above line 3924,
> there is no place to fill the {{tdCache}}, so the {{tdCache}} does not take
> effect in fact.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-26394) Cache in RSRpcServices.executeProcedures does not take effect
chenglei created HBASE-26394: Summary: Cache in RSRpcServices.executeProcedures does not take effect Key: HBASE-26394 URL: https://issues.apache.org/jira/browse/HBASE-26394 Project: HBase Issue Type: Bug Affects Versions: 2.4.8, 3.0.0-alpha-1 Reporter: chenglei -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-26394) Cache in RSRpcServices.executeProcedures does not take effect
[ https://issues.apache.org/jira/browse/HBASE-26394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenglei updated HBASE-26394: - Description: In > Cache in RSRpcServices.executeProcedures does not take effect > - > > Key: HBASE-26394 > URL: https://issues.apache.org/jira/browse/HBASE-26394 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-alpha-1, 2.4.8 >Reporter: chenglei >Priority: Major > > In -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3776: HBASE-26325 Backport the HBASE-26305 to branch-1. Move NavigableSet add operation to writer thread in BucketCache
Apache-HBase commented on pull request #3776: URL: https://github.com/apache/hbase/pull/3776#issuecomment-950759493 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 42s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-1 Compile Tests _ | | +1 :green_heart: | mvninstall | 10m 11s | branch-1 passed | | +1 :green_heart: | compile | 0m 48s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | compile | 0m 47s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | checkstyle | 1m 42s | branch-1 passed | | +1 :green_heart: | shadedjars | 3m 14s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 48s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 0m 44s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +0 :ok: | spotbugs | 3m 10s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 7s | branch-1 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 7s | the patch passed | | +1 :green_heart: | compile | 0m 45s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javac | 0m 45s | the patch passed | | +1 :green_heart: | compile | 0m 48s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | javac | 0m 48s | the patch passed | | +1 :green_heart: | checkstyle | 1m 31s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 3m 3s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 5m 5s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | +1 :green_heart: | javadoc | 0m 32s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 0m 42s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | findbugs | 2m 59s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 134m 49s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 42s | The patch does not generate ASF License warnings. | | | | 182m 43s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3776/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3776 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 988bbfa4ff32 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-3776/out/precommit/personality/provided.sh | | git revision | branch-1 / 57dbad11da | | Default Java | Azul Systems, Inc.-1.7.0_272-b10 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:Azul Systems, Inc.-1.8.0_262-b19 /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_272-b10 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3776/3/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3776/3/testReport/ | | Max. process+thread count | 4323 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3776/3/console | | versions | git=2.17.1 maven=3.6.0 findbugs=3.0.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:
