[GitHub] [hbase] comnetwork commented on a change in pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
comnetwork commented on a change in pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#discussion_r737125535
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHStore.java
##
@@ -1921,6 +1922,202 @@ private void
doWriteTestLargeCellAndSmallCellConcurrently(
}
}
+ /**
+ *
+ * This test is for HBASE-26384,
+ * test {@link CompactingMemStore#flattenOneSegment} and {@link
CompactingMemStore#snapshot()}
+ * execute concurrently.
+ * The threads sequence is:
Review comment:
This is the expected behavior,I modified the comments to explictly state
it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Apache-HBase commented on pull request #3777: URL: https://github.com/apache/hbase/pull/3777#issuecomment-952594672 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 54s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 17s | master passed | | +1 :green_heart: | compile | 3m 9s | master passed | | +1 :green_heart: | checkstyle | 1m 6s | master passed | | +1 :green_heart: | spotbugs | 2m 8s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 48s | the patch passed | | +1 :green_heart: | compile | 3m 14s | the patch passed | | +1 :green_heart: | javac | 3m 14s | the patch passed | | +1 :green_heart: | checkstyle | 1m 4s | hbase-server: The patch generated 0 new + 34 unchanged - 1 fixed = 34 total (was 35) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 19m 42s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 21s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 53m 15s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3777 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 7d6f3546d3ad 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9b7ba7f300 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 95 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3777/5/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3787: HBASE-26353 Support loadable dictionaries in hbase-compression-zstd
Apache-HBase commented on pull request #3787: URL: https://github.com/apache/hbase/pull/3787#issuecomment-952580153 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 26s | master passed | | +1 :green_heart: | compile | 2m 1s | master passed | | +1 :green_heart: | shadedjars | 8m 15s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 24s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 29s | the patch passed | | +1 :green_heart: | compile | 2m 0s | the patch passed | | +1 :green_heart: | javac | 2m 0s | the patch passed | | +1 :green_heart: | shadedjars | 8m 13s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 24s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 8s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 140m 50s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 5m 14s | hbase-compression-zstd in the patch passed. | | | | 183m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3787/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3787 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2e3ef5ede53c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9b7ba7f300 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3787/5/testReport/ | | Max. process+thread count | 3911 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server hbase-compression/hbase-compression-zstd U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3787/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] comnetwork commented on a change in pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
comnetwork commented on a change in pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#discussion_r737125535
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHStore.java
##
@@ -1921,6 +1922,202 @@ private void
doWriteTestLargeCellAndSmallCellConcurrently(
}
}
+ /**
+ *
+ * This test is for HBASE-26384,
+ * test {@link CompactingMemStore#flattenOneSegment} and {@link
CompactingMemStore#snapshot()}
+ * execute concurrently.
+ * The threads sequence is:
Review comment:
This is the expected behavior,I would modify the comments to explictly
state it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] comnetwork commented on a change in pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
comnetwork commented on a change in pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#discussion_r737075750
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionPipeline.java
##
@@ -95,7 +103,7 @@ public VersionedSegmentsList getVersionedList() {
public VersionedSegmentsList getVersionedTail() {
synchronized (pipeline){
- List segmentList = new ArrayList<>();
+ LinkedList segmentList = new LinkedList<>();
Review comment:
@Apache9 ， thank you very much for review，Here I change to `LinkedList
`because I expect `VersionedSegmentsList.storeSegments` is `LinkedList`， so in
the following `CompactionPipline.matchAndRemoveSuffixFromPipeline` I could use
`LinkedList.descendingIterator` to simplify code.
I think here change to LinkedList is somwhat ok, because:
1. Here the `List `has only one element, `LinkedList `and `ArrayList `make
no difference.
2. Furthermore, the` CompactionPipeline.pipeline` and
`CompactionPipeline.readOnlyCopy` which used in`
CompactionPipeline.getVersionedList` are all LinkedList, that is to say,
`VersionedSegmentsList.storeSegments` is actually LinkedList in
fact(`CompactionPipline.getVersionedTail` is only used for test), so here
change to LinkedList could also make the code looks consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-26398) CellCounter fails for large tables filling up local disk
[ https://issues.apache.org/jira/browse/HBASE-26398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Istvan Toth updated HBASE-26398: Affects Version/s: 2.5.0 2.2.7 2.3.7 2.4.8 > CellCounter fails for large tables filling up local disk > > > Key: HBASE-26398 > URL: https://issues.apache.org/jira/browse/HBASE-26398 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 2.2.7, 2.5.0, 3.0.0-alpha-2, 2.3.7, 2.4.8 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Minor > > CellCounter dumps all cell coordinates into its output, which can become huge. > The spill can fill the local disk on the reducer. > CellCounter hardcodes *mapreduce.job.reduces* to *1*, so it is not possible > to use multiple reducers to get around this. > Fixing this is easy, by not hardcoding *mapreduce.job.reduces*, it still > defaults to 1, but can be overriden by the user. > CellCounter also generates two extra records with constant keys for each > cell, which have to be processed by the reducer. > Even with multiple reducers, these (1/3 of the totcal records) will go the > same reducer, which can also fill up the disk. > This can be fixed by adding a Combiner to the Mapper, which sums the counter > records, thereby reducing the Mapper output records to 1/3 of their previous > amount, which can be evenly distibuted between the reducers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3787: HBASE-26353 Support loadable dictionaries in hbase-compression-zstd
Apache-HBase commented on pull request #3787: URL: https://github.com/apache/hbase/pull/3787#issuecomment-952529796 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 5s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 50s | master passed | | +1 :green_heart: | compile | 5m 33s | master passed | | +1 :green_heart: | checkstyle | 2m 13s | master passed | | +1 :green_heart: | spotbugs | 4m 12s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 1s | the patch passed | | +1 :green_heart: | compile | 5m 41s | the patch passed | | +1 :green_heart: | javac | 5m 41s | the patch passed | | +1 :green_heart: | checkstyle | 0m 29s | hbase-common: The patch generated 0 new + 0 unchanged - 2 fixed = 0 total (was 2) | | +1 :green_heart: | checkstyle | 1m 21s | The patch passed checkstyle in hbase-server | | +1 :green_heart: | checkstyle | 0m 13s | The patch passed checkstyle in hbase-compression-zstd | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 24m 18s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | -1 :x: | spotbugs | 1m 13s | hbase-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 38s | The patch does not generate ASF License warnings. | | | | 72m 38s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-common | | | Possible doublecheck on org.apache.hadoop.hbase.io.compress.DictionaryCache.CACHE in org.apache.hadoop.hbase.io.compress.DictionaryCache.getDictionary(Configuration, String) At DictionaryCache.java:org.apache.hadoop.hbase.io.compress.DictionaryCache.getDictionary(Configuration, String) At DictionaryCache.java:[lines 65-67] | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3787/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3787 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 7191a43d9fe1 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9b7ba7f300 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3787/5/artifact/yetus-general-check/output/new-spotbugs-hbase-common.html | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server hbase-compression/hbase-compression-zstd U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3787/5/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26398) CellCounter fails for large tables filling up local disk
[ https://issues.apache.org/jira/browse/HBASE-26398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Istvan Toth updated HBASE-26398: Description: CellCounter dumps all cell coordinates into its output, which can become huge. The spill can fill the local disk on the reducer. CellCounter hardcodes *mapreduce.job.reduces* to *1*, so it is not possible to use multiple reducers to get around this. Fixing this is easy, by not hardcoding *mapreduce.job.reduces*, it still defaults to 1, but can be overriden by the user. CellCounter also generates two extra records with constant keys for each cell, which have to be processed by the reducer. Even with multiple reducers, these (1/3 of the totcal records) will go the same reducer, which can also fill up the disk. This can be fixed by adding a Combiner to the Mapper, which sums the counter records, thereby reducing the Mapper output records to 1/3 of their previous amount, which can be evenly distibuted between the reducers. was: CellCounter dumps all cell coordinates into its output, which can become huge. The spill can fill the local disk on the reducer. CellCounter hardcodes *mapreduce.job.reduces* to *1*, so it is not possible to use multiple reducers to get around this. Fixing this is easy, by not hardcoding *mapreduce.job.reduces*, it still defaults to 1, but can be overriden by the user. CellCounter also generates two extra records with constant keys for each cell, which have to be processed by the reducer. Even with multiple reducers, these (1/3 of the totcal records) will got the same reducer, which can also fill up the disk. This can be fixed by adding a Combiner to the Mapper, which sums the counter records, thereby reducing the Mapper output records to 1/3 of their previous amount. > CellCounter fails for large tables filling up local disk > > > Key: HBASE-26398 > URL: https://issues.apache.org/jira/browse/HBASE-26398 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 3.0.0-alpha-2 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Minor > > CellCounter dumps all cell coordinates into its output, which can become huge. > The spill can fill the local disk on the reducer. > CellCounter hardcodes *mapreduce.job.reduces* to *1*, so it is not possible > to use multiple reducers to get around this. > Fixing this is easy, by not hardcoding *mapreduce.job.reduces*, it still > defaults to 1, but can be overriden by the user. > CellCounter also generates two extra records with constant keys for each > cell, which have to be processed by the reducer. > Even with multiple reducers, these (1/3 of the totcal records) will go the > same reducer, which can also fill up the disk. > This can be fixed by adding a Combiner to the Mapper, which sums the counter > records, thereby reducing the Mapper output records to 1/3 of their previous > amount, which can be evenly distibuted between the reducers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] comnetwork commented on a change in pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
comnetwork commented on a change in pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#discussion_r737075750
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionPipeline.java
##
@@ -95,7 +103,7 @@ public VersionedSegmentsList getVersionedList() {
public VersionedSegmentsList getVersionedTail() {
synchronized (pipeline){
- List segmentList = new ArrayList<>();
+ LinkedList segmentList = new LinkedList<>();
Review comment:
@Apache9 ， thank you very much for review，Here I change to `LinkedList
`because I expect `VersionedSegmentsList.storeSegments` is `LinkedList`， so in
the following `CompactionPipline.matchAndRemoveSuffixFromPipeline` I could use
`LinkedList.descendingIterator` to to simplify code.
I think here change to LinkedList is somwhat ok, because:
1. Here the `List `has only one element, `LinkedList `and `ArrayList `make
no difference.
2. Furthermore, the` CompactionPipeline.pipeline` and
`CompactionPipeline.readOnlyCopy` which used in`
CompactionPipeline.getVersionedList` are all LinkedList, that is to say,
`VersionedSegmentsList.storeSegments` is actually LinkedList in
fact(`CompactionPipline.getVersionedTail` is only used for test), so here I
change to LinkedList could also make the code looks consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] sunhelly closed pull request #3797: HBASE-26347 Support detect and exclude slow DNs in fan-out of WAL
sunhelly closed pull request #3797: URL: https://github.com/apache/hbase/pull/3797 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3787: HBASE-26353 Support loadable dictionaries in hbase-compression-zstd
Apache-HBase commented on pull request #3787: URL: https://github.com/apache/hbase/pull/3787#issuecomment-952500962 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 48s | master passed | | +1 :green_heart: | compile | 2m 0s | master passed | | +1 :green_heart: | shadedjars | 9m 52s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 25s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 39s | the patch passed | | +1 :green_heart: | compile | 2m 3s | the patch passed | | +1 :green_heart: | javac | 2m 3s | the patch passed | | +1 :green_heart: | shadedjars | 9m 45s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 21s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 8s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 239m 25s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 6m 25s | hbase-compression-zstd in the patch passed. | | | | 289m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3787/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3787 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3058cc0ee4c3 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9b7ba7f300 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3787/4/testReport/ | | Max. process+thread count | 2874 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server hbase-compression/hbase-compression-zstd U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3787/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-26402) Remove branch-1.4 from compatibility matrix and release manager section in our ref guide
Duo Zhang created HBASE-26402: - Summary: Remove branch-1.4 from compatibility matrix and release manager section in our ref guide Key: HBASE-26402 URL: https://issues.apache.org/jira/browse/HBASE-26402 Project: HBase Issue Type: Sub-task Components: documentation Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HBASE-26401) Remove jenkins files for branch-1.4
[ https://issues.apache.org/jira/browse/HBASE-26401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-26401 started by Duo Zhang. - > Remove jenkins files for branch-1.4 > --- > > Key: HBASE-26401 > URL: https://issues.apache.org/jira/browse/HBASE-26401 > Project: HBase > Issue Type: Sub-task > Components: jenkins, scripts >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > The nightly for branch-1.4 has been broken for a very long time. > Since now we have decided to EOL branch-1.4, let's remove these jenkins files > for branch-1.4. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26401) Remove jenkins files for branch-1.4
Duo Zhang created HBASE-26401: - Summary: Remove jenkins files for branch-1.4 Key: HBASE-26401 URL: https://issues.apache.org/jira/browse/HBASE-26401 Project: HBase Issue Type: Sub-task Components: jenkins, scripts Reporter: Duo Zhang Assignee: Duo Zhang The nightly for branch-1.4 has been broken for a very long time. Since now we have decided to EOL branch-1.4, let's remove these jenkins files for branch-1.4. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26400) EOL 1.4.x
Duo Zhang created HBASE-26400: - Summary: EOL 1.4.x Key: HBASE-26400 URL: https://issues.apache.org/jira/browse/HBASE-26400 Project: HBase Issue Type: Task Reporter: Duo Zhang Assignee: Duo Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26355) Release 1.4.14
[ https://issues.apache.org/jira/browse/HBASE-26355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26355. --- Resolution: Fixed Done. > Release 1.4.14 > -- > > Key: HBASE-26355 > URL: https://issues.apache.org/jira/browse/HBASE-26355 > Project: HBase > Issue Type: Task > Components: community >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Per the end of this thread: > https://lists.apache.org/thread.html/r34d3fb86d667f8b3e58cbba78655733ac76e10f5883650f4910adc5c%40%3Cdev.hbase.apache.org%3E > Let's do a final 1.4.14 release and mark branch-1.4 as EOL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26327) Replicas cohosted on a rack shouldn't keep triggering Balancer
[
https://issues.apache.org/jira/browse/HBASE-26327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434638#comment-17434638
]
Hudson commented on HBASE-26327:
Results for branch branch-2.4
[build #223 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/223/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/223/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/223/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/223/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/223/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Replicas cohosted on a rack shouldn't keep triggering Balancer
> --
>
> Key: HBASE-26327
> URL: https://issues.apache.org/jira/browse/HBASE-26327
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer
>Reporter: Clara Xiong
>Assignee: Clara Xiong
>Priority: Major
>
> Currently, Balancer has a shortcut check for cohosted replicas of the same
> region/host/rack and will keep triggering balancer if it is non-zero.
> With the trending of kube and cloud solution for HBase, operators don't have
> full control of the topology or are not even aware of the topology. There are
> cases that it is not possible to satisfy or requires sacrificing other
> constraints such as region count balancing on RS. We want to keep there per
> RS/host check for availability of regions, especially for meta region. We
> haven't heard problem with rack so far. The cost functions will still be
> considered during balancing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26394) Cache in RSRpcServices.executeProcedures does not take effect
[
https://issues.apache.org/jira/browse/HBASE-26394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434639#comment-17434639
]
Hudson commented on HBASE-26394:
Results for branch branch-2.4
[build #223 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/223/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/223/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/223/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/223/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/223/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Cache in RSRpcServices.executeProcedures does not take effect
> -
>
> Key: HBASE-26394
> URL: https://issues.apache.org/jira/browse/HBASE-26394
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.4.8
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.9
>
>
> In {{RSRpcServices.executeProcedures}} , we use tdCache to avoid reading
> from the {{TableDescritor}} every time(usually it will read from the file
> system), just as following line 3923 :
> {code:java}
> 3914 public ExecuteProceduresResponse executeProcedures(RpcController
> controller,
> 3915 ExecuteProceduresRequest request) throws ServiceException {
> 3916 try {
> 3917 checkOpen();
> 3918 throwOnWrongStartCode(request);
> 3919
> regionServer.getRegionServerCoprocessorHost().preExecuteProcedures();
> 3920 if (request.getOpenRegionCount() > 0) {
> 3921// Avoid reading from the TableDescritor every time(usually it
> will read from the file
> 3922// system)
> 3923Map tdCache = new HashMap<>();
> 3924request.getOpenRegionList().forEach(req ->
> executeOpenRegionProcedures(req, tdCache));
> 3925 }
>..
> {code}
> but for {{RSRpcServices.executeOpenRegionProcedures}} in above line 3924,
> there is no place to fill the {{tdCache}}, so the {{tdCache}} does not take
> effect in fact.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3787: HBASE-26353 Support loadable dictionaries in hbase-compression-zstd
Apache-HBase commented on pull request #3787: URL: https://github.com/apache/hbase/pull/3787#issuecomment-952459569 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 26s | master passed | | +1 :green_heart: | compile | 2m 0s | master passed | | +1 :green_heart: | shadedjars | 8m 11s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 24s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 24s | the patch passed | | +1 :green_heart: | compile | 2m 3s | the patch passed | | +1 :green_heart: | javac | 2m 3s | the patch passed | | +1 :green_heart: | shadedjars | 8m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 24s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 8s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 140m 36s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 5m 39s | hbase-compression-zstd in the patch passed. | | | | 184m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3787/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3787 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 254b1fb64b4a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9b7ba7f300 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3787/4/testReport/ | | Max. process+thread count | 3808 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server hbase-compression/hbase-compression-zstd U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3787/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3787: HBASE-26353 Support loadable dictionaries in hbase-compression-zstd
Apache-HBase commented on pull request #3787: URL: https://github.com/apache/hbase/pull/3787#issuecomment-952411487 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 14s | master passed | | +1 :green_heart: | compile | 5m 40s | master passed | | +1 :green_heart: | checkstyle | 2m 3s | master passed | | +1 :green_heart: | spotbugs | 4m 13s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 58s | the patch passed | | +1 :green_heart: | compile | 5m 33s | the patch passed | | +1 :green_heart: | javac | 5m 33s | the patch passed | | +1 :green_heart: | checkstyle | 0m 29s | hbase-common: The patch generated 0 new + 0 unchanged - 2 fixed = 0 total (was 2) | | +1 :green_heart: | checkstyle | 1m 20s | The patch passed checkstyle in hbase-server | | +1 :green_heart: | checkstyle | 0m 13s | The patch passed checkstyle in hbase-compression-zstd | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 24m 27s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | -1 :x: | spotbugs | 1m 16s | hbase-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 40s | The patch does not generate ASF License warnings. | | | | 72m 13s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-common | | | Possible doublecheck on org.apache.hadoop.hbase.io.compress.DictionaryCache.CACHE in org.apache.hadoop.hbase.io.compress.DictionaryCache.getDictionary(Configuration, String) At DictionaryCache.java:org.apache.hadoop.hbase.io.compress.DictionaryCache.getDictionary(Configuration, String) At DictionaryCache.java:[lines 65-67] | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3787/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3787 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 1816d25a65f6 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9b7ba7f300 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3787/4/artifact/yetus-general-check/output/new-spotbugs-hbase-common.html | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server hbase-compression/hbase-compression-zstd U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3787/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-26399) branch-2 modules dependencies are confused
Mike Drob created HBASE-26399:
-
Summary: branch-2 modules dependencies are confused
Key: HBASE-26399
URL: https://issues.apache.org/jira/browse/HBASE-26399
Project: HBase
Issue Type: Task
Components: build
Affects Versions: 2.5.0
Environment: branch-2
Java version: 11.0.12
Apache Maven 3.8.3
Reporter: Mike Drob
When trying to build branch-2, I get several dependency resolution issues.
Different issues based on what approach I try.
{{mvn -e -B --no-transfer-progress -Dhadoop.profile=3.0
-Dhadoop-three.version=3.0.0-cdh6.3.2 clean install -Dmaven.test.skip=true}}
will fail with:
{noformat}
231.4 [ERROR] Failed to execute goal
org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M3:enforce
(hadoop-profile-min-maven-min-java-banned-xerces) on project
hbase-build-configuration: Execution
hadoop-profile-min-maven-min-java-banned-xerces of goal
org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M3:enforce failed:
org.apache.maven.shared.dependency.graph.DependencyGraphBuilderException: Could
not resolve following dependencies:
[org.apache.hbase:hbase-annotations:jar:tests:2.5.0-SNAPSHOT (test)]: Could not
resolve dependencies for project
org.apache.hbase:hbase-build-configuration:pom:2.5.0-SNAPSHOT: Could not find
artifact org.apache.hbase:hbase-annotations:jar:tests:2.5.0-SNAPSHOT in
snapshots{noformat}
Note that I'm running this behind a corporate firewall, so if there is a
snapshot available upstream in apache snapshots repo, then I'm not able to
access it. I would have thought that it can use the locally built and installed
one instead, since build-configuration does correctly come after annotations.
If I change to {{-DskipTests}} then this target succeeds, but later {{mvn
site}} fails with
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-site-plugin:3.7.1:site (default-site) on project
hbase: Error generating maven-javadoc-plugin:3.2.0:aggregate-no-fork report:
[ERROR] Exit code: 1 -
/home/jenkins-agent/hbase/hbase-testing-util/src/main/java/org/apache/hadoop/hbase/testing/TestingHBaseClusterOption.java:24:
error: cannot find symbol
[ERROR] import org.apache.hadoop.hbase.StartMiniClusterOption;
[ERROR] ^
[ERROR] symbol: class StartMiniClusterOption
[ERROR] location: package org.apache.hadoop.hbase {noformat}
Again, I'm missing testing packages, this time for javadocs.
I believe I was getting similar errors on main branch, although I don't have
logs for that anymore.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] clarax commented on a change in pull request #3724: HBASE-26311 Balancer gets stuck in cohosted replica distribution
clarax commented on a change in pull request #3724:
URL: https://github.com/apache/hbase/pull/3724#discussion_r736888044
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/DoubleArrayCost.java
##
@@ -66,17 +66,21 @@ void applyCostsChange(Consumer consumer) {
}
private static double computeCost(double[] stats) {
+if (stats == null || stats.length == 0) {
+ return 0;
+}
double totalCost = 0;
double total = getSum(stats);
double count = stats.length;
double mean = total / count;
-
for (int i = 0; i < stats.length; i++) {
double n = stats[i];
double diff = (mean - n) * (mean - n);
totalCost += diff;
}
+// No need to compute standard deviation with division by cluster size
when scaling.
+totalCost = Math.sqrt(totalCost);
Review comment:
Total cost = sqrt(sum(square of the deviation)). Since we scale total
cost on min cost and max cost, linearly, I skipped the step to divided all
three of total cost, min cost and max cost by sqrt(count).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] clarax commented on a change in pull request #3724: HBASE-26311 Balancer gets stuck in cohosted replica distribution
clarax commented on a change in pull request #3724:
URL: https://github.com/apache/hbase/pull/3724#discussion_r736885376
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/DoubleArrayCost.java
##
@@ -74,7 +74,7 @@ private static double computeCost(double[] stats) {
for (int i = 0; i < stats.length; i++) {
double n = stats[i];
- double diff = Math.abs(mean - n);
+ double diff = (mean - n) * (mean - n);
Review comment:
Either way is fine. I think multiplication is slightly faster and more
accurate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] clarax commented on a change in pull request #3723: HBASE-26309 Balancer tends to move regions to the server at the end o…
clarax commented on a change in pull request #3723:
URL: https://github.com/apache/hbase/pull/3723#discussion_r736884334
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/LoadCandidateGenerator.java
##
@@ -34,27 +35,51 @@ BalanceAction generate(BalancerClusterState cluster) {
private int pickLeastLoadedServer(final BalancerClusterState cluster, int
thisServer) {
Integer[] servers = cluster.serverIndicesSortedByRegionCount;
-int index = 0;
-while (servers[index] == null || servers[index] == thisServer) {
- index++;
- if (index == servers.length) {
-return -1;
+int selectedIndex = -1;
+double currentLargestRandom = -1;
+for (int i = 0; i < servers.length; i++) {
+ if (servers[i] == null || servers[i] == thisServer) {
+continue;
+ }
+ if (selectedIndex != -1
+&& cluster.getNumRegionsComparator().compare(servers[i],
servers[selectedIndex]) != 0) {
+// Exhausted servers of the same region count
+break;
+ }
+ // we don't know how many servers have the same region count, we will
randomly select one
Review comment:
Oh yes, it used an Integer[] The reason was to hold null value. I am
actually tempted to change it to ArrayList so we don't have to deal
with null value in a fixed size array. The reason not to use the utility class
is not to construct a new array for every iteration. The existing code sorts
the data in place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-26327) Replicas cohosted on a rack shouldn't keep triggering Balancer
[
https://issues.apache.org/jira/browse/HBASE-26327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434512#comment-17434512
]
Hudson commented on HBASE-26327:
Results for branch branch-2
[build #377 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/377/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/377/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/377/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/377/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/377/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Replicas cohosted on a rack shouldn't keep triggering Balancer
> --
>
> Key: HBASE-26327
> URL: https://issues.apache.org/jira/browse/HBASE-26327
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer
>Reporter: Clara Xiong
>Assignee: Clara Xiong
>Priority: Major
>
> Currently, Balancer has a shortcut check for cohosted replicas of the same
> region/host/rack and will keep triggering balancer if it is non-zero.
> With the trending of kube and cloud solution for HBase, operators don't have
> full control of the topology or are not even aware of the topology. There are
> cases that it is not possible to satisfy or requires sacrificing other
> constraints such as region count balancing on RS. We want to keep there per
> RS/host check for availability of regions, especially for meta region. We
> haven't heard problem with rack so far. The cost functions will still be
> considered during balancing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26390) Upload src tarball to nightlies for nightly jobs
[
https://issues.apache.org/jira/browse/HBASE-26390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434511#comment-17434511
]
Hudson commented on HBASE-26390:
Results for branch branch-2
[build #377 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/377/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/377/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/377/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/377/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/377/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Upload src tarball to nightlies for nightly jobs
>
>
> Key: HBASE-26390
> URL: https://issues.apache.org/jira/browse/HBASE-26390
> Project: HBase
> Issue Type: Sub-task
> Components: jenkins, scripts
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 1.7.2, 2.4.9
>
>
> The src tarball is around 30-40MB, so it is also a bit space consuming.
> Let’s upload it to nightlies and link it from the Jenkins artifacts.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26394) Cache in RSRpcServices.executeProcedures does not take effect
[
https://issues.apache.org/jira/browse/HBASE-26394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434513#comment-17434513
]
Hudson commented on HBASE-26394:
Results for branch branch-2
[build #377 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/377/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/377/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/377/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/377/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/377/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Cache in RSRpcServices.executeProcedures does not take effect
> -
>
> Key: HBASE-26394
> URL: https://issues.apache.org/jira/browse/HBASE-26394
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.4.8
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.9
>
>
> In {{RSRpcServices.executeProcedures}} , we use tdCache to avoid reading
> from the {{TableDescritor}} every time(usually it will read from the file
> system), just as following line 3923 :
> {code:java}
> 3914 public ExecuteProceduresResponse executeProcedures(RpcController
> controller,
> 3915 ExecuteProceduresRequest request) throws ServiceException {
> 3916 try {
> 3917 checkOpen();
> 3918 throwOnWrongStartCode(request);
> 3919
> regionServer.getRegionServerCoprocessorHost().preExecuteProcedures();
> 3920 if (request.getOpenRegionCount() > 0) {
> 3921// Avoid reading from the TableDescritor every time(usually it
> will read from the file
> 3922// system)
> 3923Map tdCache = new HashMap<>();
> 3924request.getOpenRegionList().forEach(req ->
> executeOpenRegionProcedures(req, tdCache));
> 3925 }
>..
> {code}
> but for {{RSRpcServices.executeOpenRegionProcedures}} in above line 3924,
> there is no place to fill the {{tdCache}}, so the {{tdCache}} does not take
> effect in fact.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3798: HBASE-26398 CellCounter fails for large tables filling up local disk
Apache-HBase commented on pull request #3798: URL: https://github.com/apache/hbase/pull/3798#issuecomment-952194639 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 6m 48s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 54s | master passed | | +1 :green_heart: | compile | 0m 31s | master passed | | +1 :green_heart: | shadedjars | 9m 41s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 39s | the patch passed | | +1 :green_heart: | compile | 0m 30s | the patch passed | | +1 :green_heart: | javac | 0m 30s | the patch passed | | +1 :green_heart: | shadedjars | 9m 41s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 20s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 17m 1s | hbase-mapreduce in the patch passed. | | | | 55m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3798/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3798 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a2352d83365a 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9b7ba7f300 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3798/1/testReport/ | | Max. process+thread count | 2815 (vs. ulimit of 3) | | modules | C: hbase-mapreduce U: hbase-mapreduce | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3798/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3798: HBASE-26398 CellCounter fails for large tables filling up local disk
Apache-HBase commented on pull request #3798: URL: https://github.com/apache/hbase/pull/3798#issuecomment-952187469 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 39s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 27s | master passed | | +1 :green_heart: | compile | 0m 52s | master passed | | +1 :green_heart: | checkstyle | 0m 21s | master passed | | +1 :green_heart: | spotbugs | 0m 48s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 58s | the patch passed | | +1 :green_heart: | compile | 0m 52s | the patch passed | | +1 :green_heart: | javac | 0m 52s | the patch passed | | +1 :green_heart: | checkstyle | 0m 21s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 20m 22s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 0m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 46m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3798/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3798 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 30e6ec1451f7 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9b7ba7f300 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-mapreduce U: hbase-mapreduce | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3798/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3798: HBASE-26398 CellCounter fails for large tables filling up local disk
Apache-HBase commented on pull request #3798: URL: https://github.com/apache/hbase/pull/3798#issuecomment-952182447 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 45s | master passed | | +1 :green_heart: | compile | 0m 29s | master passed | | +1 :green_heart: | shadedjars | 8m 15s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 27s | the patch passed | | +1 :green_heart: | compile | 0m 30s | the patch passed | | +1 :green_heart: | javac | 0m 30s | the patch passed | | +1 :green_heart: | shadedjars | 8m 14s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 21s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 10m 57s | hbase-mapreduce in the patch passed. | | | | 40m 9s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3798/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3798 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b917dfe9c24a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9b7ba7f300 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3798/1/testReport/ | | Max. process+thread count | 4048 (vs. ulimit of 3) | | modules | C: hbase-mapreduce U: hbase-mapreduce | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3798/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #3724: HBASE-26311 Balancer gets stuck in cohosted replica distribution
huaxiangsun commented on a change in pull request #3724:
URL: https://github.com/apache/hbase/pull/3724#discussion_r736784410
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/DoubleArrayCost.java
##
@@ -106,7 +113,7 @@ public static double getMinSkew(double total, double
numServers) {
min = numHigh * (Math.ceil(mean) - mean) * (Math.ceil(mean) - mean) +
numLow * (mean - Math.floor(mean)) * (mean - Math.floor(mean));
}
-return min;
+return Math.sqrt(min);
Review comment:
Same question as above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #3724: HBASE-26311 Balancer gets stuck in cohosted replica distribution
huaxiangsun commented on a change in pull request #3724:
URL: https://github.com/apache/hbase/pull/3724#discussion_r736784079
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/DoubleArrayCost.java
##
@@ -66,17 +66,21 @@ void applyCostsChange(Consumer consumer) {
}
private static double computeCost(double[] stats) {
+if (stats == null || stats.length == 0) {
+ return 0;
+}
double totalCost = 0;
double total = getSum(stats);
double count = stats.length;
double mean = total / count;
-
for (int i = 0; i < stats.length; i++) {
double n = stats[i];
double diff = (mean - n) * (mean - n);
totalCost += diff;
}
+// No need to compute standard deviation with division by cluster size
when scaling.
+totalCost = Math.sqrt(totalCost);
Review comment:
A bit confused. totalCost is the sum of standard deviation. What is
square root of this sum?
If there is no need to compute standard deviation, should the standard
deviation be avoided at the first place?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #3724: HBASE-26311 Balancer gets stuck in cohosted replica distribution
huaxiangsun commented on a change in pull request #3724:
URL: https://github.com/apache/hbase/pull/3724#discussion_r736775723
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/DoubleArrayCost.java
##
@@ -98,12 +98,13 @@ public static double getMinSkew(double total, double
numServers) {
// It's possible that there aren't enough regions to go around
double min;
if (numServers > total) {
- min = ((numServers - total) * mean + (1 - mean) * total) ;
+ min = ((numServers - total) * mean * mean + (1 - mean) * (1 - mean) *
total) ;
} else {
// Some will have 1 more than everything else.
int numHigh = (int) (total - (Math.floor(mean) * numServers));
int numLow = (int) (numServers - numHigh);
- min = numHigh * (Math.ceil(mean) - mean) + numLow * (mean -
Math.floor(mean));
+ min = numHigh * (Math.ceil(mean) - mean) * (Math.ceil(mean) - mean) +
Review comment:
Same for rest of the square calculation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #3724: HBASE-26311 Balancer gets stuck in cohosted replica distribution
huaxiangsun commented on a change in pull request #3724:
URL: https://github.com/apache/hbase/pull/3724#discussion_r736775124
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/DoubleArrayCost.java
##
@@ -98,12 +98,13 @@ public static double getMinSkew(double total, double
numServers) {
// It's possible that there aren't enough regions to go around
double min;
if (numServers > total) {
- min = ((numServers - total) * mean + (1 - mean) * total) ;
+ min = ((numServers - total) * mean * mean + (1 - mean) * (1 - mean) *
total) ;
Review comment:
Ditto for mean and (1-mean).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #3724: HBASE-26311 Balancer gets stuck in cohosted replica distribution
huaxiangsun commented on a change in pull request #3724:
URL: https://github.com/apache/hbase/pull/3724#discussion_r736774777
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/DoubleArrayCost.java
##
@@ -74,7 +74,7 @@ private static double computeCost(double[] stats) {
for (int i = 0; i < stats.length; i++) {
double n = stats[i];
- double diff = Math.abs(mean - n);
+ double diff = (mean - n) * (mean - n);
Review comment:
Can it be changed to Math.pow(mean -n, 2)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-26398) CellCounter fails for large tables filling up local disk
[ https://issues.apache.org/jira/browse/HBASE-26398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434482#comment-17434482 ] Istvan Toth commented on HBASE-26398: - [~elserj] this is the fix we discussed earlier. > CellCounter fails for large tables filling up local disk > > > Key: HBASE-26398 > URL: https://issues.apache.org/jira/browse/HBASE-26398 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 3.0.0-alpha-2 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Minor > > CellCounter dumps all cell coordinates into its output, which can become huge. > The spill can fill the local disk on the reducer. > CellCounter hardcodes *mapreduce.job.reduces* to *1*, so it is not possible > to use multiple reducers to get around this. > Fixing this is easy, by not hardcoding *mapreduce.job.reduces*, it still > defaults to 1, but can be overriden by the user. > CellCounter also generates two extra records with constant keys for each > cell, which have to be processed by the reducer. > Even with multiple reducers, these (1/3 of the totcal records) will got the > same reducer, which can also fill up the disk. > This can be fixed by adding a Combiner to the Mapper, which sums the counter > records, thereby reducing the Mapper output records to 1/3 of their previous > amount. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26398) CellCounter fails for large tables filling up local disk
Istvan Toth created HBASE-26398: --- Summary: CellCounter fails for large tables filling up local disk Key: HBASE-26398 URL: https://issues.apache.org/jira/browse/HBASE-26398 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 3.0.0-alpha-2 Reporter: Istvan Toth Assignee: Istvan Toth CellCounter dumps all cell coordinates into its output, which can become huge. The spill can fill the local disk on the reducer. CellCounter hardcodes *mapreduce.job.reduces* to *1*, so it is not possible to use multiple reducers to get around this. Fixing this is easy, by not hardcoding *mapreduce.job.reduces*, it still defaults to 1, but can be overriden by the user. CellCounter also generates two extra records with constant keys for each cell, which have to be processed by the reducer. Even with multiple reducers, these (1/3 of the totcal records) will got the same reducer, which can also fill up the disk. This can be fixed by adding a Combiner to the Mapper, which sums the counter records, thereby reducing the Mapper output records to 1/3 of their previous amount. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26398) CellCounter fails for large tables filling up local disk
[ https://issues.apache.org/jira/browse/HBASE-26398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434481#comment-17434481 ] Istvan Toth commented on HBASE-26398: - The linked PR also reduced the runtime from 60-70 seconds to 40-50 seconds on my test cluster, even with a single reducer. > CellCounter fails for large tables filling up local disk > > > Key: HBASE-26398 > URL: https://issues.apache.org/jira/browse/HBASE-26398 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 3.0.0-alpha-2 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Minor > > CellCounter dumps all cell coordinates into its output, which can become huge. > The spill can fill the local disk on the reducer. > CellCounter hardcodes *mapreduce.job.reduces* to *1*, so it is not possible > to use multiple reducers to get around this. > Fixing this is easy, by not hardcoding *mapreduce.job.reduces*, it still > defaults to 1, but can be overriden by the user. > CellCounter also generates two extra records with constant keys for each > cell, which have to be processed by the reducer. > Even with multiple reducers, these (1/3 of the totcal records) will got the > same reducer, which can also fill up the disk. > This can be fixed by adding a Combiner to the Mapper, which sums the counter > records, thereby reducing the Mapper output records to 1/3 of their previous > amount. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-26398) CellCounter fails for large tables filling up local disk
[ https://issues.apache.org/jira/browse/HBASE-26398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Istvan Toth updated HBASE-26398: Priority: Minor (was: Major) > CellCounter fails for large tables filling up local disk > > > Key: HBASE-26398 > URL: https://issues.apache.org/jira/browse/HBASE-26398 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 3.0.0-alpha-2 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Minor > > CellCounter dumps all cell coordinates into its output, which can become huge. > The spill can fill the local disk on the reducer. > CellCounter hardcodes *mapreduce.job.reduces* to *1*, so it is not possible > to use multiple reducers to get around this. > Fixing this is easy, by not hardcoding *mapreduce.job.reduces*, it still > defaults to 1, but can be overriden by the user. > CellCounter also generates two extra records with constant keys for each > cell, which have to be processed by the reducer. > Even with multiple reducers, these (1/3 of the totcal records) will got the > same reducer, which can also fill up the disk. > This can be fixed by adding a Combiner to the Mapper, which sums the counter > records, thereby reducing the Mapper output records to 1/3 of their previous > amount. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3662: HBASE-26253 Backport HBASE-23834 to branch 2.3
Apache-HBase commented on pull request #3662: URL: https://github.com/apache/hbase/pull/3662#issuecomment-952125255 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 6m 35s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2.3 Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 30s | branch-2.3 passed | | +1 :green_heart: | compile | 11m 19s | branch-2.3 passed | | +1 :green_heart: | checkstyle | 2m 42s | branch-2.3 passed | | +1 :green_heart: | spotbugs | 18m 28s | branch-2.3 passed | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 0m 8s | root in branch-2.3 failed. | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 0m 8s | root in the patch failed. | | -1 :x: | compile | 0m 8s | root in the patch failed. | | -0 :warning: | javac | 0m 8s | root in the patch failed. | | -0 :warning: | checkstyle | 0m 6s | The patch fails to run checkstyle in root | | +1 :green_heart: | shellcheck | 0m 0s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 17s | The patch has no ill-formed XML file. | | -1 :x: | hadoopcheck | 0m 6s | The patch causes 13 errors with Hadoop v2.10.0. | | -1 :x: | hadoopcheck | 0m 12s | The patch causes 13 errors with Hadoop v3.1.2. | | -1 :x: | hadoopcheck | 0m 17s | The patch causes 13 errors with Hadoop v3.2.1. | | -1 :x: | spotbugs | 0m 8s | root in the patch failed. | | -1 :x: | spotbugs | 0m 7s | hbase-http in the patch failed. | | -1 :x: | spotbugs | 0m 8s | hbase-it in the patch failed. | | -1 :x: | spotbugs | 0m 7s | hbase-rest in the patch failed. | | -1 :x: | spotbugs | 0m 7s | hbase-server in the patch failed. | | -1 :x: | spotbugs | 0m 7s | hbase-thrift in the patch failed. | ||| _ Other Tests _ | | +0 :ok: | asflicense | 1m 34s | ASF License check generated no output? | | | | 51m 26s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3662 | | Optional Tests | dupname asflicense javac hadoopcheck xml compile spotbugs hbaseanti checkstyle shellcheck shelldocs | | uname | Linux c875682f1cd2 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.3 / 75f47f75ed | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-general-check/output/branch-mvninstall-root.txt | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-general-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-general-check/output/patch-compile-root.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-general-check/output/patch-compile-root.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-general-check/output/buildtool-patch-checkstyle-root.txt | | hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-general-check/output/patch-javac-2.10.0.txt | | hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-general-check/output/patch-javac-3.1.2.txt | | hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-general-check/output/patch-javac-3.2.1.txt | | spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-general-check/output/patch-spotbugs-root.txt | | spotbugs | https://ci-hadoop.apache.org/job/HBase/j
[GitHub] [hbase] Apache-HBase commented on pull request #3662: HBASE-26253 Backport HBASE-23834 to branch 2.3
Apache-HBase commented on pull request #3662: URL: https://github.com/apache/hbase/pull/3662#issuecomment-952102611 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 7s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.3 Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 45s | branch-2.3 passed | | +1 :green_heart: | compile | 3m 0s | branch-2.3 passed | | +1 :green_heart: | shadedjars | 6m 31s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 22s | root in branch-2.3 failed. | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 0m 7s | root in branch-2.3 failed. | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 0m 7s | root in the patch failed. | | -1 :x: | compile | 0m 8s | root in the patch failed. | | -0 :warning: | javac | 0m 8s | root in the patch failed. | | -1 :x: | shadedjars | 0m 5s | patch has 13 errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 7s | root in the patch failed. | | -0 :warning: | javadoc | 0m 8s | hbase-http in the patch failed. | | -0 :warning: | javadoc | 0m 6s | hbase-it in the patch failed. | | -0 :warning: | javadoc | 0m 7s | hbase-resource-bundle in the patch failed. | | -0 :warning: | javadoc | 0m 7s | hbase-rest in the patch failed. | | -0 :warning: | javadoc | 0m 6s | hbase-server in the patch failed. | | -0 :warning: | javadoc | 0m 7s | hbase-shaded in the patch failed. | | -0 :warning: | javadoc | 0m 7s | hbase-shaded-check-invariants in the patch failed. | | -0 :warning: | javadoc | 0m 6s | hbase-shaded-client in the patch failed. | | -0 :warning: | javadoc | 0m 7s | hbase-shaded-mapreduce in the patch failed. | | -0 :warning: | javadoc | 0m 7s | hbase-shaded-testing-util in the patch failed. | | -0 :warning: | javadoc | 0m 6s | hbase-shaded-with-hadoop-check-invariants in the patch failed. | | -0 :warning: | javadoc | 0m 7s | hbase-thrift in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 0m 7s | root in the patch failed. | | | | 24m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3662 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6befa7dbb577 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.3 / 75f47f75ed | | Default Java | AdoptOpenJDK-11.0.10+9 | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-root.txt | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-jdk11-hadoop3-check/output/branch-mvninstall-root.txt | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-jdk11-hadoop3-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-root.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-root.txt | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-root.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-http.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-it.txt | | javadoc | https://ci-hadoop.apache.o
[GitHub] [hbase] Apache-HBase commented on pull request #3662: HBASE-26253 Backport HBASE-23834 to branch 2.3
Apache-HBase commented on pull request #3662: URL: https://github.com/apache/hbase/pull/3662#issuecomment-952100380 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 8s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.3 Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 33s | branch-2.3 passed | | +1 :green_heart: | compile | 2m 11s | branch-2.3 passed | | +1 :green_heart: | shadedjars | 5m 2s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 5m 25s | branch-2.3 passed | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 0m 7s | root in branch-2.3 failed. | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 0m 7s | root in the patch failed. | | -1 :x: | compile | 0m 7s | root in the patch failed. | | -0 :warning: | javac | 0m 7s | root in the patch failed. | | -1 :x: | shadedjars | 0m 5s | patch has 13 errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 7s | root in the patch failed. | | -0 :warning: | javadoc | 0m 6s | hbase-http in the patch failed. | | -0 :warning: | javadoc | 0m 7s | hbase-it in the patch failed. | | -0 :warning: | javadoc | 0m 7s | hbase-resource-bundle in the patch failed. | | -0 :warning: | javadoc | 0m 7s | hbase-rest in the patch failed. | | -0 :warning: | javadoc | 0m 7s | hbase-server in the patch failed. | | -0 :warning: | javadoc | 0m 7s | hbase-shaded in the patch failed. | | -0 :warning: | javadoc | 0m 7s | hbase-shaded-check-invariants in the patch failed. | | -0 :warning: | javadoc | 0m 7s | hbase-shaded-client in the patch failed. | | -0 :warning: | javadoc | 0m 8s | hbase-shaded-mapreduce in the patch failed. | | -0 :warning: | javadoc | 0m 7s | hbase-shaded-testing-util in the patch failed. | | -0 :warning: | javadoc | 0m 8s | hbase-shaded-with-hadoop-check-invariants in the patch failed. | | -0 :warning: | javadoc | 0m 7s | hbase-thrift in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 0m 7s | root in the patch failed. | | | | 22m 0s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3662 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 27e12b94cd64 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.3 / 75f47f75ed | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-jdk8-hadoop2-check/output/branch-mvninstall-root.txt | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-jdk8-hadoop2-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-jdk8-hadoop2-check/output/patch-compile-root.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-jdk8-hadoop2-check/output/patch-compile-root.txt | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-jdk8-hadoop2-check/output/patch-shadedjars.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-jdk8-hadoop2-check/output/patch-javadoc-root.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-jdk8-hadoop2-check/output/patch-javadoc-hbase-http.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-jdk8-hadoop2-check/output/patch-javadoc-hbase-it.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3662/1/artifact/yetus-jdk8-hadoop2-check/output/patch-javadoc-hbase-resource-bundle.txt | | javadoc | https://ci-hadoop.apache
[jira] [Commented] (HBASE-26390) Upload src tarball to nightlies for nightly jobs
[
https://issues.apache.org/jira/browse/HBASE-26390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434428#comment-17434428
]
Hudson commented on HBASE-26390:
Results for branch branch-1
[build #178 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/178/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/178//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/178//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/178//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Upload src tarball to nightlies for nightly jobs
>
>
> Key: HBASE-26390
> URL: https://issues.apache.org/jira/browse/HBASE-26390
> Project: HBase
> Issue Type: Sub-task
> Components: jenkins, scripts
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 1.7.2, 2.4.9
>
>
> The src tarball is around 30-40MB, so it is also a bit space consuming.
> Let’s upload it to nightlies and link it from the Jenkins artifacts.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on a change in pull request #3777: HBASE-26384 Segment already flushed to hfile may still be remained in…
Apache9 commented on a change in pull request #3777:
URL: https://github.com/apache/hbase/pull/3777#discussion_r735716202
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionPipeline.java
##
@@ -95,7 +103,7 @@ public VersionedSegmentsList getVersionedList() {
public VersionedSegmentsList getVersionedTail() {
synchronized (pipeline){
- List segmentList = new ArrayList<>();
+ LinkedList segmentList = new LinkedList<>();
Review comment:
Why change to LinkedList? Usually it will be much slower than ArrayList.
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHStore.java
##
@@ -1921,6 +1922,202 @@ private void
doWriteTestLargeCellAndSmallCellConcurrently(
}
}
+ /**
+ *
+ * This test is for HBASE-26384,
+ * test {@link CompactingMemStore#flattenOneSegment} and {@link
CompactingMemStore#snapshot()}
+ * execute concurrently.
+ * The threads sequence is:
Review comment:
This is the expected behavior or the broken behavior?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (HBASE-26395) Add 1.4.14 to download page
[ https://issues.apache.org/jira/browse/HBASE-26395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26395. --- Resolution: Fixed Pushed an addendum to fix the 1.4.14 src sha512 file link. > Add 1.4.14 to download page > --- > > Key: HBASE-26395 > URL: https://issues.apache.org/jira/browse/HBASE-26395 > Project: HBase > Issue Type: Sub-task > Components: website >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-26395) Add 1.4.14 to download page
[ https://issues.apache.org/jira/browse/HBASE-26395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-26395: --- > Add 1.4.14 to download page > --- > > Key: HBASE-26395 > URL: https://issues.apache.org/jira/browse/HBASE-26395 > Project: HBase > Issue Type: Sub-task > Components: website >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] BukrosSzabolcs commented on a change in pull request #3786: HBASE-26271: Cleanup the broken store files under data directory
BukrosSzabolcs commented on a change in pull request #3786:
URL: https://github.com/apache/hbase/pull/3786#discussion_r736641941
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
##
@@ -1895,6 +1900,22 @@ private void initializeThreads() {
this.storefileRefresher = new
StorefileRefresherChore(storefileRefreshPeriod,
onlyMetaRefresh, this, this);
}
+
+int fileBasedStoreFileCleanerPeriod = conf.getInt(
+ FileBasedStoreFileCleaner.FILEBASED_STOREFILE_CLEANER_PERIOD,
+ FileBasedStoreFileCleaner.DEFAULT_FILEBASED_STOREFILE_CLEANER_PERIOD);
+int fileBasedStoreFileCleanerDelay = conf.getInt(
+ FileBasedStoreFileCleaner.FILEBASED_STOREFILE_CLEANER_DELAY,
+ FileBasedStoreFileCleaner.DEFAULT_FILEBASED_STOREFILE_CLEANER_DELAY);
+double fileBasedStoreFileCleanerDelayJitter = conf.getDouble(
+ FileBasedStoreFileCleaner.FILEBASED_STOREFILE_CLEANER_DELAY_JITTER,
+
FileBasedStoreFileCleaner.DEFAULT_FILEBASED_STOREFILE_CLEANER_DELAY_JITTER);
+double jitterRate = (RandomUtils.nextDouble() - 0.5D) *
fileBasedStoreFileCleanerDelayJitter;
+long jitterValue = Math.round(fileBasedStoreFileCleanerDelay * jitterRate);
Review comment:
After actually trying to move them to constructor I realized why it got
implemented like this:
These values are needed for the super class constructor. So the resulting
constructor would look something like this:
```
public BrokenStoreFileCleaner(final Stoppable stopper, Configuration conf,
HRegionServer regionServer) {
this((int)
(conf.getInt(BrokenStoreFileCleaner.BROKEN_STOREFILE_CLEANER_DELAY,
BrokenStoreFileCleaner.DEFAULT_BROKEN_STOREFILE_CLEANER_DELAY)
+ (RandomUtils.nextDouble() - 0.5D) * conf.getDouble(
BrokenStoreFileCleaner.BROKEN_STOREFILE_CLEANER_DELAY_JITTER,
BrokenStoreFileCleaner.DEFAULT_BROKEN_STOREFILE_CLEANER_DELAY_JITTER)),
conf.getInt(BrokenStoreFileCleaner.BROKEN_STOREFILE_CLEANER_PERIOD,
BrokenStoreFileCleaner.DEFAULT_BROKEN_STOREFILE_CLEANER_PERIOD),
stopper, conf,
regionServer);
}
```
Considering readability I would prefer to leave the config handling where it
is. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3797: HBASE-26347 Support detect and exclude slow DNs in fan-out of WAL
Apache-HBase commented on pull request #3797: URL: https://github.com/apache/hbase/pull/3797#issuecomment-952028296 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 46s | master passed | | +1 :green_heart: | compile | 3m 43s | master passed | | +1 :green_heart: | checkstyle | 1m 15s | master passed | | +1 :green_heart: | spotbugs | 2m 33s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 47s | the patch passed | | +1 :green_heart: | compile | 3m 40s | the patch passed | | -0 :warning: | javac | 0m 29s | hbase-asyncfs generated 7 new + 23 unchanged - 6 fixed = 30 total (was 29) | | -0 :warning: | checkstyle | 0m 14s | hbase-asyncfs: The patch generated 3 new + 1 unchanged - 1 fixed = 4 total (was 2) | | -0 :warning: | checkstyle | 1m 3s | hbase-server: The patch generated 5 new + 10 unchanged - 0 fixed = 15 total (was 10) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 19m 20s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 58s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. | | | | 52m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3797 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 4b0c34906b2f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / a2e690c02e | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/4/artifact/yetus-general-check/output/diff-compile-javac-hbase-asyncfs.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-asyncfs.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 95 (vs. ulimit of 3) | | modules | C: hbase-asyncfs hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3797: HBASE-26347 Support detect and exclude slow DNs in fan-out of WAL
Apache-HBase commented on pull request #3797: URL: https://github.com/apache/hbase/pull/3797#issuecomment-952021782 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 53s | master passed | | +1 :green_heart: | compile | 1m 36s | master passed | | +1 :green_heart: | shadedjars | 8m 9s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 1s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 30s | the patch passed | | +1 :green_heart: | compile | 1m 36s | the patch passed | | +1 :green_heart: | javac | 1m 36s | the patch passed | | +1 :green_heart: | shadedjars | 8m 15s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 1s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 28s | hbase-asyncfs in the patch passed. | | -1 :x: | unit | 9m 25s | hbase-server in the patch failed. | | | | 44m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3797 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5b14a2d0f658 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / a2e690c02e | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/4/testReport/ | | Max. process+thread count | 925 (vs. ulimit of 3) | | modules | C: hbase-asyncfs hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3797: HBASE-26347 Support detect and exclude slow DNs in fan-out of WAL
Apache-HBase commented on pull request #3797: URL: https://github.com/apache/hbase/pull/3797#issuecomment-952018555 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 47s | master passed | | +1 :green_heart: | compile | 1m 22s | master passed | | +1 :green_heart: | shadedjars | 8m 14s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 54s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 52s | the patch passed | | +1 :green_heart: | compile | 1m 25s | the patch passed | | +1 :green_heart: | javac | 1m 25s | the patch passed | | +1 :green_heart: | shadedjars | 8m 16s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 55s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 33s | hbase-asyncfs in the patch passed. | | -1 :x: | unit | 8m 17s | hbase-server in the patch failed. | | | | 41m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3797 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 86dce8dba88d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / a2e690c02e | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/4/testReport/ | | Max. process+thread count | 931 (vs. ulimit of 3) | | modules | C: hbase-asyncfs hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #3784: HBASE-26392 Update ClassSize.BYTE_BUFFER for JDK17
Apache9 commented on a change in pull request #3784:
URL: https://github.com/apache/hbase/pull/3784#discussion_r736575423
##
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/util/JVM.java
##
@@ -60,6 +60,21 @@
private static final String JVMVersion = System.getProperty("java.version");
+ /**
+ * The raw String of java specification version.
+ * "1.8" for java8, "9","10"... for Java 9, 10...
+ */
+ private static final String JVM_SPEC_VERSION_STRING =
+System.getProperty("java.specification.version");
+
+ /**
+ * The Integer represent of JVM_SPEC_VERSION, for the JVM version comparison.
+ * NOTE: Java 8 will be represented by 1.
+ */
+ private static final int JVM_SPEC_VERSION =
JVM_SPEC_VERSION_STRING.contains(".") ?
+(int) Float.parseFloat(System.getProperty("java.specification.version")) :
Review comment:
Why not just use the above JVM_SPEC_VERSION_STRING? And let's use
substring and then parse 1.8 to 8? Or use float and multiply 10 and then minus
10 to get 8? It is a bit strange that we return 1 for 1.8...
##
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/util/JVM.java
##
@@ -106,6 +121,10 @@ public static boolean isGZIPOutputStreamFinishBroken() {
return ibmvendor && JVMVersion.contains("1.6.0");
}
+ public static float getJVMSpecVersion() {
Review comment:
Should return int here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-26327) Replicas cohosted on a rack shouldn't keep triggering Balancer
[
https://issues.apache.org/jira/browse/HBASE-26327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434374#comment-17434374
]
Hudson commented on HBASE-26327:
Results for branch master
[build #423 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/423/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/423/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/423/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/423/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Replicas cohosted on a rack shouldn't keep triggering Balancer
> --
>
> Key: HBASE-26327
> URL: https://issues.apache.org/jira/browse/HBASE-26327
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer
>Reporter: Clara Xiong
>Assignee: Clara Xiong
>Priority: Major
>
> Currently, Balancer has a shortcut check for cohosted replicas of the same
> region/host/rack and will keep triggering balancer if it is non-zero.
> With the trending of kube and cloud solution for HBase, operators don't have
> full control of the topology or are not even aware of the topology. There are
> cases that it is not possible to satisfy or requires sacrificing other
> constraints such as region count balancing on RS. We want to keep there per
> RS/host check for availability of regions, especially for meta region. We
> haven't heard problem with rack so far. The cost functions will still be
> considered during balancing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26390) Upload src tarball to nightlies for nightly jobs
[
https://issues.apache.org/jira/browse/HBASE-26390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434373#comment-17434373
]
Hudson commented on HBASE-26390:
Results for branch master
[build #423 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/423/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/423/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/423/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/423/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Upload src tarball to nightlies for nightly jobs
>
>
> Key: HBASE-26390
> URL: https://issues.apache.org/jira/browse/HBASE-26390
> Project: HBase
> Issue Type: Sub-task
> Components: jenkins, scripts
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 1.7.2, 2.4.9
>
>
> The src tarball is around 30-40MB, so it is also a bit space consuming.
> Let’s upload it to nightlies and link it from the Jenkins artifacts.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3797: HBASE-26347 Support detect and exclude slow DNs in fan-out of WAL
Apache-HBase commented on pull request #3797: URL: https://github.com/apache/hbase/pull/3797#issuecomment-951934216 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 52s | master passed | | +1 :green_heart: | compile | 1m 42s | master passed | | +1 :green_heart: | shadedjars | 8m 40s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 3s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 2m 2s | root in the patch failed. | | -1 :x: | compile | 0m 14s | hbase-asyncfs in the patch failed. | | -1 :x: | compile | 0m 47s | hbase-server in the patch failed. | | -0 :warning: | javac | 0m 14s | hbase-asyncfs in the patch failed. | | -0 :warning: | javac | 0m 47s | hbase-server in the patch failed. | | -1 :x: | shadedjars | 5m 31s | patch has 10 errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 24s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 0m 14s | hbase-asyncfs in the patch failed. | | -1 :x: | unit | 0m 43s | hbase-server in the patch failed. | | | | 29m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3797 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b898cf9fff92 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 773ff28967 | | Default Java | AdoptOpenJDK-11.0.10+9 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-jdk11-hadoop3-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-asyncfs.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-server.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-asyncfs.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-server.txt | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-asyncfs.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/testReport/ | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-asyncfs hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-26395) Add 1.4.14 to download page
[ https://issues.apache.org/jira/browse/HBASE-26395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26395. --- Hadoop Flags: Reviewed Resolution: Fixed Merged to master. Thanks [~haxiaolin] for reviewing. > Add 1.4.14 to download page > --- > > Key: HBASE-26395 > URL: https://issues.apache.org/jira/browse/HBASE-26395 > Project: HBase > Issue Type: Sub-task > Components: website >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-26355) Release 1.4.14
[ https://issues.apache.org/jira/browse/HBASE-26355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26355: -- Component/s: community > Release 1.4.14 > -- > > Key: HBASE-26355 > URL: https://issues.apache.org/jira/browse/HBASE-26355 > Project: HBase > Issue Type: Task > Components: community >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Per the end of this thread: > https://lists.apache.org/thread.html/r34d3fb86d667f8b3e58cbba78655733ac76e10f5883650f4910adc5c%40%3Cdev.hbase.apache.org%3E > Let's do a final 1.4.14 release and mark branch-1.4 as EOL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3797: HBASE-26347 Support detect and exclude slow DNs in fan-out of WAL
Apache-HBase commented on pull request #3797: URL: https://github.com/apache/hbase/pull/3797#issuecomment-951931675 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 48s | master passed | | +1 :green_heart: | compile | 1m 25s | master passed | | +1 :green_heart: | shadedjars | 8m 16s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 55s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 1m 39s | root in the patch failed. | | -1 :x: | compile | 0m 14s | hbase-asyncfs in the patch failed. | | -1 :x: | compile | 0m 37s | hbase-server in the patch failed. | | -0 :warning: | javac | 0m 14s | hbase-asyncfs in the patch failed. | | -0 :warning: | javac | 0m 37s | hbase-server in the patch failed. | | -1 :x: | shadedjars | 5m 29s | patch has 10 errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 37s | hbase-server generated 14 new + 21 unchanged - 0 fixed = 35 total (was 21) | ||| _ Other Tests _ | | -1 :x: | unit | 0m 15s | hbase-asyncfs in the patch failed. | | -1 :x: | unit | 0m 38s | hbase-server in the patch failed. | | | | 26m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3797 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ff3f1346c598 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 773ff28967 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-jdk8-hadoop3-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-asyncfs.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-server.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-asyncfs.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-server.txt | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-server.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-asyncfs.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/testReport/ | | Max. process+thread count | 88 (vs. ulimit of 3) | | modules | C: hbase-asyncfs hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 merged pull request #3796: HBASE-26395 Add 1.4.14 to download page
Apache9 merged pull request #3796: URL: https://github.com/apache/hbase/pull/3796 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3797: HBASE-26347 Support detect and exclude slow DNs in fan-out of WAL
Apache-HBase commented on pull request #3797: URL: https://github.com/apache/hbase/pull/3797#issuecomment-951930449 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 11s | master passed | | +1 :green_heart: | compile | 3m 48s | master passed | | +1 :green_heart: | checkstyle | 1m 15s | master passed | | +1 :green_heart: | spotbugs | 2m 38s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 1m 39s | root in the patch failed. | | -1 :x: | compile | 0m 11s | hbase-asyncfs in the patch failed. | | -1 :x: | compile | 0m 42s | hbase-server in the patch failed. | | -0 :warning: | javac | 0m 11s | hbase-asyncfs in the patch failed. | | -0 :warning: | javac | 0m 42s | hbase-server in the patch failed. | | -0 :warning: | checkstyle | 0m 11s | hbase-asyncfs: The patch generated 5 new + 1 unchanged - 1 fixed = 6 total (was 2) | | -0 :warning: | checkstyle | 1m 3s | hbase-server: The patch generated 4 new + 4 unchanged - 0 fixed = 8 total (was 4) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | -1 :x: | hadoopcheck | 1m 46s | The patch causes 10 errors with Hadoop v3.1.2. | | -1 :x: | hadoopcheck | 3m 33s | The patch causes 10 errors with Hadoop v3.2.1. | | -1 :x: | hadoopcheck | 5m 19s | The patch causes 10 errors with Hadoop v3.3.0. | | -1 :x: | spotbugs | 0m 11s | hbase-asyncfs in the patch failed. | | -1 :x: | spotbugs | 0m 36s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 21s | The patch does not generate ASF License warnings. | | | | 25m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3797 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux d1d3bcfaa20e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 773ff28967 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-general-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-general-check/output/patch-compile-hbase-asyncfs.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-general-check/output/patch-compile-hbase-server.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-general-check/output/patch-compile-hbase-asyncfs.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-general-check/output/patch-compile-hbase-server.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-asyncfs.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-general-check/output/patch-javac-3.1.2.txt | | hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-general-check/output/patch-javac-3.2.1.txt | | hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-general-check/output/patch-javac-3.3.0.txt | | spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/3/artifact/yetus-general-check/output/patch-spotbugs-hbase-as
[GitHub] [hbase] Apache-HBase commented on pull request #3797: HBASE-26347 Support detect and exclude slow DNs in fan-out of WAL
Apache-HBase commented on pull request #3797: URL: https://github.com/apache/hbase/pull/3797#issuecomment-951903008 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 21s | master passed | | +1 :green_heart: | compile | 1m 34s | master passed | | +1 :green_heart: | shadedjars | 8m 16s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 1m 58s | root in the patch failed. | | -1 :x: | compile | 0m 14s | hbase-asyncfs in the patch failed. | | -1 :x: | compile | 0m 44s | hbase-server in the patch failed. | | -0 :warning: | javac | 0m 14s | hbase-asyncfs in the patch failed. | | -0 :warning: | javac | 0m 44s | hbase-server in the patch failed. | | -1 :x: | shadedjars | 5m 22s | patch has 10 errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 24s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 0m 14s | hbase-asyncfs in the patch failed. | | -1 :x: | unit | 0m 43s | hbase-server in the patch failed. | | | | 27m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3797 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 600e3eb3465a 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 773ff28967 | | Default Java | AdoptOpenJDK-11.0.10+9 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-jdk11-hadoop3-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-asyncfs.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-server.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-asyncfs.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-server.txt | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-asyncfs.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/testReport/ | | Max. process+thread count | 102 (vs. ulimit of 3) | | modules | C: hbase-asyncfs hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3797: HBASE-26347 Support detect and exclude slow DNs in fan-out of WAL
Apache-HBase commented on pull request #3797: URL: https://github.com/apache/hbase/pull/3797#issuecomment-951902437 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 49s | master passed | | +1 :green_heart: | compile | 1m 23s | master passed | | +1 :green_heart: | shadedjars | 8m 13s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 55s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 1m 41s | root in the patch failed. | | -1 :x: | compile | 0m 13s | hbase-asyncfs in the patch failed. | | -1 :x: | compile | 0m 39s | hbase-server in the patch failed. | | -0 :warning: | javac | 0m 13s | hbase-asyncfs in the patch failed. | | -0 :warning: | javac | 0m 39s | hbase-server in the patch failed. | | -1 :x: | shadedjars | 5m 29s | patch has 10 errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 37s | hbase-server generated 14 new + 21 unchanged - 0 fixed = 35 total (was 21) | ||| _ Other Tests _ | | -1 :x: | unit | 0m 14s | hbase-asyncfs in the patch failed. | | -1 :x: | unit | 0m 39s | hbase-server in the patch failed. | | | | 26m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3797 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux eba64fe90c29 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 773ff28967 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-jdk8-hadoop3-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-asyncfs.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-server.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-asyncfs.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-server.txt | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-server.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-asyncfs.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/testReport/ | | Max. process+thread count | 88 (vs. ulimit of 3) | | modules | C: hbase-asyncfs hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3797: HBASE-26347 Support detect and exclude slow DNs in fan-out of WAL
Apache-HBase commented on pull request #3797: URL: https://github.com/apache/hbase/pull/3797#issuecomment-95190 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 49s | master passed | | +1 :green_heart: | compile | 3m 38s | master passed | | +1 :green_heart: | checkstyle | 1m 16s | master passed | | +1 :green_heart: | spotbugs | 2m 35s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 1m 38s | root in the patch failed. | | -1 :x: | compile | 0m 12s | hbase-asyncfs in the patch failed. | | -1 :x: | compile | 0m 44s | hbase-server in the patch failed. | | -0 :warning: | javac | 0m 12s | hbase-asyncfs in the patch failed. | | -0 :warning: | javac | 0m 44s | hbase-server in the patch failed. | | -0 :warning: | checkstyle | 0m 11s | hbase-asyncfs: The patch generated 5 new + 1 unchanged - 1 fixed = 6 total (was 2) | | -0 :warning: | checkstyle | 1m 0s | hbase-server: The patch generated 4 new + 4 unchanged - 0 fixed = 8 total (was 4) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | -1 :x: | hadoopcheck | 1m 38s | The patch causes 10 errors with Hadoop v3.1.2. | | -1 :x: | hadoopcheck | 3m 15s | The patch causes 10 errors with Hadoop v3.2.1. | | -1 :x: | hadoopcheck | 4m 52s | The patch causes 10 errors with Hadoop v3.3.0. | | -1 :x: | spotbugs | 0m 11s | hbase-asyncfs in the patch failed. | | -1 :x: | spotbugs | 0m 36s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 21s | The patch does not generate ASF License warnings. | | | | 24m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3797 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 4d14f554290c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 773ff28967 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-general-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-general-check/output/patch-compile-hbase-asyncfs.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-general-check/output/patch-compile-hbase-server.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-general-check/output/patch-compile-hbase-asyncfs.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-general-check/output/patch-compile-hbase-server.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-asyncfs.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-general-check/output/patch-javac-3.1.2.txt | | hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-general-check/output/patch-javac-3.2.1.txt | | hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-general-check/output/patch-javac-3.3.0.txt | | spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/2/artifact/yetus-general-check/output/patch-spotbugs-hbase-as
[GitHub] [hbase] Apache-HBase commented on pull request #3797: HBASE-26347 Support detect and exclude slow DNs in fan-out of WAL
Apache-HBase commented on pull request #3797: URL: https://github.com/apache/hbase/pull/3797#issuecomment-951880376 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 38s | master passed | | +1 :green_heart: | compile | 1m 36s | master passed | | +1 :green_heart: | shadedjars | 8m 19s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 0s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 2m 2s | root in the patch failed. | | -1 :x: | compile | 0m 15s | hbase-asyncfs in the patch failed. | | -1 :x: | compile | 0m 42s | hbase-server in the patch failed. | | -0 :warning: | javac | 0m 14s | hbase-asyncfs in the patch failed. | | -0 :warning: | javac | 0m 42s | hbase-server in the patch failed. | | -1 :x: | shadedjars | 5m 22s | patch has 10 errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 24s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 0m 14s | hbase-asyncfs in the patch failed. | | -1 :x: | unit | 0m 43s | hbase-server in the patch failed. | | | | 27m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3797 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 27425e1ebb47 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 773ff28967 | | Default Java | AdoptOpenJDK-11.0.10+9 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-jdk11-hadoop3-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-asyncfs.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-server.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-asyncfs.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-server.txt | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-asyncfs.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/testReport/ | | Max. process+thread count | 99 (vs. ulimit of 3) | | modules | C: hbase-asyncfs hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3797: HBASE-26347 Support detect and exclude slow DNs in fan-out of WAL
Apache-HBase commented on pull request #3797: URL: https://github.com/apache/hbase/pull/3797#issuecomment-951879609 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 0s | master passed | | +1 :green_heart: | compile | 1m 22s | master passed | | +1 :green_heart: | shadedjars | 8m 21s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 55s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 1m 42s | root in the patch failed. | | -1 :x: | compile | 0m 14s | hbase-asyncfs in the patch failed. | | -1 :x: | compile | 0m 37s | hbase-server in the patch failed. | | -0 :warning: | javac | 0m 14s | hbase-asyncfs in the patch failed. | | -0 :warning: | javac | 0m 37s | hbase-server in the patch failed. | | -1 :x: | shadedjars | 5m 27s | patch has 10 errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 38s | hbase-server generated 14 new + 21 unchanged - 0 fixed = 35 total (was 21) | ||| _ Other Tests _ | | -1 :x: | unit | 0m 14s | hbase-asyncfs in the patch failed. | | -1 :x: | unit | 0m 38s | hbase-server in the patch failed. | | | | 26m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3797 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 454d7cc976a7 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 773ff28967 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-jdk8-hadoop3-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-asyncfs.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-server.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-asyncfs.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-server.txt | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-server.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-asyncfs.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/testReport/ | | Max. process+thread count | 88 (vs. ulimit of 3) | | modules | C: hbase-asyncfs hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3797: HBASE-26347 Support detect and exclude slow DNs in fan-out of WAL
Apache-HBase commented on pull request #3797: URL: https://github.com/apache/hbase/pull/3797#issuecomment-95183 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 4s | master passed | | +1 :green_heart: | compile | 3m 44s | master passed | | +1 :green_heart: | checkstyle | 1m 17s | master passed | | +1 :green_heart: | spotbugs | 2m 36s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 1m 38s | root in the patch failed. | | -1 :x: | compile | 0m 11s | hbase-asyncfs in the patch failed. | | -1 :x: | compile | 0m 43s | hbase-server in the patch failed. | | -0 :warning: | javac | 0m 11s | hbase-asyncfs in the patch failed. | | -0 :warning: | javac | 0m 43s | hbase-server in the patch failed. | | -0 :warning: | checkstyle | 0m 11s | hbase-asyncfs: The patch generated 5 new + 1 unchanged - 1 fixed = 6 total (was 2) | | -0 :warning: | checkstyle | 0m 59s | hbase-server: The patch generated 4 new + 4 unchanged - 0 fixed = 8 total (was 4) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | -1 :x: | hadoopcheck | 1m 39s | The patch causes 10 errors with Hadoop v3.1.2. | | -1 :x: | hadoopcheck | 3m 19s | The patch causes 10 errors with Hadoop v3.2.1. | | -1 :x: | hadoopcheck | 5m 2s | The patch causes 10 errors with Hadoop v3.3.0. | | -1 :x: | spotbugs | 0m 11s | hbase-asyncfs in the patch failed. | | -1 :x: | spotbugs | 0m 36s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 22s | The patch does not generate ASF License warnings. | | | | 24m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3797 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 113862e7e32b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 773ff28967 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-general-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-general-check/output/patch-compile-hbase-asyncfs.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-general-check/output/patch-compile-hbase-server.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-general-check/output/patch-compile-hbase-asyncfs.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-general-check/output/patch-compile-hbase-server.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-asyncfs.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-general-check/output/patch-javac-3.1.2.txt | | hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-general-check/output/patch-javac-3.2.1.txt | | hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-general-check/output/patch-javac-3.3.0.txt | | spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3797/1/artifact/yetus-general-check/output/patch-spotbugs-hbase-as
[jira] [Created] (HBASE-26397) Display the excluded datanodes on regionserver UI
Xiaolin Ha created HBASE-26397: -- Summary: Display the excluded datanodes on regionserver UI Key: HBASE-26397 URL: https://issues.apache.org/jira/browse/HBASE-26397 Project: HBase Issue Type: Sub-task Reporter: Xiaolin Ha Assignee: Xiaolin Ha -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3796: HBASE-26395 Add 1.4.14 to download page
Apache-HBase commented on pull request #3796: URL: https://github.com/apache/hbase/pull/3796#issuecomment-951853732 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 8s | master passed | | +1 :green_heart: | mvnsite | 9m 13s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 47s | the patch passed | | +1 :green_heart: | mvnsite | 9m 11s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 20s | The patch does not generate ASF License warnings. | | | | 29m 26s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3796/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3796 | | Optional Tests | dupname asflicense mvnsite xml | | uname | Linux 82288c153794 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 773ff28967 | | Max. process+thread count | 89 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3796/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3796: HBASE-26395 Add 1.4.14 to download page
Apache-HBase commented on pull request #3796: URL: https://github.com/apache/hbase/pull/3796#issuecomment-951837469 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 5s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | ||| _ Other Tests _ | | | | 3m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3796/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3796 | | Optional Tests | | | uname | Linux d066904ba9e5 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 773ff28967 | | Max. process+thread count | 50 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3796/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3796: HBASE-26395 Add 1.4.14 to download page
Apache-HBase commented on pull request #3796: URL: https://github.com/apache/hbase/pull/3796#issuecomment-951836559 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 46s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | ||| _ Other Tests _ | | | | 3m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3796/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3796 | | Optional Tests | | | uname | Linux 1f9cb63525cd 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 773ff28967 | | Max. process+thread count | 46 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3796/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] BukrosSzabolcs commented on a change in pull request #3786: HBASE-26271: Cleanup the broken store files under data directory
BukrosSzabolcs commented on a change in pull request #3786:
URL: https://github.com/apache/hbase/pull/3786#discussion_r736407220
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
##
@@ -348,8 +353,14 @@ private InternalScanner
postCompactScannerOpen(CompactionRequestImpl request, Sc
smallestReadPoint = Math.min(fd.minSeqIdToKeep, smallestReadPoint);
cleanSeqId = true;
}
+ if (writer != null){
+LOG.warn("Writer exists when it should not: " +
getCompactionTargets().stream()
+ .map(n -> n.toString())
+ .collect(Collectors.joining(", ", "{ ", " }")));
+writer = null;
Review comment:
Oh, you are absolutely right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] BukrosSzabolcs commented on a change in pull request #3786: HBASE-26271: Cleanup the broken store files under data directory
BukrosSzabolcs commented on a change in pull request #3786:
URL: https://github.com/apache/hbase/pull/3786#discussion_r736406688
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
##
@@ -537,4 +547,17 @@ protected InternalScanner createScanner(HStore store,
ScanInfo scanInfo,
return new StoreScanner(store, scanInfo, scanners, smallestReadPoint,
earliestPutTs,
dropDeletesFromRow, dropDeletesToRow);
}
+
+ public List getCompactionTargets(){
+if (writer == null){
Review comment:
> Or do you plan to move the nulling of write to a resetWriter method?
I'm planing to remove the `writer = null ` lines from `commitWriter`
methods, it was not very pretty to begin with and create a `resetWriter` in
`Compactor` we could expose through `StoreEngine` and call in
`HStore.doCompaction`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] BukrosSzabolcs commented on a change in pull request #3786: HBASE-26271: Cleanup the broken store files under data directory
BukrosSzabolcs commented on a change in pull request #3786:
URL: https://github.com/apache/hbase/pull/3786#discussion_r736402871
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FileBasedStoreFileCleaner.java
##
@@ -0,0 +1,191 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.io.HFileLink;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.ipc.RemoteException;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * This Chore, every time it runs, will clear the unsused HFiles in the data
+ * folder.
+ */
+@InterfaceAudience.Private public class FileBasedStoreFileCleaner extends
ScheduledChore {
+ private static final Logger LOG =
LoggerFactory.getLogger(FileBasedStoreFileCleaner.class);
+ public static final String FILEBASED_STOREFILE_CLEANER_ENABLED =
+ "hbase.region.filebased.storefilecleaner.enabled";
+ public static final boolean DEFAULT_FILEBASED_STOREFILE_CLEANER_ENABLED =
false;
+ public static final String FILEBASED_STOREFILE_CLEANER_TTL =
+ "hbase.region.filebased.storefilecleaner.ttl";
+ public static final long DEFAULT_FILEBASED_STOREFILE_CLEANER_TTL = 1000 * 60
* 60 * 12; //12h
+ public static final String FILEBASED_STOREFILE_CLEANER_DELAY =
+ "hbase.region.filebased.storefilecleaner.delay";
+ public static final int DEFAULT_FILEBASED_STOREFILE_CLEANER_DELAY = 1000 *
60 * 60 * 2; //2h
+ public static final String FILEBASED_STOREFILE_CLEANER_DELAY_JITTER =
+ "hbase.region.filebased.storefilecleaner.delay.jitter";
+ public static final double DEFAULT_FILEBASED_STOREFILE_CLEANER_DELAY_JITTER
= 0.25D;
+ public static final String FILEBASED_STOREFILE_CLEANER_PERIOD =
+ "hbase.region.filebased.storefilecleaner.period";
+ public static final int DEFAULT_FILEBASED_STOREFILE_CLEANER_PERIOD = 1000 *
60 * 60 * 6; //6h
+
+ private HRegionServer regionServer;
+ private final AtomicBoolean enabled = new AtomicBoolean(true);
+ private long ttl;
+
+ public FileBasedStoreFileCleaner(final int delay, final int period, final
Stoppable stopper, Configuration conf,
+ HRegionServer regionServer) {
+super("FileBasedStoreFileCleaner", stopper, period, delay);
+this.regionServer = regionServer;
+setEnabled(conf.getBoolean(FILEBASED_STOREFILE_CLEANER_ENABLED,
DEFAULT_FILEBASED_STOREFILE_CLEANER_ENABLED));
+ttl = conf.getLong(FILEBASED_STOREFILE_CLEANER_TTL,
DEFAULT_FILEBASED_STOREFILE_CLEANER_TTL);
+ }
+
+ public boolean setEnabled(final boolean enabled) {
+return this.enabled.getAndSet(enabled);
+ }
+
+ public boolean getEnabled() {
+return this.enabled.get();
+ }
+
+ @InterfaceAudience.Private
+ @Override public void chore() {
+if (getEnabled()) {
+ long start = EnvironmentEdgeManager.currentTime();
+ AtomicLong deletedFiles = new AtomicLong(0);
+ AtomicLong failedDeletes = new AtomicLong(0);
+ for (HRegion region : regionServer.getRegions()) {
+for (HStore store : region.getStores()) {
+ //only clean do cleanup in store using file based storefile tracking
+ if (store.getStoreEngine().requireWritingToTmpDirFirst()) {
+continue;
+ }
+ Path storePath =
+ new Path(region.getRegionFileSystem().getRegionDir(),
store.getColumnFamilyName());
+
+ try {
+List fsStoreFiles =
Arrays.asList(region.getRegionFileSystem().fs.listStatus(storePath));
+fsStoreFiles.forEach(file -> cleanFileIfNeeded(file, store,
deletedFiles, failedDeletes));
+ } catch (IOException e) {
+LOG.warn("Failed to list files i
[GitHub] [hbase] wchevreuil commented on a change in pull request #3786: HBASE-26271: Cleanup the broken store files under data directory
wchevreuil commented on a change in pull request #3786:
URL: https://github.com/apache/hbase/pull/3786#discussion_r736329025
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
##
@@ -537,4 +547,17 @@ protected InternalScanner createScanner(HStore store,
ScanInfo scanInfo,
return new StoreScanner(store, scanInfo, scanners, smallestReadPoint,
earliestPutTs,
dropDeletesFromRow, dropDeletesToRow);
}
+
+ public List getCompactionTargets(){
+if (writer == null){
Review comment:
> I would prefer to put the writer reset call to HStore in doCompaction,
just after the replaceStoreFiles call.
So move `commitWriter` call out of `Compactor.compact`? Or do you plan to
move the nulling of write to a `resetWriter` method?
> It would have the downside that StoreEngine has to expose the reset
method, but I would argue moving it any lower on the call chain would just
involve additional complications ( being called during replayCompactionMarker,
handling different StoreEngine implementations )
Yeah, looks like your suggestion is less complex, and you have an idea of
the impacts, so I just trust your judgement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #3786: HBASE-26271: Cleanup the broken store files under data directory
wchevreuil commented on a change in pull request #3786:
URL: https://github.com/apache/hbase/pull/3786#discussion_r736316274
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
##
@@ -1895,6 +1900,22 @@ private void initializeThreads() {
this.storefileRefresher = new
StorefileRefresherChore(storefileRefreshPeriod,
onlyMetaRefresh, this, this);
}
+
+int fileBasedStoreFileCleanerPeriod = conf.getInt(
+ FileBasedStoreFileCleaner.FILEBASED_STOREFILE_CLEANER_PERIOD,
+ FileBasedStoreFileCleaner.DEFAULT_FILEBASED_STOREFILE_CLEANER_PERIOD);
+int fileBasedStoreFileCleanerDelay = conf.getInt(
+ FileBasedStoreFileCleaner.FILEBASED_STOREFILE_CLEANER_DELAY,
+ FileBasedStoreFileCleaner.DEFAULT_FILEBASED_STOREFILE_CLEANER_DELAY);
+double fileBasedStoreFileCleanerDelayJitter = conf.getDouble(
+ FileBasedStoreFileCleaner.FILEBASED_STOREFILE_CLEANER_DELAY_JITTER,
+
FileBasedStoreFileCleaner.DEFAULT_FILEBASED_STOREFILE_CLEANER_DELAY_JITTER);
+double jitterRate = (RandomUtils.nextDouble() - 0.5D) *
fileBasedStoreFileCleanerDelayJitter;
+long jitterValue = Math.round(fileBasedStoreFileCleanerDelay * jitterRate);
Review comment:
Yeah, since we already pass the Configuration and these are all specific
for this Cleaner itself, I think it's better to move it out of HRegionServer
scope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #3786: HBASE-26271: Cleanup the broken store files under data directory
wchevreuil commented on a change in pull request #3786:
URL: https://github.com/apache/hbase/pull/3786#discussion_r736315277
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FileBasedStoreFileCleaner.java
##
@@ -0,0 +1,191 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.io.HFileLink;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.ipc.RemoteException;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * This Chore, every time it runs, will clear the unsused HFiles in the data
+ * folder.
+ */
+@InterfaceAudience.Private public class FileBasedStoreFileCleaner extends
ScheduledChore {
Review comment:
Just another name suggestion: "LeftoversStoreFileCleaner"... Up to you,
@BukrosSzabolcs !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #3786: HBASE-26271: Cleanup the broken store files under data directory
wchevreuil commented on a change in pull request #3786:
URL: https://github.com/apache/hbase/pull/3786#discussion_r736312453
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
##
@@ -348,8 +353,14 @@ private InternalScanner
postCompactScannerOpen(CompactionRequestImpl request, Sc
smallestReadPoint = Math.min(fd.minSeqIdToKeep, smallestReadPoint);
cleanSeqId = true;
}
+ if (writer != null){
+LOG.warn("Writer exists when it should not: " +
getCompactionTargets().stream()
+ .map(n -> n.toString())
+ .collect(Collectors.joining(", ", "{ ", " }")));
+writer = null;
Review comment:
I agree with the WARN, what I meant is that we don't need to explicitly
set it to null in line #360, when we just go and set it again on line #362.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (HBASE-26355) Release 1.4.14
[ https://issues.apache.org/jira/browse/HBASE-26355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-26355: - Assignee: Duo Zhang > Release 1.4.14 > -- > > Key: HBASE-26355 > URL: https://issues.apache.org/jira/browse/HBASE-26355 > Project: HBase > Issue Type: Task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Per the end of this thread: > https://lists.apache.org/thread.html/r34d3fb86d667f8b3e58cbba78655733ac76e10f5883650f4910adc5c%40%3Cdev.hbase.apache.org%3E > Let's do a final 1.4.14 release and mark branch-1.4 as EOL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-26394) Cache in RSRpcServices.executeProcedures does not take effect
[
https://issues.apache.org/jira/browse/HBASE-26394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-26394:
--
Fix Version/s: 2.4.9
3.0.0-alpha-2
2.5.0
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to branch-2.4+.
Thanks [~comnetwork] and [~haxiaolin]!
> Cache in RSRpcServices.executeProcedures does not take effect
> -
>
> Key: HBASE-26394
> URL: https://issues.apache.org/jira/browse/HBASE-26394
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.4.8
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.9
>
>
> In {{RSRpcServices.executeProcedures}} , we use tdCache to avoid reading
> from the {{TableDescritor}} every time(usually it will read from the file
> system), just as following line 3923 :
> {code:java}
> 3914 public ExecuteProceduresResponse executeProcedures(RpcController
> controller,
> 3915 ExecuteProceduresRequest request) throws ServiceException {
> 3916 try {
> 3917 checkOpen();
> 3918 throwOnWrongStartCode(request);
> 3919
> regionServer.getRegionServerCoprocessorHost().preExecuteProcedures();
> 3920 if (request.getOpenRegionCount() > 0) {
> 3921// Avoid reading from the TableDescritor every time(usually it
> will read from the file
> 3922// system)
> 3923Map tdCache = new HashMap<>();
> 3924request.getOpenRegionList().forEach(req ->
> executeOpenRegionProcedures(req, tdCache));
> 3925 }
>..
> {code}
> but for {{RSRpcServices.executeOpenRegionProcedures}} in above line 3924,
> there is no place to fill the {{tdCache}}, so the {{tdCache}} does not take
> effect in fact.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26354) [hbase-connectors] Added python client for HBase thrift service
[ https://issues.apache.org/jira/browse/HBASE-26354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434177#comment-17434177 ] Yutong Xiao commented on HBASE-26354: - [~busbey] Hello, I used to pushed a new commit to fix the license and packing problem. Not sure if it is OK now. Could you please have a look? > [hbase-connectors] Added python client for HBase thrift service > --- > > Key: HBASE-26354 > URL: https://issues.apache.org/jira/browse/HBASE-26354 > Project: HBase > Issue Type: Improvement >Reporter: Yutong Xiao >Assignee: Yutong Xiao >Priority: Major > > A python client for HBase thrift service. Has request retry mechanism and > exception handling. Also encapsulated redundant parameters to make the usage > easier. -- This message was sent by Atlassian Jira (v8.3.4#803005)
