[GitHub] [hbase] virajjasani commented on issue #1458: HBASE-24140 : Move CandidateGenerator and their implementors out of S…
virajjasani commented on issue #1458: HBASE-24140 : Move CandidateGenerator and their implementors out of S… URL: https://github.com/apache/hbase/pull/1458#issuecomment-611344363 @busbey Please review when you get time. Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1458: HBASE-24140 : Move CandidateGenerator and their implementors out of S…
Apache-HBase commented on issue #1458: HBASE-24140 : Move CandidateGenerator and their implementors out of S… URL: https://github.com/apache/hbase/pull/1458#issuecomment-611343256 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 45s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 54s | master passed | | +1 :green_heart: | checkstyle | 1m 19s | master passed | | +1 :green_heart: | spotbugs | 2m 29s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 49s | the patch passed | | -0 :warning: | checkstyle | 1m 27s | hbase-server: The patch generated 1 new + 28 unchanged - 5 fixed = 29 total (was 33) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 13m 42s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2. | | +1 :green_heart: | spotbugs | 2m 23s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 44m 58s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1458 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux d56bef5f3ec0 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2907260204 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 83 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (HBASE-24138) Ensure StochasticLoadBalancer can log details of decision to not run balancer
[
https://issues.apache.org/jira/browse/HBASE-24138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey resolved HBASE-24138.
-
Fix Version/s: 2.2.5
2.4.0
1.7.0
2.3.0
3.0.0
Resolution: Fixed
> Ensure StochasticLoadBalancer can log details of decision to not run balancer
> -
>
> Key: HBASE-24138
> URL: https://issues.apache.org/jira/browse/HBASE-24138
> Project: HBase
> Issue Type: Task
> Components: Balancer, Operability
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 1.7.0, 2.4.0, 2.2.5
>
>
> Ran into a customer case where the StochasticLoadBalancer was consistently
> deciding not to balance when bringing new region servers on line. Even
> setting the class to TRACE logging would only log a summary statement like:
> {code}
> 2020-04-03 00:29:55,133 TRACE
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Skipping load
> balancing because balanced cluster; total cost is 25.24853189705185, sum
> multiplier is 602.0 min cost which need balance is 0.05
> {code}
> Without any details about what went into that decision it's really hard to
> figure out what we need to tune to get the behavior we want.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Gkkkk302 opened a new pull request #1466: HBASE-24151 The master server aborted for IllegalThreadStateException
G302 opened a new pull request #1466: HBASE-24151 The master server aborted for IllegalThreadStateException URL: https://github.com/apache/hbase/pull/1466 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;…
Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;… URL: https://github.com/apache/hbase/pull/1461#issuecomment-611340588 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 46s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 38s | branch-2 passed | | +1 :green_heart: | compile | 3m 7s | branch-2 passed | | +1 :green_heart: | shadedjars | 4m 16s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 49s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 9s | the patch passed | | +1 :green_heart: | compile | 2m 57s | the patch passed | | +1 :green_heart: | javac | 2m 56s | the patch passed | | +1 :green_heart: | shadedjars | 4m 17s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 43s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 73m 9s | root in the patch failed. | | | | 107m 32s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/3/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1461 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 153ad2e0026d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 611c62f52a | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/3/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/3/testReport/ | | Max. process+thread count | 5159 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Gkkkk302 closed pull request #1035: HBASE-23669 ZNodeClearer#clear() throws IllegalArgumentException
G302 closed pull request #1035: HBASE-23669 ZNodeClearer#clear() throws IllegalArgumentException URL: https://github.com/apache/hbase/pull/1035 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Gkkkk302 opened a new pull request #1465: HBASE-24151 The master server aborted for IllegalThreadStateException
G302 opened a new pull request #1465: HBASE-24151 The master server aborted for IllegalThreadStateException URL: https://github.com/apache/hbase/pull/1465 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Reidddddd commented on issue #1439: HBASE-24121 [Authorization] ServiceAuthorizationManager isn't dynamically updatable. And it should be
Reidd commented on issue #1439: HBASE-24121 [Authorization] ServiceAuthorizationManager isn't dynamically updatable. And it should be URL: https://github.com/apache/hbase/pull/1439#issuecomment-611338142 Yes, related. Just pushed a new commit. Waiting QA. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-24120) Flakey Test: TestReplicationAdminWithClusters timeout
[
https://issues.apache.org/jira/browse/HBASE-24120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078934#comment-17078934
]
Hudson commented on HBASE-24120:
Results for branch branch-2
[build #2591 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2591/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2591/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2591/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2591/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2591/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Flakey Test: TestReplicationAdminWithClusters timeout
> --
>
> Key: HBASE-24120
> URL: https://issues.apache.org/jira/browse/HBASE-24120
> Project: HBase
> Issue Type: Test
> Components: Replication
>Affects Versions: 2.3.0, master, 2.4.0
>Reporter: Huaxiang Sun
>Assignee: Hua Xiang
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> {code:java}
> 2020-04-05 23:36:53,092 ERROR
> [RS_REFRESH_PEER-regionserver/asf905:0-0.replicationSource,2.replicationSource.shipperasf905.gq1.ygridcore.net%2C42849%2C1586129728118,2]
> regionserver.ReplicationSource(397): Unexpected exception in
> RS_REFRESH_PEER-regionserver/asf905:0-0.replicationSource,2.replicationSource.shipperasf905.gq1.ygridcore.net%2C42849%2C1586129728118,2
>
> currentPath=hdfs://localhost:34203/user/jenkins/test-data/03854f9d-2780-eeaa-9645-c341240b62bf/WALs/asf905.gq1.ygridcore.net,42849,1586129728118/asf905.gq1.ygridcore.net%2C42849%2C1586129728118.1586129730509
> java.lang.RuntimeException: Thread is interrupted, the replication source may
> be terminated
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.interruptOrAbortWhenFail(ReplicationSourceManager.java:477)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:519)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.updateLogPosition(ReplicationSourceShipper.java:264)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.shipEdits(ReplicationSourceShipper.java:160)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.run(ReplicationSourceShipper.java:118)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24099) Use a fair ReentrantReadWriteLock for the region close lock
[
https://issues.apache.org/jira/browse/HBASE-24099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078933#comment-17078933
]
Hudson commented on HBASE-24099:
Results for branch branch-2
[build #2591 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2591/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2591/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2591/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2591/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2591/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Use a fair ReentrantReadWriteLock for the region close lock
> ---
>
> Key: HBASE-24099
> URL: https://issues.apache.org/jira/browse/HBASE-24099
> Project: HBase
> Issue Type: Improvement
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 1.3.7, 1.7.0, 2.4.0, 2.1.10, 1.4.14, 2.2.5
>
> Attachments: ltt_results.pdf, pe_results.pdf, ycsb_results.pdf
>
>
> Consider creating the region's ReentrantReadWriteLock with the fair locking
> policy. We have had a couple of production incidents where a regionserver
> stalled in shutdown for a very very long time, leading to RIT (FAILED_CLOSE).
> The latest example is a 43 minute shutdown, ~40 minutes (2465280 ms) of that
> time was spent waiting to acquire the write lock on the region in order to
> finish closing it.
> {quote}
> ...
> Finished memstore flush of ~66.92 MB/70167112, currentsize=0 B/0 for region
> . in 927ms, sequenceid=6091133815, compaction requested=false at
> 1585175635349 (+60 ms)
> Disabling writes for close at 1585178100629 (+2465280 ms)
> {quote}
> This time was spent in between the memstore flush and the task status change
> "Disabling writes for close at...". This is at HRegion.java:1481 in 1.3.6:
> {code}
> 1480: // block waiting for the lock for closing
> 1481: lock.writeLock().lock(); // FindBugs: Complains
> UL_UNRELEASED_LOCK_EXCEPTION_PATH but seems fine
> {code}
>
> The close lock is operating in unfair mode. The table in question is under
> constant high query load. When the close request was received, there were
> active readers. After the close request there were more active readers,
> near-continuous contention. Although the clients would receive
> RegionServerStoppingException and other error notifications, because the
> region could not be reassigned, they kept coming, region (re-)location would
> find the region still hosted on the stuck server. Finally the closing thread
> waiting for the write lock became no longer starved (by chance) after 40
> minutes.
> The ReentrantReadWriteLock javadoc is clear about the possibility of
> starvation when continuously contended: "_When constructed as non-fair (the
> default), the order of entry to the read and write lock is unspecified,
> subject to reentrancy constraints. A nonfair lock that is continuously
> contended may indefinitely postpone one or more reader or writer threads, but
> will normally have higher throughput than a fair lock._"
> We could try changing the acquisition semantics of this lock to fair. This is
> a one line change, where we call the RW lock constructor. Then:
> "_When constructed as fair, threads contend for entry using an approximately
> arrival-order policy. When the currently held lock is released, either the
> longest-waiting single writer thread will be assigned the write lock, or if
> there is a group of reader threads waiting longer than all waiting writer
> threads, that group will be assigned the read lock._"
> This could be better. The close process will have to wait until all readers
> and writers already waiting for acquisition either acquire and release or go
> away but won't be starved by future/incoming requests.
> There could be a throughput loss in request handling, though, because this is
> the global reentrant RW lock for the region.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24138) Ensure StochasticLoadBalancer can log details of decision to not run balancer
[
https://issues.apache.org/jira/browse/HBASE-24138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078935#comment-17078935
]
Hudson commented on HBASE-24138:
Results for branch branch-2
[build #2591 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2591/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2591/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2591/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2591/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2591/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Ensure StochasticLoadBalancer can log details of decision to not run balancer
> -
>
> Key: HBASE-24138
> URL: https://issues.apache.org/jira/browse/HBASE-24138
> Project: HBase
> Issue Type: Task
> Components: Balancer, Operability
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
>
> Ran into a customer case where the StochasticLoadBalancer was consistently
> deciding not to balance when bringing new region servers on line. Even
> setting the class to TRACE logging would only log a summary statement like:
> {code}
> 2020-04-03 00:29:55,133 TRACE
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Skipping load
> balancing because balanced cluster; total cost is 25.24853189705185, sum
> multiplier is 602.0 min cost which need balance is 0.05
> {code}
> Without any details about what went into that decision it's really hard to
> figure out what we need to tune to get the behavior we want.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24151) [rsgroup] The master server aborted for IllegalThreadStateException
kangkang.guo created HBASE-24151:
Summary: [rsgroup] The master server aborted for
IllegalThreadStateException
Key: HBASE-24151
URL: https://issues.apache.org/jira/browse/HBASE-24151
Project: HBase
Issue Type: Bug
Components: rsgroup
Affects Versions: 2.2.3, 2.1.7
Reporter: kangkang.guo
Assignee: kangkang.guo
if
'hbase.master.loadbalancer.class=org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer'
and
'hbase.rsgroup.grouploadbalancer.class=org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer',
The master server will aborted for IllegalThreadStateException.
{code:java}
2020-04-09 11:24:26,142 ERROR [master/xx.xx.xx.xx:16000:becomeActiveMaster]
master.HMaster: Failed to become active master
java.lang.IllegalThreadStateException
at java.lang.Thread.start(Thread.java:705)
at
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl.start(RSGroupInfoManagerImpl.java:172)
at
org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.initialize(RSGroupBasedLoadBalancer.java:417)
at
org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.initialize(RSGroupBasedLoadBalancer.java:430)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:968)
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2275)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:584)
at java.lang.Thread.run(Thread.java:745)
2020-04-09 11:24:26,144 ERROR [master/xx.xx.xx.xx:16000:becomeActiveMaster]
master.HMaster: Master server abort: loaded coprocessors are:
[org.apache.hadoop.hbase.security.access.AccessController,
org.apache.hadoop.hbase.quotas.MasterQuotasObserver,
org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint]{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24131) [Flakey Tests] TestExportSnapshot takes too long; up against 13min max
[ https://issues.apache.org/jira/browse/HBASE-24131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078925#comment-17078925 ] Viraj Jasani edited comment on HBASE-24131 at 4/9/20, 5:13 AM: --- [~stack] [~psomogyi] TestExportSnapshot.testExportFileSystemState() calls seem to be using incorrect arguments for snapshotName and targetName. Just committed an addendum patch to fix it. Edit: branch-2 is all good, the signature change is present in master only. was (Author: vjasani): [~stack] [~psomogyi] TestExportSnapshot.testExportFileSystemState() calls seem to be using incorrect arguments for snapshotName and targetName. Just committed an addendum patch to fix it. > [Flakey Tests] TestExportSnapshot takes too long; up against 13min max > -- > > Key: HBASE-24131 > URL: https://issues.apache.org/jira/browse/HBASE-24131 > Project: HBase > Issue Type: Bug >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > TestExportSnapshot fails fairly regularly locally. Looking, its test timeout. > Looking at how long it ran, its 13minutes plus. Looking at recent successful > branch-2 run, it passed but it took about 7-8minutes. Let me break up the > test into to pieces. > org.junit.runners.model.TestTimedOutException: test timed out after 780 > seconds >at > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:227) >at > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportRetry(TestExportSnapshot.java:267) > ... I see this in the log: > > TEST TIMED OUT. PRINTING THREAD DUMP. < > Test started at: > 2020-04-06 17:19:21,739 INFO > ... and the timestamp just above the TIMED OUT was > 2020-04-06 17:31:01,758 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24131) [Flakey Tests] TestExportSnapshot takes too long; up against 13min max
[ https://issues.apache.org/jira/browse/HBASE-24131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078926#comment-17078926 ] Michael Stack commented on HBASE-24131: --- [~vjasani] Thanks. > [Flakey Tests] TestExportSnapshot takes too long; up against 13min max > -- > > Key: HBASE-24131 > URL: https://issues.apache.org/jira/browse/HBASE-24131 > Project: HBase > Issue Type: Bug >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > TestExportSnapshot fails fairly regularly locally. Looking, its test timeout. > Looking at how long it ran, its 13minutes plus. Looking at recent successful > branch-2 run, it passed but it took about 7-8minutes. Let me break up the > test into to pieces. > org.junit.runners.model.TestTimedOutException: test timed out after 780 > seconds >at > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:227) >at > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportRetry(TestExportSnapshot.java:267) > ... I see this in the log: > > TEST TIMED OUT. PRINTING THREAD DUMP. < > Test started at: > 2020-04-06 17:19:21,739 INFO > ... and the timestamp just above the TIMED OUT was > 2020-04-06 17:31:01,758 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24131) [Flakey Tests] TestExportSnapshot takes too long; up against 13min max
[ https://issues.apache.org/jira/browse/HBASE-24131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078925#comment-17078925 ] Viraj Jasani commented on HBASE-24131: -- [~stack] [~psomogyi] TestExportSnapshot.testExportFileSystemState() calls seem to be using incorrect arguments for snapshotName and targetName. Just committed an addendum patch to fix it. > [Flakey Tests] TestExportSnapshot takes too long; up against 13min max > -- > > Key: HBASE-24131 > URL: https://issues.apache.org/jira/browse/HBASE-24131 > Project: HBase > Issue Type: Bug >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > TestExportSnapshot fails fairly regularly locally. Looking, its test timeout. > Looking at how long it ran, its 13minutes plus. Looking at recent successful > branch-2 run, it passed but it took about 7-8minutes. Let me break up the > test into to pieces. > org.junit.runners.model.TestTimedOutException: test timed out after 780 > seconds >at > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:227) >at > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportRetry(TestExportSnapshot.java:267) > ... I see this in the log: > > TEST TIMED OUT. PRINTING THREAD DUMP. < > Test started at: > 2020-04-06 17:19:21,739 INFO > ... and the timestamp just above the TIMED OUT was > 2020-04-06 17:31:01,758 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1439: HBASE-24121 [Authorization] ServiceAuthorizationManager isn't dynamically updatable. And it should be
Apache-HBase commented on issue #1439: HBASE-24121 [Authorization] ServiceAuthorizationManager isn't dynamically updatable. And it should be URL: https://github.com/apache/hbase/pull/1439#issuecomment-611330027 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 5s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | -1 :x: | mvninstall | 6m 34s | root in master failed. | | +1 :green_heart: | checkstyle | 1m 20s | master passed | | +1 :green_heart: | spotbugs | 2m 38s | master passed | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 3m 15s | root in the patch failed. | | +1 :green_heart: | checkstyle | 1m 22s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | -1 :x: | hadoopcheck | 2m 8s | The patch causes 17 errors with Hadoop v2.10.0. | | -1 :x: | hadoopcheck | 4m 23s | The patch causes 17 errors with Hadoop v3.1.2. | | -1 :x: | spotbugs | 0m 42s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 24m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1439 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 251475ef95a3 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / eb7df0498c | | mvninstall | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-general-check/output/branch-mvninstall-root.txt | | mvninstall | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-general-check/output/patch-mvninstall-root.txt | | hadoopcheck | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-general-check/output/patch-javac-2.10.0.txt | | hadoopcheck | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-general-check/output/patch-javac-3.1.2.txt | | spotbugs | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-general-check/output/patch-spotbugs-hbase-server.txt | | Max. process+thread count | 83 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on issue #1453: HBASE-24132 Upgrade to Apache ZooKeeper 3.5.7
saintstack commented on issue #1453: HBASE-24132 Upgrade to Apache ZooKeeper 3.5.7 URL: https://github.com/apache/hbase/pull/1453#issuecomment-611330475 Can 3.5.7 client talk to a 3.4 ensemble and vice-versa? Are their compatibility issues? Would be good to note in release note for this JIRA. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1439: HBASE-24121 [Authorization] ServiceAuthorizationManager isn't dynamically updatable. And it should be
Apache-HBase commented on issue #1439: HBASE-24121 [Authorization] ServiceAuthorizationManager isn't dynamically updatable. And it should be URL: https://github.com/apache/hbase/pull/1439#issuecomment-611330236 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | -1 :x: | mvninstall | 5m 58s | root in master failed. | | +1 :green_heart: | compile | 1m 11s | master passed | | -1 :x: | shadedjars | 4m 39s | branch has 16 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | master passed | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 3m 1s | root in the patch failed. | | -1 :x: | compile | 0m 40s | hbase-server in the patch failed. | | -0 :warning: | javac | 0m 40s | hbase-server in the patch failed. | | -1 :x: | shadedjars | 3m 52s | patch has 17 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 0m 37s | hbase-server in the patch failed. | | | | 25m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1439 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e1823bd758c4 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / eb7df0498c | | Default Java | 1.8.0_232 | | mvninstall | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-jdk8-hadoop2-check/output/branch-mvninstall-root.txt | | shadedjars | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-jdk8-hadoop2-check/output/branch-shadedjars.txt | | mvninstall | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-jdk8-hadoop2-check/output/patch-mvninstall-root.txt | | compile | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-jdk8-hadoop2-check/output/patch-compile-hbase-server.txt | | javac | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-jdk8-hadoop2-check/output/patch-compile-hbase-server.txt | | shadedjars | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-jdk8-hadoop2-check/output/patch-shadedjars.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/testReport/ | | Max. process+thread count | 77 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1439: HBASE-24121 [Authorization] ServiceAuthorizationManager isn't dynamically updatable. And it should be
Apache-HBase commented on issue #1439: HBASE-24121 [Authorization] ServiceAuthorizationManager isn't dynamically updatable. And it should be URL: https://github.com/apache/hbase/pull/1439#issuecomment-611329232 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | -1 :x: | mvninstall | 5m 0s | root in master failed. | | +1 :green_heart: | compile | 1m 2s | master passed | | -1 :x: | shadedjars | 4m 29s | branch has 16 errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 38s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 2m 48s | root in the patch failed. | | -1 :x: | compile | 0m 38s | hbase-server in the patch failed. | | -0 :warning: | javac | 0m 38s | hbase-server in the patch failed. | | -1 :x: | shadedjars | 3m 48s | patch has 17 errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 37s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 0m 38s | hbase-server in the patch failed. | | | | 21m 16s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1439 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 809be8a06d48 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / eb7df0498c | | Default Java | 2020-01-14 | | mvninstall | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-jdk11-hadoop3-check/output/branch-mvninstall-root.txt | | shadedjars | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-jdk11-hadoop3-check/output/branch-shadedjars.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | mvninstall | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-jdk11-hadoop3-check/output/patch-mvninstall-root.txt | | compile | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-server.txt | | javac | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-server.txt | | shadedjars | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/testReport/ | | Max. process+thread count | 100 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1439/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1464: HBASE-24150 Allow module tests run in parallel
Apache-HBase commented on issue #1464: HBASE-24150 Allow module tests run in parallel URL: https://github.com/apache/hbase/pull/1464#issuecomment-611322273 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 46s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 7s | Maven dependency ordering for patch | ||| _ Other Tests _ | | | | 2m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1464/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1464 | | Optional Tests | | | uname | Linux a159c2394bdf 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 611c62f52a | | Max. process+thread count | 46 (vs. ulimit of 12500) | | modules | C: U: | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1464/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1464: HBASE-24150 Allow module tests run in parallel
Apache-HBase commented on issue #1464: HBASE-24150 Allow module tests run in parallel URL: https://github.com/apache/hbase/pull/1464#issuecomment-611322277 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 46s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 7s | Maven dependency ordering for patch | ||| _ Other Tests _ | | | | 2m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1464/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1464 | | Optional Tests | | | uname | Linux f0856ce1ca6b 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 611c62f52a | | Max. process+thread count | 57 (vs. ulimit of 12500) | | modules | C: U: | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1464/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1464: HBASE-24150 Allow module tests run in parallel
Apache-HBase commented on issue #1464: HBASE-24150 Allow module tests run in parallel URL: https://github.com/apache/hbase/pull/1464#issuecomment-611322291 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 43s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 7s | Maven dependency ordering for patch | | +1 :green_heart: | shellcheck | 0m 2s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? | | | | 2m 15s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1464/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1464 | | Optional Tests | dupname asflicense shellcheck shelldocs | | uname | Linux 21fa6e0453e3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 611c62f52a | | Max. process+thread count | 50 (vs. ulimit of 12500) | | modules | C: U: | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1464/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) shellcheck=0.4.6 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1464: HBASE-24150 Allow module tests run in parallel
Apache-HBase commented on issue #1464: HBASE-24150 Allow module tests run in parallel URL: https://github.com/apache/hbase/pull/1464#issuecomment-611321993 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1464/1/console in case of problems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1464: HBASE-24150 Allow module tests run in parallel
Apache-HBase commented on issue #1464: HBASE-24150 Allow module tests run in parallel URL: https://github.com/apache/hbase/pull/1464#issuecomment-611321949 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1464/1/console in case of problems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1464: HBASE-24150 Allow module tests run in parallel
Apache-HBase commented on issue #1464: HBASE-24150 Allow module tests run in parallel URL: https://github.com/apache/hbase/pull/1464#issuecomment-611321986 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1464/1/console in case of problems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23779) Up the default fork count to make builds complete faster; make count relative to CPU count
[ https://issues.apache.org/jira/browse/HBASE-23779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078905#comment-17078905 ] Michael Stack commented on HBASE-23779: --- The new subissue HBASE-24150 takes a different tack. It allows two modules when possible to run their tests in parallel. It leaves the forkcount at 0.25C but ups the mvn --threads argument from default of '1' to '2'. In tests running local builds, a machine completed full test suite in 1hr 15mins w/ forkcount of 1.0C and --threads=1. When same machine ran full suite of tests with forkcount of 0.5C and --threads=2, it completed in 53mins. Lets see what we get. > Up the default fork count to make builds complete faster; make count relative > to CPU count > -- > > Key: HBASE-23779 > URL: https://issues.apache.org/jira/browse/HBASE-23779 > Project: HBase > Issue Type: Task > Components: test >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: Screen Shot 2020-04-08 at 9.19.53 AM.png, > addendum2.patch, test_yetus_934.0.patch > > > Tests take a long time. Our fork count running all tests are conservative -- > 1 (small) for first part and 5 for second part (medium and large). Rather > than hardcoding we should set the fork count to be relative to machine size. > Suggestion here is 0.75C where C is CPU count. This ups the CPU use on my box. > Looking up at jenkins, it seems like the boxes are 24 cores... at least going > by my random survey. The load reported on a few seems low though this not > representative (looking at machine/uptime). > More parallelism willl probably mean more test failure. Let me take a look > see. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-23779) Up the default fork count to make builds complete faster; make count relative to CPU count
[ https://issues.apache.org/jira/browse/HBASE-23779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078749#comment-17078749 ] Michael Stack edited comment on HBASE-23779 at 4/9/20, 4:28 AM: Branch-2 has the upped nproc patch applied (HBASE-24126) and infra upped the per-user nproc count from 1 to 3. branch-2 also has HBASE-24134 applied so we should use less memory. The new PR against this issue, https://github.com/apache/hbase/pull/1461, changes our forkcount from 0.25 to 0.5 and passes -T to mvn of 0.5C. Looking to see the test run time halved. Below are what branch-2 build times currently look like. !Screen Shot 2020-04-08 at 9.19.53 AM.png! was (Author: stack): Branch-2 has the upped nproc patch applied (HBASE-24126) and infra upped the per-user nproc count from 1 to 3. branch-2 also has HBASE-24134 applied so we should use less memory. The new PR https://github.com/apache/hbase/pull/1461 changes our forkcount from 0.25 to 0.5 and passes -T to mvn of 0.5C. Looking to see the test run time halved. Below are what branch-2 build times currently look like. !Screen Shot 2020-04-08 at 9.19.53 AM.png! > Up the default fork count to make builds complete faster; make count relative > to CPU count > -- > > Key: HBASE-23779 > URL: https://issues.apache.org/jira/browse/HBASE-23779 > Project: HBase > Issue Type: Task > Components: test >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: Screen Shot 2020-04-08 at 9.19.53 AM.png, > addendum2.patch, test_yetus_934.0.patch > > > Tests take a long time. Our fork count running all tests are conservative -- > 1 (small) for first part and 5 for second part (medium and large). Rather > than hardcoding we should set the fork count to be relative to machine size. > Suggestion here is 0.75C where C is CPU count. This ups the CPU use on my box. > Looking up at jenkins, it seems like the boxes are 24 cores... at least going > by my random survey. The load reported on a few seems low though this not > representative (looking at machine/uptime). > More parallelism willl probably mean more test failure. Let me take a look > see. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23823) Run maven with more than default single thread (--threads=0.5C)
[ https://issues.apache.org/jira/browse/HBASE-23823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23823: -- Resolution: Later Status: Resolved (was: Patch Available) Resolving this as 'later'. I learned some stuff in here and then closed the PR and the parent issue because thought this a hopeless case. Subsequently thought different. Reopened the parent. Have put up new PR that ups the forkcount from 0.25C to 0.5C against the parent. Will continue these investigations up there. > Run maven with more than default single thread (--threads=0.5C) > --- > > Key: HBASE-23823 > URL: https://issues.apache.org/jira/browse/HBASE-23823 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch, > 0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch, > 0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch, > 0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch > > > See tail of parent task where we made commits to hbase-personality trying to > make maven run more furiously on nightly builds by upping the maven thread > count (the --threads flag ... see > https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3). > The effort was abandoned because it unsettled the build. This issue > separates messing with the maven threads flag from the broader topic the > parent covers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1458: HBASE-24140 : Move CandidateGenerator and their implementors out of S…
Apache-HBase commented on issue #1458: HBASE-24140 : Move CandidateGenerator and their implementors out of S… URL: https://github.com/apache/hbase/pull/1458#issuecomment-611321135 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | -1 :x: | mvninstall | 4m 35s | root in master failed. | | +1 :green_heart: | compile | 0m 52s | master passed | | -1 :x: | shadedjars | 3m 56s | branch has 16 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | master passed | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 4m 6s | root in the patch failed. | | +1 :green_heart: | compile | 0m 54s | the patch passed | | +1 :green_heart: | javac | 0m 54s | the patch passed | | -1 :x: | shadedjars | 3m 57s | patch has 16 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 32s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 6m 0s | hbase-server in the patch failed. | | | | 27m 23s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/3/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1458 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f4411935d85d 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / eb7df0498c | | Default Java | 1.8.0_232 | | mvninstall | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/3/artifact/yetus-jdk8-hadoop2-check/output/branch-mvninstall-root.txt | | shadedjars | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/3/artifact/yetus-jdk8-hadoop2-check/output/branch-shadedjars.txt | | mvninstall | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/3/artifact/yetus-jdk8-hadoop2-check/output/patch-mvninstall-root.txt | | shadedjars | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/3/artifact/yetus-jdk8-hadoop2-check/output/patch-shadedjars.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/3/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/3/testReport/ | | Max. process+thread count | 649 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1458: HBASE-24140 : Move CandidateGenerator and their implementors out of S…
Apache-HBase commented on issue #1458: HBASE-24140 : Move CandidateGenerator and their implementors out of S… URL: https://github.com/apache/hbase/pull/1458#issuecomment-611321011 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | -1 :x: | mvninstall | 6m 1s | root in master failed. | | +1 :green_heart: | checkstyle | 1m 18s | master passed | | +1 :green_heart: | spotbugs | 2m 12s | master passed | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 4m 51s | root in the patch failed. | | -0 :warning: | checkstyle | 1m 15s | hbase-server: The patch generated 1 new + 28 unchanged - 5 fixed = 29 total (was 33) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | -1 :x: | hadoopcheck | 2m 44s | The patch causes 16 errors with Hadoop v2.10.0. | | -1 :x: | hadoopcheck | 5m 30s | The patch causes 16 errors with Hadoop v3.1.2. | | +1 :green_heart: | spotbugs | 2m 21s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 26m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1458 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 10f3456f981e 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / eb7df0498c | | mvninstall | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/3/artifact/yetus-general-check/output/branch-mvninstall-root.txt | | mvninstall | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/3/artifact/yetus-general-check/output/patch-mvninstall-root.txt | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | hadoopcheck | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/3/artifact/yetus-general-check/output/patch-javac-2.10.0.txt | | hadoopcheck | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/3/artifact/yetus-general-check/output/patch-javac-3.1.2.txt | | Max. process+thread count | 83 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;…
saintstack commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;… URL: https://github.com/apache/hbase/pull/1461#issuecomment-611320338 Rerunning tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1461: HBASE-23779 Up the default fork count to make builds complete faster;…
saintstack commented on a change in pull request #1461: HBASE-23779 Up the
default fork count to make builds complete faster;…
URL: https://github.com/apache/hbase/pull/1461#discussion_r405948274
##
File path: dev-support/hbase-personality.sh
##
@@ -140,7 +140,15 @@ function personality_modules
clear_personality_queue
- extra="-DHBasePatchProcess"
+ # Set a fork count based off the host cpu count. Pass maven a -T argument.
+ # Default -T is one thread. 0.5C on an apache box of 16 cores and 2 jenkins
+ # 'executors' per host should make for 8 threads. Setting this here for yetus
+ # to pick up. See
+ #
https://yetus.apache.org/documentation/0.11.1/precommit-advanced/#global-definitions
+ # See below for more on -T:
+ #
https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3
+ forkcount="0.5C"
+ extra="-T${forkcount} -Dsurefire.firstPartForkCount=${forkcount}
-Dsurefire.secondPartForkCount=${forkcount} -DHBasePatchProcess"
Review comment:
Opened HBASE-24150 Might be more speed up there than here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack opened a new pull request #1464: HBASE-24150 Allow module tests run in parallel
saintstack opened a new pull request #1464: HBASE-24150 Allow module tests run in parallel URL: https://github.com/apache/hbase/pull/1464 Pass --threads=2 to mvn when yetus runs so some parallelism when dependencies allow. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;…
Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;… URL: https://github.com/apache/hbase/pull/1461#issuecomment-611318664 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 46s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | ||| _ Patch Compile Tests _ | | +1 :green_heart: | shellcheck | 0m 2s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 10m 55s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 17s | The patch does not generate ASF License warnings. | | | | 18m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1461 | | Optional Tests | dupname asflicense shellcheck shelldocs hadoopcheck xml | | uname | Linux 277bf39c0c66 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 611c62f52a | | Max. process+thread count | 76 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) shellcheck=0.4.6 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1453: HBASE-24132 Upgrade to Apache ZooKeeper 3.5.7
Apache-HBase commented on issue #1453: HBASE-24132 Upgrade to Apache ZooKeeper 3.5.7 URL: https://github.com/apache/hbase/pull/1453#issuecomment-611317847 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for branch | | -1 :x: | mvninstall | 5m 43s | root in master failed. | | +1 :green_heart: | checkstyle | 3m 13s | master passed | | -1 :x: | spotbugs | 9m 44s | root in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 5m 20s | root in the patch failed. | | +1 :green_heart: | checkstyle | 2m 52s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 3s | The patch has no ill-formed XML file. | | -1 :x: | hadoopcheck | 2m 47s | The patch causes 16 errors with Hadoop v2.10.0. | | -1 :x: | hadoopcheck | 5m 40s | The patch causes 16 errors with Hadoop v3.1.2. | | -1 :x: | spotbugs | 9m 42s | root in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 48s | The patch does not generate ASF License warnings. | | | | 53m 32s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1453/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1453 | | Optional Tests | dupname asflicense xml spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 18ee689fb500 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / eb7df0498c | | mvninstall | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1453/3/artifact/yetus-general-check/output/branch-mvninstall-root.txt | | spotbugs | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1453/3/artifact/yetus-general-check/output/branch-spotbugs-root.txt | | mvninstall | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1453/3/artifact/yetus-general-check/output/patch-mvninstall-root.txt | | hadoopcheck | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1453/3/artifact/yetus-general-check/output/patch-javac-2.10.0.txt | | hadoopcheck | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1453/3/artifact/yetus-general-check/output/patch-javac-3.1.2.txt | | spotbugs | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1453/3/artifact/yetus-general-check/output/patch-spotbugs-root.txt | | Max. process+thread count | 83 (vs. ulimit of 1) | | modules | C: hbase-zookeeper hbase-resource-bundle hbase-server . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1453/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Reidddddd commented on issue #1439: HBASE-24121 [Authorization] ServiceAuthorizationManager isn't dynamically updatable. And it should be
Reidd commented on issue #1439: HBASE-24121 [Authorization] ServiceAuthorizationManager isn't dynamically updatable. And it should be URL: https://github.com/apache/hbase/pull/1439#issuecomment-611317003 > Meaning someone updates the policy.xml with a new policy, the test runs something to make sure the new policy is in affect, something along those lines. Yeah, this is what I said, it is exactly about functionality about HDFS's ServiceAuthorizationManager which shouldn't be tested on HBase side. Even if it doesn't take effect, we can do nothing on HBase side (BTW, it does take effect). To me, it is a kind of over-development. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (HBASE-24150) Allow module tests run in parallel
Michael Stack created HBASE-24150: - Summary: Allow module tests run in parallel Key: HBASE-24150 URL: https://issues.apache.org/jira/browse/HBASE-24150 Project: HBase Issue Type: Sub-task Components: build Reporter: Michael Stack Parent issue is about upping forkcount. This one is about mildly letting modules run tests in parallel. Parent wants forkcount to be 0.5C so we use half the processors. This is about keeping forkcount at 0.25C but allowing two modules to run their tests at same time when mvn figures it is possible. In a test on a local machine, using forkcount of 1.0C and -T == 1 -- default -- the full test suite took 1hr 15mins. With forkcount at 0.5 and T == 2, full suite took 53mins. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;…
Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;… URL: https://github.com/apache/hbase/pull/1461#issuecomment-611314800 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/3/console in case of problems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;…
Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;… URL: https://github.com/apache/hbase/pull/1461#issuecomment-611314804 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/3/console in case of problems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;…
Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;… URL: https://github.com/apache/hbase/pull/1461#issuecomment-611314789 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/3/console in case of problems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1461: HBASE-23779 Up the default fork count to make builds complete faster;…
saintstack commented on a change in pull request #1461: HBASE-23779 Up the
default fork count to make builds complete faster;…
URL: https://github.com/apache/hbase/pull/1461#discussion_r405942044
##
File path: dev-support/hbase-personality.sh
##
@@ -140,7 +140,15 @@ function personality_modules
clear_personality_queue
- extra="-DHBasePatchProcess"
+ # Set a fork count based off the host cpu count. Pass maven a -T argument.
+ # Default -T is one thread. 0.5C on an apache box of 16 cores and 2 jenkins
+ # 'executors' per host should make for 8 threads. Setting this here for yetus
+ # to pick up. See
+ #
https://yetus.apache.org/documentation/0.11.1/precommit-advanced/#global-definitions
+ # See below for more on -T:
+ #
https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3
+ forkcount="0.5C"
+ extra="-T${forkcount} -Dsurefire.firstPartForkCount=${forkcount}
-Dsurefire.secondPartForkCount=${forkcount} -DHBasePatchProcess"
Review comment:
Hmm... on big box, with 1.0C and no T, build takes 1hr 15mins. With 0.5C and
T==2, takes 53minutes. There are probably spurts of lots of processes but
interesting that overall time taken goes down significantly. Let me come back
and set the T in another follow-on issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (HBASE-24099) Use a fair ReentrantReadWriteLock for the region close lock
[
https://issues.apache.org/jira/browse/HBASE-24099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1707#comment-1707
]
Hudson commented on HBASE-24099:
Results for branch branch-2.1
[build #1851 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1851/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1851//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1851//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1851//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Use a fair ReentrantReadWriteLock for the region close lock
> ---
>
> Key: HBASE-24099
> URL: https://issues.apache.org/jira/browse/HBASE-24099
> Project: HBase
> Issue Type: Improvement
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 1.3.7, 1.7.0, 2.4.0, 2.1.10, 1.4.14, 2.2.5
>
> Attachments: ltt_results.pdf, pe_results.pdf, ycsb_results.pdf
>
>
> Consider creating the region's ReentrantReadWriteLock with the fair locking
> policy. We have had a couple of production incidents where a regionserver
> stalled in shutdown for a very very long time, leading to RIT (FAILED_CLOSE).
> The latest example is a 43 minute shutdown, ~40 minutes (2465280 ms) of that
> time was spent waiting to acquire the write lock on the region in order to
> finish closing it.
> {quote}
> ...
> Finished memstore flush of ~66.92 MB/70167112, currentsize=0 B/0 for region
> . in 927ms, sequenceid=6091133815, compaction requested=false at
> 1585175635349 (+60 ms)
> Disabling writes for close at 1585178100629 (+2465280 ms)
> {quote}
> This time was spent in between the memstore flush and the task status change
> "Disabling writes for close at...". This is at HRegion.java:1481 in 1.3.6:
> {code}
> 1480: // block waiting for the lock for closing
> 1481: lock.writeLock().lock(); // FindBugs: Complains
> UL_UNRELEASED_LOCK_EXCEPTION_PATH but seems fine
> {code}
>
> The close lock is operating in unfair mode. The table in question is under
> constant high query load. When the close request was received, there were
> active readers. After the close request there were more active readers,
> near-continuous contention. Although the clients would receive
> RegionServerStoppingException and other error notifications, because the
> region could not be reassigned, they kept coming, region (re-)location would
> find the region still hosted on the stuck server. Finally the closing thread
> waiting for the write lock became no longer starved (by chance) after 40
> minutes.
> The ReentrantReadWriteLock javadoc is clear about the possibility of
> starvation when continuously contended: "_When constructed as non-fair (the
> default), the order of entry to the read and write lock is unspecified,
> subject to reentrancy constraints. A nonfair lock that is continuously
> contended may indefinitely postpone one or more reader or writer threads, but
> will normally have higher throughput than a fair lock._"
> We could try changing the acquisition semantics of this lock to fair. This is
> a one line change, where we call the RW lock constructor. Then:
> "_When constructed as fair, threads contend for entry using an approximately
> arrival-order policy. When the currently held lock is released, either the
> longest-waiting single writer thread will be assigned the write lock, or if
> there is a group of reader threads waiting longer than all waiting writer
> threads, that group will be assigned the read lock._"
> This could be better. The close process will have to wait until all readers
> and writers already waiting for acquisition either acquire and release or go
> away but won't be starved by future/incoming requests.
> There could be a throughput loss in request handling, though, because this is
> the global reentrant RW lock for the region.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;…
Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;… URL: https://github.com/apache/hbase/pull/1461#issuecomment-611312241 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 58s | Docker mode activated. | | -0 :warning: | yetus | 0m 8s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 8m 17s | branch-2 passed | | +1 :green_heart: | compile | 4m 25s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 54s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 23s | root in branch-2 failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 7m 15s | the patch passed | | +1 :green_heart: | compile | 4m 25s | the patch passed | | +1 :green_heart: | javac | 4m 25s | the patch passed | | +1 :green_heart: | shadedjars | 5m 58s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 19s | root in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 165m 35s | root in the patch passed. | | | | 206m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1461 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux bd34ff2524ef 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 611c62f52a | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-root.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/2/testReport/ | | Max. process+thread count | 6467 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1462: HBASE-24148: Upgrade Thrift to 0.13.0
Apache-HBase commented on issue #1462: HBASE-24148: Upgrade Thrift to 0.13.0 URL: https://github.com/apache/hbase/pull/1462#issuecomment-611307008 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 9s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 39s | Maven dependency ordering for branch | | -1 :x: | mvninstall | 5m 42s | root in master failed. | | -1 :x: | compile | 2m 36s | root in master failed. | | -1 :x: | shadedjars | 4m 59s | branch has 16 errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 19s | root in master failed. | | -0 :warning: | javadoc | 0m 18s | hbase-mapreduce in master failed. | | -0 :warning: | javadoc | 0m 56s | hbase-thrift in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 6m 47s | the patch passed | | +1 :green_heart: | compile | 4m 0s | the patch passed | | +1 :green_heart: | javac | 4m 0s | root generated 0 new + 34 unchanged - 4 fixed = 34 total (was 38) | | +1 :green_heart: | shadedjars | 5m 42s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 19s | hbase-mapreduce in the patch failed. | | -0 :warning: | javadoc | 0m 56s | hbase-thrift in the patch failed. | | -0 :warning: | javadoc | 0m 18s | root in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 247m 43s | root in the patch passed. | | | | 285m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1462 | | JIRA Issue | HBASE-24148 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ebb093013a07 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / eb7df0498c | | Default Java | 2020-01-14 | | mvninstall | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/artifact/yetus-jdk11-hadoop3-check/output/branch-mvninstall-root.txt | | compile | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/artifact/yetus-jdk11-hadoop3-check/output/branch-compile-root.txt | | shadedjars | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/artifact/yetus-jdk11-hadoop3-check/output/branch-shadedjars.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-root.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-mapreduce.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-thrift.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-mapreduce.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-thrift.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/testReport/ | | Max. process+thread count | 3383 (vs. ulimit of 1) | | modules | C: hbase-mapreduce hbase-thrift . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With
[GitHub] [hbase] Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;…
Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;… URL: https://github.com/apache/hbase/pull/1461#issuecomment-611294271 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 36s | branch-2 passed | | +1 :green_heart: | compile | 2m 59s | branch-2 passed | | +1 :green_heart: | shadedjars | 4m 17s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 46s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 5s | the patch passed | | +1 :green_heart: | compile | 2m 59s | the patch passed | | +1 :green_heart: | javac | 3m 0s | the patch passed | | +1 :green_heart: | shadedjars | 4m 20s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 43s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 99m 11s | root in the patch failed. | | | | 133m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1461 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux dca13b9f23b7 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 611c62f52a | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/2/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/2/testReport/ | | Max. process+thread count | 7529 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-15161) Umbrella: Miscellaneous improvements from production usage
[ https://issues.apache.org/jira/browse/HBASE-15161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078859#comment-17078859 ] Yu Li commented on HBASE-15161: --- +1 on updating the document to reflect the metrics improvements (if not already). And I believe HBASE-15595 deserves a release-note to remind our users/operators, as a remedy for the inadequate notification of existing sub-tasks which are actually "Improvement" or "New Feature". It's also true that sub-tasks have already been merged for different versions, so I think we could directly close this one w/o any fix version or release note - it's just an umbrella for tracking purpose. Wdyt? Thanks. > Umbrella: Miscellaneous improvements from production usage > -- > > Key: HBASE-15161 > URL: https://issues.apache.org/jira/browse/HBASE-15161 > Project: HBase > Issue Type: Improvement >Reporter: Yu Li >Assignee: Yu Li >Priority: Major > > We use HBase to (mainly) build index for our search engine in Alibaba. > Recently we are upgrading our online cluster from 0.98.12 to 1.x and I'd like > to take the opportunity to contribute a bunch of our private patches to > community (better late than never, I hope :-)). This is an umbrella to track > this effort. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24137) The max merge count of metafixer should be configurable in MetaFixer
[ https://issues.apache.org/jira/browse/HBASE-24137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078856#comment-17078856 ] Yu Wang commented on HBASE-24137: - [~ndimiduk] Please review it , Thanks > The max merge count of metafixer should be configurable in MetaFixer > > > Key: HBASE-24137 > URL: https://issues.apache.org/jira/browse/HBASE-24137 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Yu Wang >Priority: Minor > Fix For: 3.0.0 > > Attachments: 24137_master_1.patch > > > The max merge count of metafixer should be configurable in MetaFixer -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1462: HBASE-24148: Upgrade Thrift to 0.13.0
Apache-HBase commented on issue #1462: HBASE-24148: Upgrade Thrift to 0.13.0 URL: https://github.com/apache/hbase/pull/1462#issuecomment-611284275 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 43s | Maven dependency ordering for branch | | -1 :x: | mvninstall | 4m 18s | root in master failed. | | -1 :x: | compile | 1m 48s | root in master failed. | | -1 :x: | shadedjars | 3m 55s | branch has 16 errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 1m 57s | root in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 6s | the patch passed | | +1 :green_heart: | compile | 2m 51s | the patch passed | | +1 :green_heart: | javac | 2m 51s | root generated 0 new + 34 unchanged - 4 fixed = 34 total (was 38) | | +1 :green_heart: | shadedjars | 4m 28s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 2m 36s | root generated 72 new + 15 unchanged - 0 fixed = 87 total (was 15) | ||| _ Other Tests _ | | +1 :green_heart: | unit | 163m 21s | root in the patch passed. | | | | 196m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1462 | | JIRA Issue | HBASE-24148 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a7e449387982 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / eb7df0498c | | Default Java | 1.8.0_232 | | mvninstall | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/artifact/yetus-jdk8-hadoop2-check/output/branch-mvninstall-root.txt | | compile | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/artifact/yetus-jdk8-hadoop2-check/output/branch-compile-root.txt | | shadedjars | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/artifact/yetus-jdk8-hadoop2-check/output/branch-shadedjars.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/artifact/yetus-jdk8-hadoop2-check/output/branch-javadoc-root.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/artifact/yetus-jdk8-hadoop2-check/output/diff-javadoc-javadoc-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/testReport/ | | Max. process+thread count | 4446 (vs. ulimit of 1) | | modules | C: hbase-mapreduce hbase-thrift . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-24137) The max merge count of metafixer should be configurable in MetaFixer
[ https://issues.apache.org/jira/browse/HBASE-24137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078854#comment-17078854 ] Ted Yu commented on HBASE-24137: Please find people who have touched this class to review. I may not have time to review. > The max merge count of metafixer should be configurable in MetaFixer > > > Key: HBASE-24137 > URL: https://issues.apache.org/jira/browse/HBASE-24137 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Yu Wang >Priority: Minor > Fix For: 3.0.0 > > Attachments: 24137_master_1.patch > > > The max merge count of metafixer should be configurable in MetaFixer -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24137) The max merge count of metafixer should be configurable in MetaFixer
[ https://issues.apache.org/jira/browse/HBASE-24137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078852#comment-17078852 ] Yu Wang commented on HBASE-24137: - [~yuzhih...@gmail.com] Please review it , Thanks > The max merge count of metafixer should be configurable in MetaFixer > > > Key: HBASE-24137 > URL: https://issues.apache.org/jira/browse/HBASE-24137 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Yu Wang >Priority: Minor > Fix For: 3.0.0 > > Attachments: 24137_master_1.patch > > > The max merge count of metafixer should be configurable in MetaFixer -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24137) The max merge count of metafixer should be configurable in MetaFixer
[ https://issues.apache.org/jira/browse/HBASE-24137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Wang updated HBASE-24137: Fix Version/s: 3.0.0 > The max merge count of metafixer should be configurable in MetaFixer > > > Key: HBASE-24137 > URL: https://issues.apache.org/jira/browse/HBASE-24137 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Yu Wang >Priority: Minor > Fix For: 3.0.0 > > Attachments: 24137_master_1.patch > > > The max merge count of metafixer should be configurable in MetaFixer -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23647) Make MasterRegistry the default registry impl
[ https://issues.apache.org/jira/browse/HBASE-23647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078836#comment-17078836 ] Sean Busbey commented on HBASE-23647: - I thought this change made the new registry the default and was thus only in hbase 3.0 and not currently planned for a 2.x release? Why is it marked as 2.3.0? > Make MasterRegistry the default registry impl > - > > Key: HBASE-23647 > URL: https://issues.apache.org/jira/browse/HBASE-23647 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 3.0.0 >Reporter: Bharath Vissapragada >Assignee: Bharath Vissapragada >Priority: Major > Fix For: 3.0.0, 2.3.0, HBASE-18095 > > > Now that we are close to getting the registry implementation patch in, the > idea here is to make it the default implementation in 3.0.0 and this means > - No known bugs with the implementation > - No known performance issues > - Entire nightly test suite is green (and without flakes). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase-thirdparty] busbey commented on issue #16: HBASE-24149: Bump netty version to 4.1.48.Final
busbey commented on issue #16: HBASE-24149: Bump netty version to 4.1.48.Final URL: https://github.com/apache/hbase-thirdparty/pull/16#issuecomment-611269160 Did you already try using the updated hbase-thirdparty artifact with the main repo? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-24120) Flakey Test: TestReplicationAdminWithClusters timeout
[
https://issues.apache.org/jira/browse/HBASE-24120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-24120:
--
Fix Version/s: (was: 2.4.0)
(was: master)
3.0.0
> Flakey Test: TestReplicationAdminWithClusters timeout
> --
>
> Key: HBASE-24120
> URL: https://issues.apache.org/jira/browse/HBASE-24120
> Project: HBase
> Issue Type: Test
> Components: Replication
>Affects Versions: 2.3.0, master, 2.4.0
>Reporter: Huaxiang Sun
>Assignee: Hua Xiang
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> {code:java}
> 2020-04-05 23:36:53,092 ERROR
> [RS_REFRESH_PEER-regionserver/asf905:0-0.replicationSource,2.replicationSource.shipperasf905.gq1.ygridcore.net%2C42849%2C1586129728118,2]
> regionserver.ReplicationSource(397): Unexpected exception in
> RS_REFRESH_PEER-regionserver/asf905:0-0.replicationSource,2.replicationSource.shipperasf905.gq1.ygridcore.net%2C42849%2C1586129728118,2
>
> currentPath=hdfs://localhost:34203/user/jenkins/test-data/03854f9d-2780-eeaa-9645-c341240b62bf/WALs/asf905.gq1.ygridcore.net,42849,1586129728118/asf905.gq1.ygridcore.net%2C42849%2C1586129728118.1586129730509
> java.lang.RuntimeException: Thread is interrupted, the replication source may
> be terminated
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.interruptOrAbortWhenFail(ReplicationSourceManager.java:477)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:519)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.updateLogPosition(ReplicationSourceShipper.java:264)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.shipEdits(ReplicationSourceShipper.java:160)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.run(ReplicationSourceShipper.java:118)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24114) [Flakey Tests] TestSnapshotScannerHDFSAclController
[
https://issues.apache.org/jira/browse/HBASE-24114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-24114:
--
Fix Version/s: (was: 2.4.0)
(was: master)
3.0.0
> [Flakey Tests] TestSnapshotScannerHDFSAclController
> ---
>
> Key: HBASE-24114
> URL: https://issues.apache.org/jira/browse/HBASE-24114
> Project: HBase
> Issue Type: Test
> Components: acl
>Affects Versions: 2.3.0, master, 2.4.0
>Reporter: Huaxiang Sun
>Assignee: Hua Xiang
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> Still see the following error from the test.
> {code:java}
> ---
> Test set:
> org.apache.hadoop.hbase.security.access.TestSnapshotScannerHDFSAclController
> ---
> Tests run: 23, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 342.32 s <<<
> FAILURE! - in
> org.apache.hadoop.hbase.security.access.TestSnapshotScannerHDFSAclController
> org.apache.hadoop.hbase.security.access.TestSnapshotScannerHDFSAclController.testGrantTable
> Time elapsed: 6.82 s <<< FAILURE!
> java.lang.AssertionError: expected:<6> but was:<-1>
> at
> org.apache.hadoop.hbase.security.access.TestSnapshotScannerHDFSAclController.testGrantTable(TestSnapshotScannerHDFSAclController.java:349)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24143) [JDK11] Switch default garbage collector from CMS
[
https://issues.apache.org/jira/browse/HBASE-24143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078832#comment-17078832
]
Duo Zhang commented on HBASE-24143:
---
Let's change to G1.
> [JDK11] Switch default garbage collector from CMS
> -
>
> Key: HBASE-24143
> URL: https://issues.apache.org/jira/browse/HBASE-24143
> Project: HBase
> Issue Type: Sub-task
> Components: scripts
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Nick Dimiduk
>Priority: Major
>
> When running HBase tools on the cli, one of the warnings generated is
> {noformat}
> OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in
> version 9.0 and will likely be removed in a future release.
> {noformat}
> Java9+ use G1GC as the default collector. Maybe we simply omit GC
> configurations and use the default settings? Or someone has some suggested
> settings we can ship out of the box?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;…
Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;… URL: https://github.com/apache/hbase/pull/1461#issuecomment-611264702 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | ||| _ Patch Compile Tests _ | | +1 :green_heart: | shellcheck | 0m 2s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 10m 44s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 19s | The patch does not generate ASF License warnings. | | | | 18m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1461 | | Optional Tests | dupname asflicense shellcheck shelldocs hadoopcheck xml | | uname | Linux 6796ca787704 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 611c62f52a | | Max. process+thread count | 76 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) shellcheck=0.4.6 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #1292: HBASE-23994:Add WebUI to Canary
Apache9 commented on a change in pull request #1292: HBASE-23994:Add WebUI to
Canary
URL: https://github.com/apache/hbase/pull/1292#discussion_r405892777
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java
##
@@ -274,16 +303,50 @@ public void publishReadTiming(String znode, String
server, long msTime) {
private Map perTableReadLatency = new HashMap<>();
private LongAdder writeLatency = new LongAdder();
private final Map> regionMap = new
ConcurrentHashMap<>();
+private Map perServerFailuresCount = new
ConcurrentHashMap<>();
Review comment:
I mean change `Map` to `ConcurrentMap`, and also for the above `regionMap`. Not a question why you use
`ConcurrentHashMap`...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (HBASE-24134) Down forked JVM heap size from 2800m to 2200m for jdk8 and jdk11
[
https://issues.apache.org/jira/browse/HBASE-24134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078826#comment-17078826
]
Hudson commented on HBASE-24134:
Results for branch branch-2
[build #2590 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2590/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2590/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2590/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2590/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2590/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Down forked JVM heap size from 2800m to 2200m for jdk8 and jdk11
>
>
> Key: HBASE-24134
> URL: https://issues.apache.org/jira/browse/HBASE-24134
> Project: HBase
> Issue Type: Sub-task
>Reporter: Michael Stack
>Priority: Major
>
> Docker container we run tests in is hardcoded 20G. Our fork count is 0.25C
> currently. The jenkins machines are 16CPUs. 0.25C means 4CPUs so 4 forks
> running at any one time. Our current forked JVM heap size is 2800m if jdk8 or
> 3200m if jdk11. If we want to move our fork count to 0.5C, then that means 8
> forks possibly running at once: i.e. 8 * 2800 = 22.4G... i.e. too big for the
> container. This issue is about running w/ smaller forked jvm heap size. If
> 2200m then, its 19.6G which is cutting it close but might allow us sqeeze
> through (mvn takes 4G!). Lets see.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24131) [Flakey Tests] TestExportSnapshot takes too long; up against 13min max
[
https://issues.apache.org/jira/browse/HBASE-24131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078827#comment-17078827
]
Hudson commented on HBASE-24131:
Results for branch branch-2
[build #2590 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2590/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2590/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2590/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2590/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2590/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> [Flakey Tests] TestExportSnapshot takes too long; up against 13min max
> --
>
> Key: HBASE-24131
> URL: https://issues.apache.org/jira/browse/HBASE-24131
> Project: HBase
> Issue Type: Bug
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> TestExportSnapshot fails fairly regularly locally. Looking, its test timeout.
> Looking at how long it ran, its 13minutes plus. Looking at recent successful
> branch-2 run, it passed but it took about 7-8minutes. Let me break up the
> test into to pieces.
> org.junit.runners.model.TestTimedOutException: test timed out after 780
> seconds
>at
> org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:227)
>at
> org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportRetry(TestExportSnapshot.java:267)
> ... I see this in the log:
> > TEST TIMED OUT. PRINTING THREAD DUMP. <
> Test started at:
> 2020-04-06 17:19:21,739 INFO
> ... and the timestamp just above the TIMED OUT was
> 2020-04-06 17:31:01,758
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] saintstack commented on a change in pull request #1450: HBASE-24126 Up the container nproc uplimit from 10000 to 12500
saintstack commented on a change in pull request #1450: HBASE-24126 Up the
container nproc uplimit from 1 to 12500
URL: https://github.com/apache/hbase/pull/1450#discussion_r405890340
##
File path: dev-support/hbase_nightly_yetus.sh
##
@@ -66,7 +66,7 @@ YETUS_ARGS=("--branch=${BRANCH_NAME}" "${YETUS_ARGS[@]}")
YETUS_ARGS=("--tests-filter=${TESTS_FILTER}" "${YETUS_ARGS[@]}")
YETUS_ARGS=("--ignore-unknown-options=true" "${YETUS_ARGS[@]}")
# Why are these not being picked up from hbase-personality?
-YETUS_ARGS=("--proclimit=1" "${YETUS_ARGS[@]}")
+YETUS_ARGS=("--proclimit=12500" "${YETUS_ARGS[@]}")
Review comment:
Yeah. For now they needed. Let me file an issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1450: HBASE-24126 Up the container nproc uplimit from 10000 to 12500
saintstack commented on a change in pull request #1450: HBASE-24126 Up the
container nproc uplimit from 1 to 12500
URL: https://github.com/apache/hbase/pull/1450#discussion_r405890055
##
File path: dev-support/flaky-tests/flaky-reporting.Jenkinsfile
##
@@ -43,7 +43,7 @@ pipeline {
flaky_args=("${flaky_args[@]}" --urls
"${JENKINS_URL}/job/HBase%20Nightly/job/${BRANCH_NAME}" --is-yetus True
--max-builds 10)
flaky_args=("${flaky_args[@]}" --urls
"${JENKINS_URL}/job/HBase-Flaky-Tests/job/${BRANCH_NAME}" --is-yetus False
--max-builds 30)
docker build -t hbase-dev-support dev-support
- docker run -v "${WORKSPACE}":/hbase --workdir=/hbase
hbase-dev-support python dev-support/flaky-tests/report-flakies.py --mvn -v
"${flaky_args[@]}"
+ docker run --ulimit nproc=12500 -v "${WORKSPACE}":/hbase
--workdir=/hbase hbase-dev-support python
dev-support/flaky-tests/report-flakies.py --mvn -v "${flaky_args[@]}"
Review comment:
Yeah, one way to run all builds would be sweet.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1450: HBASE-24126 Up the container nproc uplimit from 10000 to 12500
saintstack commented on a change in pull request #1450: HBASE-24126 Up the container nproc uplimit from 1 to 12500 URL: https://github.com/apache/hbase/pull/1450#discussion_r405890250 ## File path: dev-support/hbase-personality.sh ## @@ -81,8 +81,11 @@ function personality_globals # Yetus 0.7.0 enforces limits. Default proclimit is 1000. # Up it. See HBASE-19902 for how we arrived at this number. + # NOTE: I don't think changing this has an effect. Set the + # --proclimit passed to yetus. This seems to do what we + # need changing proclimit. Review comment: Let me try filing an issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;…
Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;… URL: https://github.com/apache/hbase/pull/1461#issuecomment-611259883 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/2/console in case of problems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;…
Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;… URL: https://github.com/apache/hbase/pull/1461#issuecomment-611259789 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/2/console in case of problems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;…
Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;… URL: https://github.com/apache/hbase/pull/1461#issuecomment-611259804 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/2/console in case of problems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1461: HBASE-23779 Up the default fork count to make builds complete faster;…
saintstack commented on a change in pull request #1461: HBASE-23779 Up the
default fork count to make builds complete faster;…
URL: https://github.com/apache/hbase/pull/1461#discussion_r405887215
##
File path: dev-support/hbase-personality.sh
##
@@ -140,7 +140,15 @@ function personality_modules
clear_personality_queue
- extra="-DHBasePatchProcess"
+ # Set a fork count based off the host cpu count. Pass maven a -T argument.
+ # Default -T is one thread. 0.5C on an apache box of 16 cores and 2 jenkins
+ # 'executors' per host should make for 8 threads. Setting this here for yetus
+ # to pick up. See
+ #
https://yetus.apache.org/documentation/0.11.1/precommit-advanced/#global-definitions
+ # See below for more on -T:
+ #
https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3
+ forkcount="0.5C"
+ extra="-T${forkcount} -Dsurefire.firstPartForkCount=${forkcount}
-Dsurefire.secondPartForkCount=${forkcount} -DHBasePatchProcess"
Review comment:
I simplified the patch after the above. Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;…
Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;… URL: https://github.com/apache/hbase/pull/1461#issuecomment-611258387 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 56s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 49s | branch-2 passed | | +1 :green_heart: | compile | 2m 30s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 41s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 18s | root in branch-2 failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 33s | the patch passed | | +1 :green_heart: | compile | 2m 29s | the patch passed | | +1 :green_heart: | javac | 2m 29s | the patch passed | | +1 :green_heart: | shadedjars | 5m 29s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 16s | root in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 116m 20s | root in the patch failed. | | | | 146m 9s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1461 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9b5f2f07aad6 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 611c62f52a | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-root.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-root.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/1/testReport/ | | Max. process+thread count | 6625 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1461: HBASE-23779 Up the default fork count to make builds complete faster;…
saintstack commented on a change in pull request #1461: HBASE-23779 Up the
default fork count to make builds complete faster;…
URL: https://github.com/apache/hbase/pull/1461#discussion_r405883519
##
File path: dev-support/hbase-personality.sh
##
@@ -140,7 +140,15 @@ function personality_modules
clear_personality_queue
- extra="-DHBasePatchProcess"
+ # Set a fork count based off the host cpu count. Pass maven a -T argument.
+ # Default -T is one thread. 0.5C on an apache box of 16 cores and 2 jenkins
+ # 'executors' per host should make for 8 threads. Setting this here for yetus
+ # to pick up. See
+ #
https://yetus.apache.org/documentation/0.11.1/precommit-advanced/#global-definitions
+ # See below for more on -T:
+ #
https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3
+ forkcount="0.5C"
+ extra="-T${forkcount} -Dsurefire.firstPartForkCount=${forkcount}
-Dsurefire.secondPartForkCount=${forkcount} -DHBasePatchProcess"
Review comment:
Thanks for the review.
bq. How does maven --threads interact with surefire forks?
There is no correlation. There may be cases where it acts as a multiplier
where modules are independent of each other but generally, there is little
opportunity for -T benefit in hbase builds.
I went back and checked a few machines. Couldn't find instance where fork
count rose above forkcount limit. Doesn't mean it doesn't happen.
Let me untangle the two.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #1415: HBASE-24051 Allows indirect inheritance to CanUnbuffer
ndimiduk commented on a change in pull request #1415: HBASE-24051 Allows
indirect inheritance to CanUnbuffer
URL: https://github.com/apache/hbase/pull/1415#discussion_r405877469
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/FSDataInputStreamWrapper.java
##
@@ -331,22 +332,18 @@ public void unbuffer() {
if (this.instanceOfCanUnbuffer == null) {
// To ensure we compute whether the stream is instance of CanUnbuffer
only once.
this.instanceOfCanUnbuffer = false;
-Class[] streamInterfaces = streamClass.getInterfaces();
-for (Class c : streamInterfaces) {
- if
(c.getCanonicalName().toString().equals("org.apache.hadoop.fs.CanUnbuffer")) {
-try {
- this.unbuffer = streamClass.getDeclaredMethod("unbuffer");
-} catch (NoSuchMethodException | SecurityException e) {
- if (isLogTraceEnabled) {
-LOG.trace("Failed to find 'unbuffer' method in class " +
streamClass
-+ " . So there may be a TCP socket connection "
-+ "left open in CLOSE_WAIT state.", e);
- }
- return;
+if(wrappedStream instanceof CanUnbuffer){
Review comment:
nit: whitespace.
```java
if (...) {
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;…
Apache-HBase commented on issue #1461: HBASE-23779 Up the default fork count to make builds complete faster;… URL: https://github.com/apache/hbase/pull/1461#issuecomment-611254242 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 22s | branch-2 passed | | +1 :green_heart: | compile | 1m 57s | branch-2 passed | | +1 :green_heart: | shadedjars | 4m 36s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 22s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 4s | the patch passed | | +1 :green_heart: | compile | 1m 58s | the patch passed | | +1 :green_heart: | javac | 1m 58s | the patch passed | | +1 :green_heart: | shadedjars | 4m 39s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 14s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 104m 6s | root in the patch failed. | | | | 130m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1461 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a4b422329b3e 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 611c62f52a | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/1/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/1/testReport/ | | Max. process+thread count | 4309 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1461/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on issue #1440: HBASE-23113: Improve and add additional Netty configuration for RPC.
ndimiduk commented on issue #1440: HBASE-23113: Improve and add additional Netty configuration for RPC. URL: https://github.com/apache/hbase/pull/1440#issuecomment-611249408 Looks like this one supersedes #679 . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #1440: HBASE-23113: Improve and add additional Netty configuration for RPC.
ndimiduk commented on a change in pull request #1440: HBASE-23113: Improve and add additional Netty configuration for RPC. URL: https://github.com/apache/hbase/pull/1440#discussion_r405876189 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java ## @@ -67,20 +68,49 @@ * Tests may set this down from unlimited. */ public static final String HBASE_NETTY_EVENTLOOP_RPCSERVER_THREADCOUNT_KEY = -"hbase.netty.eventloop.rpcserver.thread.count"; + "hbase.netty.eventloop.rpcserver.thread.count"; private static final int EVENTLOOP_THREADCOUNT_DEFAULT = 0; + protected static final String SERVER_TCP_BACKLOG = "hbase.ipc.server.tcpbacklog"; + protected static final String SERVER_TCP_REUSEADDR = "hbase.ipc.server.tcpreuseaddr"; + protected static final String SERVER_TCP_NODELAY = "hbase.ipc.server.tcpnodelay"; + protected static final String SERVER_TCP_KEEPALIVE = "hbase.ipc.server.tcpkeepalive"; + + protected static final String SERVER_BUFFER_LOW_WATERMARK = "hbase.ipc.server.bufferlowwatermark"; + protected static final String SERVER_BUFFER_HIGH_WATERMARK = + "hbase.ipc.server.bufferhighwatermark"; + + protected static final boolean DEFAULT_SERVER_REUSEADDR = true; + protected static final int DEFAULT_SERVER_TCP_BACKLOG = 1024; Review comment: > The TCP_BACKLOG? Sorry, no. I had this on the wrong line. I mean `DEFAULT_SERVER_BUFFER_LOW_WATERMARK = 1024`. > If the number of queued bytes goes above the high mark, then channel.isWritable will return false until the number of outstanding bytes falls below the low water mark. Right. So I'm asking you (and I guess @SteNicholas as well): why do we need to flush out the buffer down almost completely before we continue receiving data? Is this assuming that HBase clients are generally writing large amounts of data, and this to avoid thrashing back and forth between threads as we fill, flush, fill, flush? A setting this low means we'd have to drain the buffer almost entirely before returning to the user thread. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #1450: HBASE-24126 Up the container nproc uplimit from 10000 to 12500
ndimiduk commented on a change in pull request #1450: HBASE-24126 Up the container nproc uplimit from 1 to 12500 URL: https://github.com/apache/hbase/pull/1450#discussion_r405874437 ## File path: dev-support/hbase-personality.sh ## @@ -81,8 +81,11 @@ function personality_globals # Yetus 0.7.0 enforces limits. Default proclimit is 1000. # Up it. See HBASE-19902 for how we arrived at this number. + # NOTE: I don't think changing this has an effect. Set the + # --proclimit passed to yetus. This seems to do what we + # need changing proclimit. Review comment: Ah, yeah. [In yetus](https://github.com/apache/yetus/blob/8b44644/precommit/src/main/shell/test-patch.sh#L96), the env variable is `PROC_LIMIT`. You found a bug :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #1450: HBASE-24126 Up the container nproc uplimit from 10000 to 12500
ndimiduk commented on a change in pull request #1450: HBASE-24126 Up the
container nproc uplimit from 1 to 12500
URL: https://github.com/apache/hbase/pull/1450#discussion_r405873034
##
File path: dev-support/hbase_docker.sh
##
@@ -159,4 +159,4 @@ done
echo "Successfully built ${IMAGE_NAME}."
Review comment:
Oh no, I didn't know this script existed. I probably broke it with the
improvements that went in with HBASE-23945.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #1450: HBASE-24126 Up the container nproc uplimit from 10000 to 12500
ndimiduk commented on a change in pull request #1450: HBASE-24126 Up the
container nproc uplimit from 1 to 12500
URL: https://github.com/apache/hbase/pull/1450#discussion_r405874635
##
File path: dev-support/hbase_nightly_yetus.sh
##
@@ -66,7 +66,7 @@ YETUS_ARGS=("--branch=${BRANCH_NAME}" "${YETUS_ARGS[@]}")
YETUS_ARGS=("--tests-filter=${TESTS_FILTER}" "${YETUS_ARGS[@]}")
YETUS_ARGS=("--ignore-unknown-options=true" "${YETUS_ARGS[@]}")
# Why are these not being picked up from hbase-personality?
-YETUS_ARGS=("--proclimit=1" "${YETUS_ARGS[@]}")
+YETUS_ARGS=("--proclimit=12500" "${YETUS_ARGS[@]}")
Review comment:
If the above change for `PROC_LIMIT` works, this line can be dropped.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #1450: HBASE-24126 Up the container nproc uplimit from 10000 to 12500
ndimiduk commented on a change in pull request #1450: HBASE-24126 Up the
container nproc uplimit from 1 to 12500
URL: https://github.com/apache/hbase/pull/1450#discussion_r405872153
##
File path: dev-support/flaky-tests/flaky-reporting.Jenkinsfile
##
@@ -43,7 +43,7 @@ pipeline {
flaky_args=("${flaky_args[@]}" --urls
"${JENKINS_URL}/job/HBase%20Nightly/job/${BRANCH_NAME}" --is-yetus True
--max-builds 10)
flaky_args=("${flaky_args[@]}" --urls
"${JENKINS_URL}/job/HBase-Flaky-Tests/job/${BRANCH_NAME}" --is-yetus False
--max-builds 30)
docker build -t hbase-dev-support dev-support
- docker run -v "${WORKSPACE}":/hbase --workdir=/hbase
hbase-dev-support python dev-support/flaky-tests/report-flakies.py --mvn -v
"${flaky_args[@]}"
+ docker run --ulimit nproc=12500 -v "${WORKSPACE}":/hbase
--workdir=/hbase hbase-dev-support python
dev-support/flaky-tests/report-flakies.py --mvn -v "${flaky_args[@]}"
Review comment:
Yikes. This is just hiding over here in the corner. TODO: update flaky
reporting to use yetus to run builds.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #1450: HBASE-24126 Up the container nproc uplimit from 10000 to 12500
ndimiduk commented on a change in pull request #1450: HBASE-24126 Up the
container nproc uplimit from 1 to 12500
URL: https://github.com/apache/hbase/pull/1450#discussion_r405874678
##
File path: dev-support/jenkins_precommit_github_yetus.sh
##
@@ -102,7 +102,7 @@ YETUS_ARGS+=("--reapermode=kill")
# set relatively high limits for ASF machines
# changing these to higher values may cause problems
# with other jobs on systemd-enabled machines
-YETUS_ARGS+=("--proclimit=1")
+YETUS_ARGS+=("--proclimit=12500")
Review comment:
likewise.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1462: HBASE-24148: Upgrade Thrift to 0.13.0
Apache-HBase commented on issue #1462: HBASE-24148: Upgrade Thrift to 0.13.0 URL: https://github.com/apache/hbase/pull/1462#issuecomment-611247379 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 44s | Maven dependency ordering for branch | | -1 :x: | mvninstall | 4m 16s | root in master failed. | | -0 :warning: | checkstyle | 2m 2s | The patch fails to run checkstyle in root | | -1 :x: | spotbugs | 7m 4s | root in master failed. | | -1 :x: | spotbugs | 0m 21s | hbase-mapreduce in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 1s | the patch passed | | -0 :warning: | checkstyle | 2m 19s | root: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 10m 22s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2. | | +1 :green_heart: | spotbugs | 13m 4s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 42s | The patch does not generate ASF License warnings. | | | | 55m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1462 | | JIRA Issue | HBASE-24148 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle xml | | uname | Linux 51ee3e438b13 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / eb7df0498c | | mvninstall | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/artifact/yetus-general-check/output/branch-mvninstall-root.txt | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/artifact/yetus-general-check/output//home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1462@2/yetus-general-check/output/maven-branch-checkstyle-root.txt | | spotbugs | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/artifact/yetus-general-check/output/branch-spotbugs-root.txt | | spotbugs | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/artifact/yetus-general-check/output/branch-spotbugs-hbase-mapreduce.txt | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/artifact/yetus-general-check/output/diff-checkstyle-root.txt | | Max. process+thread count | 93 (vs. ulimit of 1) | | modules | C: hbase-mapreduce hbase-thrift . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1462/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #1461: HBASE-23779 Up the default fork count to make builds complete faster;…
ndimiduk commented on a change in pull request #1461: HBASE-23779 Up the
default fork count to make builds complete faster;…
URL: https://github.com/apache/hbase/pull/1461#discussion_r405871037
##
File path: dev-support/hbase-personality.sh
##
@@ -140,7 +140,15 @@ function personality_modules
clear_personality_queue
- extra="-DHBasePatchProcess"
+ # Set a fork count based off the host cpu count. Pass maven a -T argument.
+ # Default -T is one thread. 0.5C on an apache box of 16 cores and 2 jenkins
+ # 'executors' per host should make for 8 threads. Setting this here for yetus
+ # to pick up. See
+ #
https://yetus.apache.org/documentation/0.11.1/precommit-advanced/#global-definitions
+ # See below for more on -T:
+ #
https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3
+ forkcount="0.5C"
+ extra="-T${forkcount} -Dsurefire.firstPartForkCount=${forkcount}
-Dsurefire.secondPartForkCount=${forkcount} -DHBasePatchProcess"
Review comment:
How does maven `--threads` interact with surefire forks? Isn't it the case
that we get `forkCount` maven process thread each building a module
simultaneously, and each of those modules is launching up to `forkCount`
concurrent test jvms? Means we're looking at 64 concurrent test processes?
Seems high to me.
If it is indeed launching like this, I would want `-T{forkCount}` when
running `compile/install`, but i want just a single core when running `test`.
I took a read through the [surefire
docs](https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html)
but I don't have a better handle on this than when I started.
If Maven is instead managing an upper-bound on concurrent test jvms at
`${forkcount}`, I say +1.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1463: HBASE-24136 Add release branch report to git-jira-release-audit tool
Apache-HBase commented on issue #1463: HBASE-24136 Add release branch report to git-jira-release-audit tool URL: https://github.com/apache/hbase/pull/1463#issuecomment-611244130 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 17s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for patch | | -0 :warning: | pylint | 0m 12s | The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | -0 :warning: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | ||| _ Other Tests _ | | +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? | | | | 4m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1463/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1463 | | Optional Tests | dupname asflicense pylint markdownlint | | uname | Linux f20c562b8082 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / eb7df0498c | | pylint | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1463/1/artifact/yetus-general-check/output/diff-patch-pylint.txt | | whitespace | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1463/1/artifact/yetus-general-check/output/whitespace-eol.txt | | Max. process+thread count | 47 (vs. ulimit of 1) | | modules | C: U: | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1463/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) pylint=2.4.4 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1463: HBASE-24136 Add release branch report to git-jira-release-audit tool
Apache-HBase commented on issue #1463: HBASE-24136 Add release branch report to git-jira-release-audit tool URL: https://github.com/apache/hbase/pull/1463#issuecomment-611244030 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 17s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for patch | ||| _ Other Tests _ | | | | 3m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1463/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1463 | | Optional Tests | | | uname | Linux 927ed4573d49 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / eb7df0498c | | Max. process+thread count | 47 (vs. ulimit of 1) | | modules | C: U: | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1463/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1463: HBASE-24136 Add release branch report to git-jira-release-audit tool
Apache-HBase commented on issue #1463: HBASE-24136 Add release branch report to git-jira-release-audit tool URL: https://github.com/apache/hbase/pull/1463#issuecomment-611243559 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for patch | ||| _ Other Tests _ | | | | 2m 15s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1463/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1463 | | Optional Tests | | | uname | Linux 0ea1460f8c74 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / eb7df0498c | | Max. process+thread count | 55 (vs. ulimit of 1) | | modules | C: U: | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1463/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work started] (HBASE-24136) Add release branch report to git-jira-release-audit tool
[
https://issues.apache.org/jira/browse/HBASE-24136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-24136 started by Nick Dimiduk.
> Add release branch report to git-jira-release-audit tool
>
>
> Key: HBASE-24136
> URL: https://issues.apache.org/jira/browse/HBASE-24136
> Project: HBase
> Issue Type: Improvement
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
>
> Update {{git-jira-release-audit}} to build a "what's new" report for a
> release branch (i.e., {{branch-2.3}}). Currently it only supports building
> such a report from a release line branch (i.e., {{branch-2}}).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ndimiduk opened a new pull request #1463: HBASE-24136 Add release branch report to git-jira-release-audit tool
ndimiduk opened a new pull request #1463: HBASE-24136 Add release branch report to git-jira-release-audit tool URL: https://github.com/apache/hbase/pull/1463 * enhancements to git_jira_release_audit.py * add aforementioned release branch report * include default values in help doc output * swap default db to a file on disk instead of in memory * set logger to match file name * add separate sql query log at DEBUG level * more detailed usage info in README.md, including example audit query * update entries in fallback_actions.csv This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-24149) Bump netty version to 4.1.48.Final
[ https://issues.apache.org/jira/browse/HBASE-24149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Penzes updated HBASE-24149: - Status: Patch Available (was: Open) > Bump netty version to 4.1.48.Final > -- > > Key: HBASE-24149 > URL: https://issues.apache.org/jira/browse/HBASE-24149 > Project: HBase > Issue Type: Improvement > Components: thirdparty >Affects Versions: thirdparty-3.2.0 >Reporter: Tamas Penzes >Assignee: Tamas Penzes >Priority: Major > > There is a new netty version to fix some bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase-thirdparty] tamaashu opened a new pull request #16: HBASE-24149: Bump netty version to 4.1.48.Final
tamaashu opened a new pull request #16: HBASE-24149: Bump netty version to 4.1.48.Final URL: https://github.com/apache/hbase-thirdparty/pull/16 Updated netty version to 4.1.48.Final This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1458: HBASE-24140 : Move CandidateGenerator and their implementors out of S…
Apache-HBase commented on issue #1458: HBASE-24140 : Move CandidateGenerator and their implementors out of S… URL: https://github.com/apache/hbase/pull/1458#issuecomment-611235972 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 22s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | -1 :x: | mvninstall | 5m 37s | root in master failed. | | +1 :green_heart: | compile | 1m 2s | master passed | | -1 :x: | shadedjars | 4m 52s | branch has 16 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | master passed | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 4m 49s | root in the patch failed. | | +1 :green_heart: | compile | 1m 19s | the patch passed | | +1 :green_heart: | javac | 1m 19s | the patch passed | | -1 :x: | shadedjars | 4m 41s | patch has 16 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 212m 35s | hbase-server in the patch passed. | | | | 239m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1458 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9def92518ad1 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / ac6060c3f9 | | Default Java | 1.8.0_232 | | mvninstall | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/2/artifact/yetus-jdk8-hadoop2-check/output/branch-mvninstall-root.txt | | shadedjars | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/2/artifact/yetus-jdk8-hadoop2-check/output/branch-shadedjars.txt | | mvninstall | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/2/artifact/yetus-jdk8-hadoop2-check/output/patch-mvninstall-root.txt | | shadedjars | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/2/artifact/yetus-jdk8-hadoop2-check/output/patch-shadedjars.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/2/testReport/ | | Max. process+thread count | 2541 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1458/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (HBASE-24149) Bump netty version to 4.1.48.Final
Tamas Penzes created HBASE-24149: Summary: Bump netty version to 4.1.48.Final Key: HBASE-24149 URL: https://issues.apache.org/jira/browse/HBASE-24149 Project: HBase Issue Type: Improvement Components: thirdparty Affects Versions: thirdparty-3.2.0 Reporter: Tamas Penzes Assignee: Tamas Penzes There is a new netty version to fix some bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24080) [flakey test] TestRegionReplicaFailover.testSecondaryRegionKill fails.
[
https://issues.apache.org/jira/browse/HBASE-24080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-24080:
-
Affects Version/s: (was: 2.4.0)
2.3.0
> [flakey test] TestRegionReplicaFailover.testSecondaryRegionKill fails.
> --
>
> Key: HBASE-24080
> URL: https://issues.apache.org/jira/browse/HBASE-24080
> Project: HBase
> Issue Type: Test
> Components: read replicas
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Huaxiang Sun
>Assignee: Hua Xiang
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> Run into the following error locally:
> {code:java}
> ---
> Test set: org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover
> ---
> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 97.391 s <<<
> FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover
> org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.testSecondaryRegionKill
> Time elapsed: 28.682 s <<< FAILURE!
> java.lang.AssertionError: Failed verification of row :0
> at org.junit.Assert.fail(Assert.java:89)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.verifyNumericRows(HBaseTestingUtility.java:2407)
> at
> org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.testSecondaryRegionKill(TestRegionReplicaFailover.java:240)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748) {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24080) [flakey test] TestRegionReplicaFailover.testSecondaryRegionKill fails.
[
https://issues.apache.org/jira/browse/HBASE-24080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-24080:
-
Fix Version/s: (was: 2.4.0)
> [flakey test] TestRegionReplicaFailover.testSecondaryRegionKill fails.
> --
>
> Key: HBASE-24080
> URL: https://issues.apache.org/jira/browse/HBASE-24080
> Project: HBase
> Issue Type: Test
> Components: read replicas
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Huaxiang Sun
>Assignee: Hua Xiang
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> Run into the following error locally:
> {code:java}
> ---
> Test set: org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover
> ---
> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 97.391 s <<<
> FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover
> org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.testSecondaryRegionKill
> Time elapsed: 28.682 s <<< FAILURE!
> java.lang.AssertionError: Failed verification of row :0
> at org.junit.Assert.fail(Assert.java:89)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.verifyNumericRows(HBaseTestingUtility.java:2407)
> at
> org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.testSecondaryRegionKill(TestRegionReplicaFailover.java:240)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748) {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24075) [Flakey Tests] teardown fails because JmxCacheBuster wants to read ClusterId from closed fs
[
https://issues.apache.org/jira/browse/HBASE-24075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-24075:
-
Fix Version/s: (was: 2.4.0)
> [Flakey Tests] teardown fails because JmxCacheBuster wants to read ClusterId
> from closed fs
> ---
>
> Key: HBASE-24075
> URL: https://issues.apache.org/jira/browse/HBASE-24075
> Project: HBase
> Issue Type: Test
> Components: flakies
>Reporter: Michael Stack
>Assignee: Bharath Vissapragada
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> I've seen this a few times. There may be an issue already but couldn't turn
> it up.
> In teardown, we fail because of below access:
> {code}
> 2020-03-28 22:01:47,484 WARN [HBase-Metrics2-1]
> master.CachedClusterId(105): Error fetching cluster ID
> java.io.IOException: Filesystem closed
>at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:468)
>at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1650)
>at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1523)
>at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520)
>at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520)
>at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1627)
>at org.apache.hadoop.hbase.util.FSUtils.getClusterId(FSUtils.java:578)
>at
> org.apache.hadoop.hbase.master.CachedClusterId.attemptFetch(CachedClusterId.java:103)
>at
> org.apache.hadoop.hbase.master.CachedClusterId.getFromCacheOrFetch(CachedClusterId.java:140)
>at org.apache.hadoop.hbase.master.HMaster.getClusterId(HMaster.java:3839)
>at
> org.apache.hadoop.hbase.master.MetricsMasterWrapperImpl.getClusterId(MetricsMasterWrapperImpl.java:68)
>at
> org.apache.hadoop.hbase.master.MetricsMasterSourceImpl.getMetrics(MetricsMasterSourceImpl.java:112)
>at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:200)
>at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:183)
>at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:156)
>at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassName(DefaultMBeanServerInterceptor.java:333)
>at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:319)
>at
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
>at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:66)
>at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:223)
>at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:101)
>at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:268)
>at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(MetricsSystemImpl.java:239)
>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>at java.lang.reflect.Method.invoke(Method.java:498)
>at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsSystemImpl.java:320)
>at com.sun.proxy.$Proxy13.postStart(Unknown Source)
>at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:193)
>at
> org.apache.hadoop.metrics2.impl.JmxCacheBuster$JmxCacheBusterRunnable.run(JmxCacheBuster.java:109)
>at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>at java.lang.Thread.run(Thread.java:748)
> {code}
> Its hadoop metrics system. Its still up trying to work.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24037) Add ut for root dir and wal root dir are different
[ https://issues.apache.org/jira/browse/HBASE-24037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-24037: - Fix Version/s: (was: 2.4.0) > Add ut for root dir and wal root dir are different > -- > > Key: HBASE-24037 > URL: https://issues.apache.org/jira/browse/HBASE-24037 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24099) Use a fair ReentrantReadWriteLock for the region close lock
[
https://issues.apache.org/jira/browse/HBASE-24099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078787#comment-17078787
]
Hudson commented on HBASE-24099:
Results for branch branch-1.3
[build #1172 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/1172/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/1172//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/1172//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/1172//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Use a fair ReentrantReadWriteLock for the region close lock
> ---
>
> Key: HBASE-24099
> URL: https://issues.apache.org/jira/browse/HBASE-24099
> Project: HBase
> Issue Type: Improvement
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 1.3.7, 1.7.0, 2.4.0, 2.1.10, 1.4.14, 2.2.5
>
> Attachments: ltt_results.pdf, pe_results.pdf, ycsb_results.pdf
>
>
> Consider creating the region's ReentrantReadWriteLock with the fair locking
> policy. We have had a couple of production incidents where a regionserver
> stalled in shutdown for a very very long time, leading to RIT (FAILED_CLOSE).
> The latest example is a 43 minute shutdown, ~40 minutes (2465280 ms) of that
> time was spent waiting to acquire the write lock on the region in order to
> finish closing it.
> {quote}
> ...
> Finished memstore flush of ~66.92 MB/70167112, currentsize=0 B/0 for region
> . in 927ms, sequenceid=6091133815, compaction requested=false at
> 1585175635349 (+60 ms)
> Disabling writes for close at 1585178100629 (+2465280 ms)
> {quote}
> This time was spent in between the memstore flush and the task status change
> "Disabling writes for close at...". This is at HRegion.java:1481 in 1.3.6:
> {code}
> 1480: // block waiting for the lock for closing
> 1481: lock.writeLock().lock(); // FindBugs: Complains
> UL_UNRELEASED_LOCK_EXCEPTION_PATH but seems fine
> {code}
>
> The close lock is operating in unfair mode. The table in question is under
> constant high query load. When the close request was received, there were
> active readers. After the close request there were more active readers,
> near-continuous contention. Although the clients would receive
> RegionServerStoppingException and other error notifications, because the
> region could not be reassigned, they kept coming, region (re-)location would
> find the region still hosted on the stuck server. Finally the closing thread
> waiting for the write lock became no longer starved (by chance) after 40
> minutes.
> The ReentrantReadWriteLock javadoc is clear about the possibility of
> starvation when continuously contended: "_When constructed as non-fair (the
> default), the order of entry to the read and write lock is unspecified,
> subject to reentrancy constraints. A nonfair lock that is continuously
> contended may indefinitely postpone one or more reader or writer threads, but
> will normally have higher throughput than a fair lock._"
> We could try changing the acquisition semantics of this lock to fair. This is
> a one line change, where we call the RW lock constructor. Then:
> "_When constructed as fair, threads contend for entry using an approximately
> arrival-order policy. When the currently held lock is released, either the
> longest-waiting single writer thread will be assigned the write lock, or if
> there is a group of reader threads waiting longer than all waiting writer
> threads, that group will be assigned the read lock._"
> This could be better. The close process will have to wait until all readers
> and writers already waiting for acquisition either acquire and release or go
> away but won't be starved by future/incoming requests.
> There could be a throughput loss in request handling, though, because this is
> the global reentrant RW lock for the region.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24033) Add ut for loading the corrupt recovered hfiles
[ https://issues.apache.org/jira/browse/HBASE-24033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-24033: - Fix Version/s: (was: 2.4.0) > Add ut for loading the corrupt recovered hfiles > --- > > Key: HBASE-24033 > URL: https://issues.apache.org/jira/browse/HBASE-24033 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24002) shadedjars check does not propagate --hadoop-profile
[
https://issues.apache.org/jira/browse/HBASE-24002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-24002:
-
Fix Version/s: (was: 2.4.0)
> shadedjars check does not propagate --hadoop-profile
>
>
> Key: HBASE-24002
> URL: https://issues.apache.org/jira/browse/HBASE-24002
> Project: HBase
> Issue Type: Task
> Components: build
>Affects Versions: 3.0.0, 2.3.0, 2.4.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> After HBASE-23829, we see the {{shadedjars}} check fail on JDK11 stages. From
> the log
> {noformat}
> Tue Mar 17 00:14:24 UTC 2020
> cd
> /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1296/yetus-jdk11-hadoop3-check/src
> /opt/maven/bin/mvn --batch-mode
> -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1296/yetus-m2/hbase-branch-2-patch-1
> clean verify -fae --batch-mode -pl
> hbase-shaded/hbase-shaded-check-invariants -am -Dtest=NoUnitTests
> -DHBasePatchProcess -Prelease -Dmaven.javadoc.skip=true
> -Dcheckstyle.skip=true -Dspotbugs.skip=true
> ...
> [INFO] --- maven-enforcer-plugin:3.0.0-M2:enforce (hadoop3-profile-required)
> @ hbase ---
> [INFO] Adding ignore: module-info
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.RequireProperty failed
> with message:
> HBase with JDK11 requires Hadoop3. Activate the profile with
> `-Dhadoop.profile=3.0`.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23977) [Flakey Tests] TestSlowLogRecorder.testOnlieSlowLogConsumption:178->confirmPayloadParams:97 expected: but was:
[
https://issues.apache.org/jira/browse/HBASE-23977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-23977:
-
Fix Version/s: (was: 2.4.0)
> [Flakey Tests]
> TestSlowLogRecorder.testOnlieSlowLogConsumption:178->confirmPayloadParams:97
> expected: but was:
> -
>
> Key: HBASE-23977
> URL: https://issues.apache.org/jira/browse/HBASE-23977
> Project: HBase
> Issue Type: Test
> Components: test
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Michael Stack
>Assignee: Viraj Jasani
>Priority: Major
> Labels: flaky-test
> Fix For: 3.0.0, 2.3.0
>
>
> I see this occasionally on a linux vm. [~vjasani] .. you might have an idea
> why this is going on (this is your test I believe and a recent addition). If
> you have stuff I can try, shout.
> Here is fail:
> {code}
> org.junit.ComparisonFailure: expected: but was:
>at
> org.apache.hadoop.hbase.regionserver.slowlog.TestSlowLogRecorder.confirmPayloadParams(TestSlowLogRecorder.java:97)
>at
> org.apache.hadoop.hbase.regionserver.slowlog.TestSlowLogRecorder.testOnlieSlowLogConsumption(TestSlowLogRecorder.java:178)
> {code}
> Here is log:
> {code}
> 2020-03-12 17:02:06,266 INFO [Time-limited test] hbase.ResourceChecker(179):
> after:
> regionserver.slowlog.TestSlowLogRecorder#testOnlineSlowLogWithDisableConfig
> Thread=7 (was 7), OpenFileDescriptor=226 (was 226),
> MaxFileDescriptor=131072 (was 131072), SystemLoadAverage=353 (was 353),
> ProcessCount=195 (was 195), AvailableMemoryMB=7037 (was 7027) -
> AvailableMemoryMB LEAK? -
> 2020-03-12 17:02:06,281 INFO [Time-limited test]
> hbase.ResourceChecker(151): before:
> regionserver.slowlog.TestSlowLogRecorder#testOnlieSlowLogConsumption
> Thread=7, OpenFileDescriptor=226, MaxFileDescriptor=131072,
> SystemLoadAverage=353, ProcessCount=195, AvailableMemoryMB=7036
> 2020-03-12 17:02:06,317 DEBUG [Time-limited test]
> slowlog.TestSlowLogRecorder(111): Initially ringbuffer of Slow Log records is
> empty
> 2020-03-12 17:02:06,326 INFO [Time-limited test] hbase.Waiter(183): Waiting
> up to [3,000] milli-secs(wait.for.ratio=[1])
> 2020-03-12 17:02:06,528 INFO [Time-limited test] hbase.Waiter(183): Waiting
> up to [3,000] milli-secs(wait.for.ratio=[1])
> 2020-03-12 17:02:06,630 INFO [Time-limited test] hbase.Waiter(183): Waiting
> up to [3,000] milli-secs(wait.for.ratio=[1])
> 2020-03-12 17:02:06,732 INFO [Time-limited test] hbase.Waiter(183): Waiting
> up to [3,000] milli-secs(wait.for.ratio=[1])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23937) Retrieve online large RPC logs
[ https://issues.apache.org/jira/browse/HBASE-23937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-23937: - Fix Version/s: (was: 2.4.0) > Retrieve online large RPC logs > -- > > Key: HBASE-23937 > URL: https://issues.apache.org/jira/browse/HBASE-23937 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0, 2.3.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > Similar to online slow RPC response logs, we should also provide get and > clear of large RPC logs which should be stored in the online RingBuffer. Many > of slow response logs would also turn out to be large response logs for > RegionServers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23936) Thrift support for get and clear slow_log APIs
[ https://issues.apache.org/jira/browse/HBASE-23936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-23936: - Fix Version/s: (was: 2.4.0) > Thrift support for get and clear slow_log APIs > -- > > Key: HBASE-23936 > URL: https://issues.apache.org/jira/browse/HBASE-23936 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0, 2.3.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > Provide thrift support for get_slowlog_responses() and > clear_slowlog_responses() Admin APIs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23741) Data loss when WAL split to HFile enabled
[ https://issues.apache.org/jira/browse/HBASE-23741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-23741: - Fix Version/s: (was: 2.4.0) > Data loss when WAL split to HFile enabled > - > > Key: HBASE-23741 > URL: https://issues.apache.org/jira/browse/HBASE-23741 > Project: HBase > Issue Type: Bug > Components: MTTR >Affects Versions: 3.0.0, 2.3.0 >Reporter: Pankaj Kumar >Assignee: Guanghao Zhang >Priority: Blocker > Fix For: 3.0.0, 2.3.0 > > > Very simple steps as below, > 1. Create table with 1 region > 2. Insert 1 record > 3. Flush the table > 4. Scan table and observe timestamp of the inserted row > 5. Insert same row key with same timestamp as previously inserted but with > different value > 6. Kill -9 RS where table region is online > 7. Start RS > Scan the table and check the result, latest cell must be returned. > Thanks [~sreenivasulureddy] for finding this issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
