[GitHub] [hbase] bsglz commented on issue #764: HBASE-23223 Support the offsetLock of bucketCache to use strong ref
bsglz commented on issue #764: HBASE-23223 Support the offsetLock of bucketCache to use strong ref URL: https://github.com/apache/hbase/pull/764#issuecomment-548672311 The ut failure seems not related to the patch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #764: HBASE-23223 Support the offsetLock of bucketCache to use strong ref
Apache-HBase commented on issue #764: HBASE-23223 Support the offsetLock of bucketCache to use strong ref URL: https://github.com/apache/hbase/pull/764#issuecomment-548670218 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 0m 33s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 1s | No case conflicting files found. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ master Compile Tests _ | | :green_heart: | mvninstall | 5m 23s | master passed | | :green_heart: | compile | 0m 58s | master passed | | :green_heart: | checkstyle | 1m 18s | master passed | | :green_heart: | shadedjars | 4m 34s | branch has no errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 0m 38s | master passed | | :blue_heart: | spotbugs | 4m 10s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :green_heart: | findbugs | 4m 9s | master passed | | :yellow_heart: | patch | 4m 18s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | :green_heart: | mvninstall | 5m 0s | the patch passed | | :green_heart: | compile | 0m 55s | the patch passed | | :green_heart: | javac | 0m 55s | the patch passed | | :green_heart: | checkstyle | 1m 18s | the patch passed | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :green_heart: | shadedjars | 4m 33s | patch has no errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 15m 39s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | :green_heart: | javadoc | 0m 36s | the patch passed | | :green_heart: | findbugs | 4m 16s | the patch passed | ||| _ Other Tests _ | | :broken_heart: | unit | 157m 9s | hbase-server in the patch failed. | | :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. | | | | 214m 4s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.client.TestBlockEvictionFromClient | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-764/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/764 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux cb6ae4768e09 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-764/out/precommit/personality/provided.sh | | git revision | master / ea5c572963 | | Default Java | 1.8.0_181 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-764/6/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-764/6/testReport/ | | Max. process+thread count | 4433 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-764/6/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-23240) branch-1 master and regionservers do not start when compiled against Hadoop 3.2.1
[ https://issues.apache.org/jira/browse/HBASE-23240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-23240: -- Affects Version/s: 1.5.0 > branch-1 master and regionservers do not start when compiled against Hadoop > 3.2.1 > - > > Key: HBASE-23240 > URL: https://issues.apache.org/jira/browse/HBASE-23240 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.5.0 >Reporter: Lars Hofhansl >Priority: Major > > Exception in thread "main" java.lang.NoSuchMethodError: > com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V > at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357) > at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338) > at org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1679) > at > org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:339) > at > org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:572) > at > org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:174) > at > org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:156) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23240) branch-1 master and regionservers do not start when compiled against Hadoop 3.2.1
[ https://issues.apache.org/jira/browse/HBASE-23240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-23240: -- Fix Version/s: 1.5.1 1.6.0 > branch-1 master and regionservers do not start when compiled against Hadoop > 3.2.1 > - > > Key: HBASE-23240 > URL: https://issues.apache.org/jira/browse/HBASE-23240 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.5.0 >Reporter: Lars Hofhansl >Priority: Major > Fix For: 1.6.0, 1.5.1 > > > Exception in thread "main" java.lang.NoSuchMethodError: > com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V > at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357) > at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338) > at org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1679) > at > org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:339) > at > org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:572) > at > org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:174) > at > org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:156) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23240) branch-1 master and regionservers do not start when compiled against Hadoop 3.2.1
Lars Hofhansl created HBASE-23240: - Summary: branch-1 master and regionservers do not start when compiled against Hadoop 3.2.1 Key: HBASE-23240 URL: https://issues.apache.org/jira/browse/HBASE-23240 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357) at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338) at org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1679) at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:339) at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:572) at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:174) at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:156) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23181) Blocked WAL archive: "LogRoller: Failed to schedule flush of XXXX, because it is not online on us"
[
https://issues.apache.org/jira/browse/HBASE-23181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964595#comment-16964595
]
Hudson commented on HBASE-23181:
Results for branch branch-2.1
[build #1696 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1696/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1696//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1696//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1696//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Blocked WAL archive: "LogRoller: Failed to schedule flush of , because it
> is not online on us"
> --
>
> Key: HBASE-23181
> URL: https://issues.apache.org/jira/browse/HBASE-23181
> Project: HBase
> Issue Type: Bug
> Components: regionserver, wal
>Affects Versions: 2.2.1
>Reporter: Michael Stack
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3
>
>
> On a heavily loaded cluster, WAL count keeps rising and we can get into a
> state where we are not rolling the logs off fast enough. In particular, there
> is this interesting state at the extreme where we pick a region to flush
> because 'Too many WALs' but the region is actually not online. As the WAL
> count rises, we keep picking a region-to-flush that is no longer on the
> server. This condition blocks our being able to clear WALs; eventually WALs
> climb into the hundreds and the RS goes zombie with a full Call queue that
> starts throwing CallQueueTooLargeExceptions (bad if this servers is the one
> carrying hbase:meta): i.e. clients fail to access the RegionServer.
> One symptom is a fast spike in WAL count for the RS. A restart of the RS will
> break the bind.
> Here is how it looks in the log:
> {code}
> # Here is region closing
> 2019-10-16 23:10:55,897 INFO
> org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler: Closed
> 8ee433ad59526778c53cc85ed3762d0b
>
> # Then soon after ...
> 2019-10-16 23:11:44,041 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
> Failed to schedule flush of 8ee433ad59526778c53cc85ed3762d0b, because it is
> not online on us
> 2019-10-16 23:11:45,006 INFO
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL: Too many WALs;
> count=45, max=32; forcing flush of 1 regions(s):
> 8ee433ad59526778c53cc85ed3762d0b
> ...
> # Later...
> 2019-10-16 23:20:25,427 INFO
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL: Too many WALs;
> count=542, max=32; forcing flush of 1 regions(s):
> 8ee433ad59526778c53cc85ed3762d0b
> 2019-10-16 23:20:25,427 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
> Failed to schedule flush of 8ee433ad59526778c53cc85ed3762d0b, because it is
> not online on us
> {code}
> I've seen this runaway WALs 2.2.1. I've seen runaway WALs in a 1.2.x version
> regularly that had HBASE-16721 fix in it, but can't say yet if it was for
> same reason as above.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23136) PartionedMobFileCompactor bulkloaded files shouldn't get replicated (addressing buklload replication related issue raised in HBASE-22380)
[
https://issues.apache.org/jira/browse/HBASE-23136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964596#comment-16964596
]
Hudson commented on HBASE-23136:
Results for branch branch-2.1
[build #1696 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1696/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1696//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1696//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1696//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> PartionedMobFileCompactor bulkloaded files shouldn't get replicated
> (addressing buklload replication related issue raised in HBASE-22380)
> -
>
> Key: HBASE-23136
> URL: https://issues.apache.org/jira/browse/HBASE-23136
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 3.0.0, 2.2.2, 2.1.8
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Critical
> Labels: bulkload, mob
> Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3
>
> Attachments: HBASE-23136.branch-2.1.patch,
> HBASE-23136.branch-2.2.patch, HBASE-23136.branch-2.patch
>
>
> Following bulkload replication fixes started in HBASE-22380, this addresses
> [~javaman_chen] observation regarding *PartitionedMobCompactor* and bulk
> loaded. As noted by [~javaman_chen], *PartitionedMobCompactor* uses bulkload
> feature to update resulting hfile into region hstore. This file, however,
> shouldn't get replicated in any condition. This PR adds required changes and
> extra test for this situation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23231) ReplicationSource do not update metrics after refresh
[
https://issues.apache.org/jira/browse/HBASE-23231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964593#comment-16964593
]
Hudson commented on HBASE-23231:
Results for branch branch-2.1
[build #1696 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1696/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1696//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1696//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1696//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> ReplicationSource do not update metrics after refresh
> -
>
> Key: HBASE-23231
> URL: https://issues.apache.org/jira/browse/HBASE-23231
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.2.2
>Reporter: Lijin Bin
>Assignee: Lijin Bin
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.2.2, 2.1.8
>
>
> When replication refresh to new state, it will create a new source and
> terminate the old source and replace the old source with new source.
> {code}
> public void refreshSources(String peerId) throws IOException {
> String terminateMessage = "Peer " + peerId +
> " state or config changed. Will close the previous replication source
> and open a new one";
> ReplicationPeer peer = replicationPeers.getPeer(peerId);
> ReplicationSourceInterface src = createSource(peerId, peer);
> // synchronized on latestPaths to avoid missing the new log
> synchronized (this.latestPaths) {
> ReplicationSourceInterface toRemove = this.sources.put(peerId, src);
> if (toRemove != null) {
> LOG.info("Terminate replication source for " + toRemove.getPeerId());
> toRemove.terminate(terminateMessage);
> }
> for (NavigableSet walsByGroup : walsById.get(peerId).values()) {
> walsByGroup.forEach(wal -> src.enqueueLog(new Path(this.logDir,
> wal)));
> }
> }
> LOG.info("Startup replication source for " + src.getPeerId());
> src.startup();
> {code}
> terminate replication source will remove all metrics, current terminate
> replication source be called after create new source which do init metrics,
> so the result is there is no corresponding metrics after refresh replication
> source.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23221) Polish the WAL interface after HBASE-23181
[
https://issues.apache.org/jira/browse/HBASE-23221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964594#comment-16964594
]
Hudson commented on HBASE-23221:
Results for branch branch-2.1
[build #1696 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1696/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1696//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1696//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1696//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Polish the WAL interface after HBASE-23181
> --
>
> Key: HBASE-23221
> URL: https://issues.apache.org/jira/browse/HBASE-23221
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, wal
>Reporter: Duo Zhang
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3
>
>
> We have a closeRegion flag which seems to be redundant with the marker
> WALEdit.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] zhaoyim commented on a change in pull request #746: HBASE-23195 FSDataInputStreamWrapper unbuffer can NOT invoke the clas…
zhaoyim commented on a change in pull request #746: HBASE-23195
FSDataInputStreamWrapper unbuffer can NOT invoke the clas…
URL: https://github.com/apache/hbase/pull/746#discussion_r341443996
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/FSDataInputStreamWrapper.java
##
@@ -270,22 +271,18 @@ public void unbuffer() {
if (this.instanceOfCanUnbuffer == null) {
// To ensure we compute whether the stream is instance of CanUnbuffer
only once.
this.instanceOfCanUnbuffer = false;
-Class[] streamInterfaces = streamClass.getInterfaces();
-for (Class c : streamInterfaces) {
- if
(c.getCanonicalName().toString().equals("org.apache.hadoop.fs.CanUnbuffer")) {
-try {
- this.unbuffer = streamClass.getDeclaredMethod("unbuffer");
-} catch (NoSuchMethodException | SecurityException e) {
- if (isLogTraceEnabled) {
-LOG.trace("Failed to find 'unbuffer' method in class " +
streamClass
-+ " . So there may be a TCP socket connection "
-+ "left open in CLOSE_WAIT state.", e);
- }
- return;
+if (CanUnbuffer.class.isAssignableFrom(streamClass)) {
Review comment:
NO, I think the logic is to confirm whether the stream implements the
CanUnbuffer. :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] bsglz commented on issue #764: HBASE-23223 Support the offsetLock of bucketCache to use strong ref
bsglz commented on issue #764: HBASE-23223 Support the offsetLock of bucketCache to use strong ref URL: https://github.com/apache/hbase/pull/764#issuecomment-548656028 Yeah, i am agree with you,if user modify the default config of bucketcache's blocksizes,the possible count of the offset maybe uncontrollable, so i make it configable,and the soft still as default. Consider in most time the count of offset is limited, so i think it is not a problem easily occur,maybe we can improve it in future. BTW,i have running with it in my live cluster than a week, with 32g heap,70g bucketCache and 16k blockSize,and seems good. Thanks for comment. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #746: HBASE-23195 FSDataInputStreamWrapper unbuffer can NOT invoke the clas…
Apache9 commented on a change in pull request #746: HBASE-23195
FSDataInputStreamWrapper unbuffer can NOT invoke the clas…
URL: https://github.com/apache/hbase/pull/746#discussion_r341443312
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/FSDataInputStreamWrapper.java
##
@@ -270,22 +271,18 @@ public void unbuffer() {
if (this.instanceOfCanUnbuffer == null) {
// To ensure we compute whether the stream is instance of CanUnbuffer
only once.
this.instanceOfCanUnbuffer = false;
-Class[] streamInterfaces = streamClass.getInterfaces();
-for (Class c : streamInterfaces) {
- if
(c.getCanonicalName().toString().equals("org.apache.hadoop.fs.CanUnbuffer")) {
-try {
- this.unbuffer = streamClass.getDeclaredMethod("unbuffer");
-} catch (NoSuchMethodException | SecurityException e) {
- if (isLogTraceEnabled) {
-LOG.trace("Failed to find 'unbuffer' method in class " +
streamClass
-+ " . So there may be a TCP socket connection "
-+ "left open in CLOSE_WAIT state.", e);
- }
- return;
+if (CanUnbuffer.class.isAssignableFrom(streamClass)) {
Review comment:
I think here we just want to confirm that it is a CanUnbuffer?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] zhaoyim commented on a change in pull request #746: HBASE-23195 FSDataInputStreamWrapper unbuffer can NOT invoke the clas…
zhaoyim commented on a change in pull request #746: HBASE-23195
FSDataInputStreamWrapper unbuffer can NOT invoke the clas…
URL: https://github.com/apache/hbase/pull/746#discussion_r341443171
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/FSDataInputStreamWrapper.java
##
@@ -270,39 +269,23 @@ public void unbuffer() {
if (this.instanceOfCanUnbuffer == null) {
// To ensure we compute whether the stream is instance of CanUnbuffer
only once.
this.instanceOfCanUnbuffer = false;
-Class[] streamInterfaces = streamClass.getInterfaces();
-for (Class c : streamInterfaces) {
- if
(c.getCanonicalName().toString().equals("org.apache.hadoop.fs.CanUnbuffer")) {
-try {
- this.unbuffer = streamClass.getDeclaredMethod("unbuffer");
-} catch (NoSuchMethodException | SecurityException e) {
- if (isLogTraceEnabled) {
-LOG.trace("Failed to find 'unbuffer' method in class " +
streamClass
-+ " . So there may be a TCP socket connection "
-+ "left open in CLOSE_WAIT state.", e);
- }
- return;
-}
-this.instanceOfCanUnbuffer = true;
-break;
- }
+if (CanUnbuffer.class.isAssignableFrom(streamClass)) {
+ this.unbuffer = (CanUnbuffer) wrappedStream;
+ this.instanceOfCanUnbuffer = true;
}
}
if (this.instanceOfCanUnbuffer) {
try {
- this.unbuffer.invoke(wrappedStream);
-} catch (IllegalAccessException | IllegalArgumentException |
InvocationTargetException e) {
+ this.unbuffer.unbuffer();
+} catch (UnsupportedOperationException e){
Review comment:
@Apache9 If a stream NOT implements CanUnbuffer the
`this.unbuffer.unbuffer();` will throw UnsupportedOperationException such as
BufferedFSInputStream. The easy way to check is remove the catch exception code
and run UT `org.apache.hadoop.hbase.io.hfile.TestHFile`, then you can get the
UnsupportedOperationException.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] zhaoyim commented on a change in pull request #746: HBASE-23195 FSDataInputStreamWrapper unbuffer can NOT invoke the clas…
zhaoyim commented on a change in pull request #746: HBASE-23195
FSDataInputStreamWrapper unbuffer can NOT invoke the clas…
URL: https://github.com/apache/hbase/pull/746#discussion_r341441861
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/FSDataInputStreamWrapper.java
##
@@ -270,22 +271,18 @@ public void unbuffer() {
if (this.instanceOfCanUnbuffer == null) {
// To ensure we compute whether the stream is instance of CanUnbuffer
only once.
this.instanceOfCanUnbuffer = false;
-Class[] streamInterfaces = streamClass.getInterfaces();
-for (Class c : streamInterfaces) {
- if
(c.getCanonicalName().toString().equals("org.apache.hadoop.fs.CanUnbuffer")) {
-try {
- this.unbuffer = streamClass.getDeclaredMethod("unbuffer");
-} catch (NoSuchMethodException | SecurityException e) {
- if (isLogTraceEnabled) {
-LOG.trace("Failed to find 'unbuffer' method in class " +
streamClass
-+ " . So there may be a TCP socket connection "
-+ "left open in CLOSE_WAIT state.", e);
- }
- return;
+if (CanUnbuffer.class.isAssignableFrom(streamClass)) {
Review comment:
@Apache9 Good question! Because here we cast the sub class to parent
class/interface, generally if want to check the up cast, we use
`isAssignableFrom`. Also here the logic we want to judge the Class inheritance
relationship, NOT the instance. Through use `(wrappedStream instanceof
CanUnbuffer)` should get the same results, but I prefer `isAssignableFrom`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #775: HBASE-23230 Enforce member visibility in HRegionServer
Apache-HBase commented on issue #775: HBASE-23230 Enforce member visibility in HRegionServer URL: https://github.com/apache/hbase/pull/775#issuecomment-548652201 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 1m 55s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 1s | No case conflicting files found. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :green_heart: | test4tests | 0m 0s | The patch appears to include 14 new or modified test files. | ||| _ master Compile Tests _ | | :green_heart: | mvninstall | 5m 31s | master passed | | :green_heart: | compile | 0m 57s | master passed | | :green_heart: | checkstyle | 1m 30s | master passed | | :green_heart: | shadedjars | 4m 39s | branch has no errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 0m 42s | master passed | | :blue_heart: | spotbugs | 4m 5s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :green_heart: | findbugs | 4m 2s | master passed | ||| _ Patch Compile Tests _ | | :green_heart: | mvninstall | 5m 13s | the patch passed | | :green_heart: | compile | 0m 58s | the patch passed | | :broken_heart: | javac | 0m 58s | hbase-server generated 1 new + 3 unchanged - 3 fixed = 4 total (was 6) | | :broken_heart: | checkstyle | 1m 29s | hbase-server: The patch generated 4 new + 266 unchanged - 41 fixed = 270 total (was 307) | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :green_heart: | shadedjars | 5m 24s | patch has no errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 16m 21s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | :green_heart: | javadoc | 0m 36s | the patch passed | | :green_heart: | findbugs | 4m 9s | the patch passed | ||| _ Other Tests _ | | :broken_heart: | unit | 159m 27s | hbase-server in the patch failed. | | :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. | | | | 219m 56s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.replication.TestReplicationDroppedTables | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-775/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/775 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 0206471d6a37 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-775/out/precommit/personality/provided.sh | | git revision | master / ea5c572963 | | Default Java | 1.8.0_181 | | javac | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-775/3/artifact/out/diff-compile-javac-hbase-server.txt | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-775/3/artifact/out/diff-checkstyle-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-775/3/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-775/3/testReport/ | | Max. process+thread count | 4590 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-775/3/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] binlijin commented on a change in pull request #775: HBASE-23230 Enforce member visibility in HRegionServer
binlijin commented on a change in pull request #775: HBASE-23230 Enforce member
visibility in HRegionServer
URL: https://github.com/apache/hbase/pull/775#discussion_r341439673
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
##
@@ -393,9 +393,13 @@ public MasterRpcServices(HMaster m) throws IOException {
}
@Override
- protected RpcServerInterface createRpcServer(Server server, Configuration
conf,
- RpcSchedulerFactory rpcSchedulerFactory, InetSocketAddress bindAddress,
String name)
- throws IOException {
+ protected RpcServerInterface createRpcServer(
Review comment:
OK
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (HBASE-23082) Backport low-latency snapshot tracking for space quotas to 2.x
[
https://issues.apache.org/jira/browse/HBASE-23082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964568#comment-16964568
]
HBase QA commented on HBASE-23082:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} prototool {color} | {color:blue} 0m
0s{color} | {color:blue} prototool was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 15 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
16s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
8s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
15s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
22s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
48s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 4m
26s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
56s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} The patch passed checkstyle in hbase-protocol-shaded
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch passed checkstyle in hbase-hadoop-compat
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} The patch passed checkstyle in hbase-hadoop2-compat
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} hbase-client: The patch generated 0 new + 2
unchanged - 1 fixed = 2 total (was 3) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
5s{color} | {color:green} hbase-server: The patch generated 0 new + 579
unchanged - 6 fixed = 579 total (was 585) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
31s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
19m 13s{color} | {color:green} Patch does not cause any errors with Hadoop
2.8.5 2.9.2 or 3.1.2. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
2m 21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} |
[GitHub] [hbase] binlijin commented on a change in pull request #775: HBASE-23230 Enforce member visibility in HRegionServer
binlijin commented on a change in pull request #775: HBASE-23230 Enforce member
visibility in HRegionServer
URL: https://github.com/apache/hbase/pull/775#discussion_r341438958
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
##
@@ -3807,10 +3688,14 @@ public SecureBulkLoadManager
getSecureBulkLoadManager() {
}
@Override
- public EntityLock regionLock(List regionInfos, String
description, Abortable abort)
- throws IOException {
-return new LockServiceClient(conf, lockStub,
asyncClusterConnection.getNonceGenerator())
- .regionLock(regionInfos, description, abort);
+ public EntityLock regionLock(
+ final List regionInfo,
+ final String description,
+ final Abortable abort
+ ) {
Review comment:
OK, i am also use the code formatter in
dev-support/hbase_eclipse_formatter.xml with intellij idea, but it does not
format the code into it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] chenxu14 commented on issue #764: HBASE-23223 Support the offsetLock of bucketCache to use strong ref
chenxu14 commented on issue #764: HBASE-23223 Support the offsetLock of bucketCache to use strong ref URL: https://github.com/apache/hbase/pull/764#issuecomment-548650427 If use strong reference obj pool, should we limit the capacity of the pool? otherwise, there will be the risk of OOM. maybe an StrongObjectPool with RingBuffer based is more suitable? just a suggestion. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] binlijin commented on a change in pull request #775: HBASE-23230 Enforce member visibility in HRegionServer
binlijin commented on a change in pull request #775: HBASE-23230 Enforce member
visibility in HRegionServer
URL: https://github.com/apache/hbase/pull/775#discussion_r341438578
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
##
@@ -1291,7 +1295,8 @@ protected RpcServerInterface createRpcServer(Server
server, Configuration conf,
}
protected Class getRpcSchedulerFactoryClass() {
-return
this.regionServer.conf.getClass(REGION_SERVER_RPC_SCHEDULER_FACTORY_CLASS,
Review comment:
Great.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] chenxu14 commented on a change in pull request #656: HBASE-23063 Add an option to enable multiget in parallel
chenxu14 commented on a change in pull request #656: HBASE-23063 Add an option to enable multiget in parallel URL: https://github.com/apache/hbase/pull/656#discussion_r341437527 ## File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java ## @@ -972,6 +972,17 @@ /** Whether nonces are enabled; default is true. */ public static final String HBASE_RS_NONCES_ENABLED = "hbase.regionserver.nonces.enabled"; + /** Some configuration param about multiget in parallel */ + public static final String HBASE_RS_PARALLEL_GET_ENABLED = + "hbase.regionserver.parallel.get.enabled"; Review comment: make sense This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] chenxu14 commented on issue #581: HBASE-22888 Share some stuffs with the initial reader when new stream reader created
chenxu14 commented on issue #581: HBASE-22888 Share some stuffs with the initial reader when new stream reader created URL: https://github.com/apache/hbase/pull/581#issuecomment-548647706 > Skimmed. This looks great. Thanks for your encouragement @saintstack ，this improve scan performance a lot in our prod env, hope the community can receive this patch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] busbey commented on a change in pull request #777: HBASE-23171 Ensure region state error messages include server making report (branch-2.1)
busbey commented on a change in pull request #777: HBASE-23171 Ensure region state error messages include server making report (branch-2.1) URL: https://github.com/apache/hbase/pull/777#discussion_r341427557 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java ## @@ -991,8 +991,8 @@ void checkOnlineRegionsReportForMeta(final ServerName serverName, final Set
[GitHub] [hbase] busbey commented on a change in pull request #777: HBASE-23171 Ensure region state error messages include server making report (branch-2.1)
busbey commented on a change in pull request #777: HBASE-23171 Ensure region state error messages include server making report (branch-2.1) URL: https://github.com/apache/hbase/pull/777#discussion_r341427366 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java ## @@ -991,8 +991,8 @@ void checkOnlineRegionsReportForMeta(final ServerName serverName, final Set
[GitHub] [hbase] bsglz commented on a change in pull request #764: HBASE-23223 Support the offsetLock of bucketCache to use strong ref
bsglz commented on a change in pull request #764: HBASE-23223 Support the offsetLock of bucketCache to use strong ref URL: https://github.com/apache/hbase/pull/764#discussion_r341425647 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/util/IdReadWriteLock.java ## @@ -18,13 +18,12 @@ */ package org.apache.hadoop.hbase.util; -import java.lang.ref.Reference; import java.util.concurrent.locks.ReentrantReadWriteLock; import org.apache.yetus.audience.InterfaceAudience; - import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting; + Review comment: Removed ,thx This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23175) Yarn unable to acquire delegation token for HBase Spark jobs
[
https://issues.apache.org/jira/browse/HBASE-23175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964518#comment-16964518
]
Hudson commented on HBASE-23175:
Results for branch branch-2
[build #2340 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Yarn unable to acquire delegation token for HBase Spark jobs
>
>
> Key: HBASE-23175
> URL: https://issues.apache.org/jira/browse/HBASE-23175
> Project: HBase
> Issue Type: Bug
> Components: security, spark
>Affects Versions: 2.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3
>
> Attachments: HBASE-23175.master.001.patch
>
>
> Spark rely on the TokenUtil.obtainToken(conf) API which is removed in
> HBase-2.0, though it has been fixed in SPARK-26432 to use the new API but
> planned for Spark-3.0, hence we need the fix in HBase until they release it
> and we upgrade it
> {code}
> 18/03/20 20:39:07 ERROR ApplicationMaster: User class threw exception:
> org.apache.hadoop.hbase.HBaseIOException:
> com.google.protobuf.ServiceException: Error calling method
> hbase.pb.AuthenticationService.GetAuthenticationToken
> org.apache.hadoop.hbase.HBaseIOException:
> com.google.protobuf.ServiceException: Error calling method
> hbase.pb.AuthenticationService.GetAuthenticationToken
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:360)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:346)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:86)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil$1.run(TokenUtil.java:121)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil$1.run(TokenUtil.java:118)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:313)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:118)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:272)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:533)
> at
> org.apache.hadoop.hbase.spark.HBaseContext.(HBaseContext.scala:73)
> at
> org.apache.hadoop.hbase.spark.JavaHBaseContext.(JavaHBaseContext.scala:46)
> at
> org.apache.hadoop.hbase.spark.example.hbasecontext.JavaHBaseBulkDeleteExample.main(JavaHBaseBulkDeleteExample.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:706)
> Caused by: com.google.protobuf.ServiceException: Error calling method
> hbase.pb.AuthenticationService.GetAuthenticationToken
> at
> org.apache.hadoop.hbase.client.SyncCoprocessorRpcChannel.callBlockingMethod(SyncCoprocessorRpcChannel.java:71)
> at
> org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService$BlockingStub.getAuthenticationToken(AuthenticationProtos.java:4512)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:81)
> ... 17 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23184) The HeapAllocation in WebUI is not accurate
[
https://issues.apache.org/jira/browse/HBASE-23184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964522#comment-16964522
]
Hudson commented on HBASE-23184:
Results for branch branch-2
[build #2340 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> The HeapAllocation in WebUI is not accurate
> ---
>
> Key: HBASE-23184
> URL: https://issues.apache.org/jira/browse/HBASE-23184
> Project: HBase
> Issue Type: Bug
> Components: UI
>Reporter: chenxu
>Assignee: chenxu
>Priority: Minor
> Fix For: 3.0.0, 2.3.0
>
>
> HeapAllocation in WebUI is always 0, the same reason as HBASE-22663
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23191) Log spams on Replication
[
https://issues.apache.org/jira/browse/HBASE-23191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964514#comment-16964514
]
Hudson commented on HBASE-23191:
Results for branch branch-2
[build #2340 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Log spams on Replication
>
>
> Key: HBASE-23191
> URL: https://issues.apache.org/jira/browse/HBASE-23191
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 3.0.0
>Reporter: Karthik Palanisamy
>Assignee: Karthik Palanisamy
>Priority: Trivial
> Fix For: 3.0.0, 2.3.0, 2.2.3
>
>
> If no new active writes in WAL edit, then *WALEntryStream#hasNext ->
> ReaderBase -> ProtobufLogReader#readNext* will reach end of file. It would be
> a good idea for changing the log level from INFO to DEBUG.
>
> {code:java}
> 2019-10-18 22:25:03,572 INFO
> [RS_REFRESH_PEER-regionserver/apache303:16020-0.replicationSource,p1hdp314.replicationSource.wal-reader.apache303.openstacklocal%2C16020%2C1571383146790,p1hdp314]
> wal.ProtobufLogReader: Reached the end of file at position 83
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23231) ReplicationSource do not update metrics after refresh
[
https://issues.apache.org/jira/browse/HBASE-23231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964519#comment-16964519
]
Hudson commented on HBASE-23231:
Results for branch branch-2
[build #2340 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> ReplicationSource do not update metrics after refresh
> -
>
> Key: HBASE-23231
> URL: https://issues.apache.org/jira/browse/HBASE-23231
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.2.2
>Reporter: Lijin Bin
>Assignee: Lijin Bin
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.2.2, 2.1.8
>
>
> When replication refresh to new state, it will create a new source and
> terminate the old source and replace the old source with new source.
> {code}
> public void refreshSources(String peerId) throws IOException {
> String terminateMessage = "Peer " + peerId +
> " state or config changed. Will close the previous replication source
> and open a new one";
> ReplicationPeer peer = replicationPeers.getPeer(peerId);
> ReplicationSourceInterface src = createSource(peerId, peer);
> // synchronized on latestPaths to avoid missing the new log
> synchronized (this.latestPaths) {
> ReplicationSourceInterface toRemove = this.sources.put(peerId, src);
> if (toRemove != null) {
> LOG.info("Terminate replication source for " + toRemove.getPeerId());
> toRemove.terminate(terminateMessage);
> }
> for (NavigableSet walsByGroup : walsById.get(peerId).values()) {
> walsByGroup.forEach(wal -> src.enqueueLog(new Path(this.logDir,
> wal)));
> }
> }
> LOG.info("Startup replication source for " + src.getPeerId());
> src.startup();
> {code}
> terminate replication source will remove all metrics, current terminate
> replication source be called after create new source which do init metrics,
> so the result is there is no corresponding metrics after refresh replication
> source.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22917) Proc-WAL roll fails always saying someone else has already created log
[
https://issues.apache.org/jira/browse/HBASE-22917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964515#comment-16964515
]
Hudson commented on HBASE-22917:
Results for branch branch-2
[build #2340 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Proc-WAL roll fails always saying someone else has already created log
> --
>
> Key: HBASE-22917
> URL: https://issues.apache.org/jira/browse/HBASE-22917
> Project: HBase
> Issue Type: Bug
> Components: proc-v2, wal
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 3.0.0, 2.3.0, 2.2.3
>
>
> Recently we met a weird scenario where Procedure WAL roll fails as it is
> already created by someone else.
> Later while going through the logs and code, observed that during Proc-WAL
> roll it failed to write the header. On failure file stream is just closed,
> {code}
> try {
> ProcedureWALFormat.writeHeader(newStream, header);
> startPos = newStream.getPos();
> } catch (IOException ioe) {
> LOG.warn("Encountered exception writing header", ioe);
> newStream.close();
> return false;
> }
> {code}
> Since we don't delete the corrupted file or increment the *flushLogId*, so on
> each retry it is trying to create the same *flushLogId* file. However Hmaster
> failover will resolve this issue, but we should handle it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23181) Blocked WAL archive: "LogRoller: Failed to schedule flush of XXXX, because it is not online on us"
[
https://issues.apache.org/jira/browse/HBASE-23181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964521#comment-16964521
]
Hudson commented on HBASE-23181:
Results for branch branch-2
[build #2340 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Blocked WAL archive: "LogRoller: Failed to schedule flush of , because it
> is not online on us"
> --
>
> Key: HBASE-23181
> URL: https://issues.apache.org/jira/browse/HBASE-23181
> Project: HBase
> Issue Type: Bug
> Components: regionserver, wal
>Affects Versions: 2.2.1
>Reporter: Michael Stack
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3
>
>
> On a heavily loaded cluster, WAL count keeps rising and we can get into a
> state where we are not rolling the logs off fast enough. In particular, there
> is this interesting state at the extreme where we pick a region to flush
> because 'Too many WALs' but the region is actually not online. As the WAL
> count rises, we keep picking a region-to-flush that is no longer on the
> server. This condition blocks our being able to clear WALs; eventually WALs
> climb into the hundreds and the RS goes zombie with a full Call queue that
> starts throwing CallQueueTooLargeExceptions (bad if this servers is the one
> carrying hbase:meta): i.e. clients fail to access the RegionServer.
> One symptom is a fast spike in WAL count for the RS. A restart of the RS will
> break the bind.
> Here is how it looks in the log:
> {code}
> # Here is region closing
> 2019-10-16 23:10:55,897 INFO
> org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler: Closed
> 8ee433ad59526778c53cc85ed3762d0b
>
> # Then soon after ...
> 2019-10-16 23:11:44,041 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
> Failed to schedule flush of 8ee433ad59526778c53cc85ed3762d0b, because it is
> not online on us
> 2019-10-16 23:11:45,006 INFO
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL: Too many WALs;
> count=45, max=32; forcing flush of 1 regions(s):
> 8ee433ad59526778c53cc85ed3762d0b
> ...
> # Later...
> 2019-10-16 23:20:25,427 INFO
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL: Too many WALs;
> count=542, max=32; forcing flush of 1 regions(s):
> 8ee433ad59526778c53cc85ed3762d0b
> 2019-10-16 23:20:25,427 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
> Failed to schedule flush of 8ee433ad59526778c53cc85ed3762d0b, because it is
> not online on us
> {code}
> I've seen this runaway WALs 2.2.1. I've seen runaway WALs in a 1.2.x version
> regularly that had HBASE-16721 fix in it, but can't say yet if it was for
> same reason as above.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23192) CatalogJanitor consistencyCheck does not log problematic row on exception
[
https://issues.apache.org/jira/browse/HBASE-23192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964516#comment-16964516
]
Hudson commented on HBASE-23192:
Results for branch branch-2
[build #2340 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> CatalogJanitor consistencyCheck does not log problematic row on exception
> -
>
> Key: HBASE-23192
> URL: https://issues.apache.org/jira/browse/HBASE-23192
> Project: HBase
> Issue Type: Bug
> Components: hbck2
>Affects Versions: 2.1.7, 2.2.2
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Minor
> Fix For: 3.0.0, 2.3.0, 2.3.3
>
>
> Small stuff. Trying to fix a cluser, cleared an info:server field. Damaged
> hbase:meta for CatalogJanitor; when it should have just logged and skipped
> the bad entity when doing consistency check, instead CJ crashed. Also doesn't
> log bad raw which would help debugging.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22739) ArrayIndexOutOfBoundsException when balance
[
https://issues.apache.org/jira/browse/HBASE-22739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964517#comment-16964517
]
Hudson commented on HBASE-22739:
Results for branch branch-2
[build #2340 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> ArrayIndexOutOfBoundsException when balance
> ---
>
> Key: HBASE-22739
> URL: https://issues.apache.org/jira/browse/HBASE-22739
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Reporter: casuallc
>Assignee: Lijin Bin
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3
>
>
>
> {code:java}
> 2019-07-25 15:19:59,828 ERROR [master/nna:16000.Chore.1]
> hbase.ScheduledChore: Caught error
> java.lang.ArrayIndexOutOfBoundsException: 3171
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.removeRegion(BaseLoadBalancer.java:873)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.doAction(BaseLoadBalancer.java:716)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:407)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:318)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1650)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1567)
> at
> org.apache.hadoop.hbase.master.balancer.BalancerChore.chore(BalancerChore.java:49)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> should check if the regionIndex is valid when removeRegion,
> java:
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
> {code:java}
> int[] removeRegion(int[] regions, int regionIndex) {
> //TODO: this maybe costly. Consider using linked lists
> int[] newRegions = new int[regions.length - 1];
> int i = 0;
> for (i = 0; i < regions.length; i++) {
> if (regions[i] == regionIndex) {
> break;
> }
> if (i == regions.length - 1) {
> return Arrays.copyOf(regions, regions.length);
> }
> newRegions[i] = regions[i];
> }
> System.arraycopy(regions, i+1, newRegions, i, newRegions.length - i);
> return newRegions;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23221) Polish the WAL interface after HBASE-23181
[
https://issues.apache.org/jira/browse/HBASE-23221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964520#comment-16964520
]
Hudson commented on HBASE-23221:
Results for branch branch-2
[build #2340 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2340//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Polish the WAL interface after HBASE-23181
> --
>
> Key: HBASE-23221
> URL: https://issues.apache.org/jira/browse/HBASE-23221
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, wal
>Reporter: Duo Zhang
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3
>
>
> We have a closeRegion flag which seems to be redundant with the marker
> WALEdit.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23191) Log spams on Replication
[
https://issues.apache.org/jira/browse/HBASE-23191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964505#comment-16964505
]
Hudson commented on HBASE-23191:
Results for branch branch-2.2
[build #679 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Log spams on Replication
>
>
> Key: HBASE-23191
> URL: https://issues.apache.org/jira/browse/HBASE-23191
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 3.0.0
>Reporter: Karthik Palanisamy
>Assignee: Karthik Palanisamy
>Priority: Trivial
> Fix For: 3.0.0, 2.3.0, 2.2.3
>
>
> If no new active writes in WAL edit, then *WALEntryStream#hasNext ->
> ReaderBase -> ProtobufLogReader#readNext* will reach end of file. It would be
> a good idea for changing the log level from INFO to DEBUG.
>
> {code:java}
> 2019-10-18 22:25:03,572 INFO
> [RS_REFRESH_PEER-regionserver/apache303:16020-0.replicationSource,p1hdp314.replicationSource.wal-reader.apache303.openstacklocal%2C16020%2C1571383146790,p1hdp314]
> wal.ProtobufLogReader: Reached the end of file at position 83
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23221) Polish the WAL interface after HBASE-23181
[
https://issues.apache.org/jira/browse/HBASE-23221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964509#comment-16964509
]
Hudson commented on HBASE-23221:
Results for branch branch-2.2
[build #679 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Polish the WAL interface after HBASE-23181
> --
>
> Key: HBASE-23221
> URL: https://issues.apache.org/jira/browse/HBASE-23221
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, wal
>Reporter: Duo Zhang
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3
>
>
> We have a closeRegion flag which seems to be redundant with the marker
> WALEdit.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22917) Proc-WAL roll fails always saying someone else has already created log
[
https://issues.apache.org/jira/browse/HBASE-22917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964506#comment-16964506
]
Hudson commented on HBASE-22917:
Results for branch branch-2.2
[build #679 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Proc-WAL roll fails always saying someone else has already created log
> --
>
> Key: HBASE-22917
> URL: https://issues.apache.org/jira/browse/HBASE-22917
> Project: HBase
> Issue Type: Bug
> Components: proc-v2, wal
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 3.0.0, 2.3.0, 2.2.3
>
>
> Recently we met a weird scenario where Procedure WAL roll fails as it is
> already created by someone else.
> Later while going through the logs and code, observed that during Proc-WAL
> roll it failed to write the header. On failure file stream is just closed,
> {code}
> try {
> ProcedureWALFormat.writeHeader(newStream, header);
> startPos = newStream.getPos();
> } catch (IOException ioe) {
> LOG.warn("Encountered exception writing header", ioe);
> newStream.close();
> return false;
> }
> {code}
> Since we don't delete the corrupted file or increment the *flushLogId*, so on
> each retry it is trying to create the same *flushLogId* file. However Hmaster
> failover will resolve this issue, but we should handle it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23181) Blocked WAL archive: "LogRoller: Failed to schedule flush of XXXX, because it is not online on us"
[
https://issues.apache.org/jira/browse/HBASE-23181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964510#comment-16964510
]
Hudson commented on HBASE-23181:
Results for branch branch-2.2
[build #679 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Blocked WAL archive: "LogRoller: Failed to schedule flush of , because it
> is not online on us"
> --
>
> Key: HBASE-23181
> URL: https://issues.apache.org/jira/browse/HBASE-23181
> Project: HBase
> Issue Type: Bug
> Components: regionserver, wal
>Affects Versions: 2.2.1
>Reporter: Michael Stack
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3
>
>
> On a heavily loaded cluster, WAL count keeps rising and we can get into a
> state where we are not rolling the logs off fast enough. In particular, there
> is this interesting state at the extreme where we pick a region to flush
> because 'Too many WALs' but the region is actually not online. As the WAL
> count rises, we keep picking a region-to-flush that is no longer on the
> server. This condition blocks our being able to clear WALs; eventually WALs
> climb into the hundreds and the RS goes zombie with a full Call queue that
> starts throwing CallQueueTooLargeExceptions (bad if this servers is the one
> carrying hbase:meta): i.e. clients fail to access the RegionServer.
> One symptom is a fast spike in WAL count for the RS. A restart of the RS will
> break the bind.
> Here is how it looks in the log:
> {code}
> # Here is region closing
> 2019-10-16 23:10:55,897 INFO
> org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler: Closed
> 8ee433ad59526778c53cc85ed3762d0b
>
> # Then soon after ...
> 2019-10-16 23:11:44,041 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
> Failed to schedule flush of 8ee433ad59526778c53cc85ed3762d0b, because it is
> not online on us
> 2019-10-16 23:11:45,006 INFO
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL: Too many WALs;
> count=45, max=32; forcing flush of 1 regions(s):
> 8ee433ad59526778c53cc85ed3762d0b
> ...
> # Later...
> 2019-10-16 23:20:25,427 INFO
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL: Too many WALs;
> count=542, max=32; forcing flush of 1 regions(s):
> 8ee433ad59526778c53cc85ed3762d0b
> 2019-10-16 23:20:25,427 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
> Failed to schedule flush of 8ee433ad59526778c53cc85ed3762d0b, because it is
> not online on us
> {code}
> I've seen this runaway WALs 2.2.1. I've seen runaway WALs in a 1.2.x version
> regularly that had HBASE-16721 fix in it, but can't say yet if it was for
> same reason as above.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23199) Error populating Table-Attribute fields
[
https://issues.apache.org/jira/browse/HBASE-23199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964512#comment-16964512
]
Hudson commented on HBASE-23199:
Results for branch branch-2.2
[build #679 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Error populating Table-Attribute fields
> ---
>
> Key: HBASE-23199
> URL: https://issues.apache.org/jira/browse/HBASE-23199
> Project: HBase
> Issue Type: Bug
> Components: master, UI
>Affects Versions: 3.0.0
>Reporter: Karthik Palanisamy
>Assignee: Karthik Palanisamy
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.2.3
>
> Attachments: Screen Shot 2019-10-21 at 3.25.40 PM.png
>
>
> if quota is enabled, then we fetch table and namespace quota info. It is not
> necessary both table and namespace will have a quota set. Sometimes, users
> only have table level quota or namespace level quota or both un-set. So we
> must add Quota "null" check before getting Throttle info(Limit, Type,
> TimeUnit, Scope).
>
> !Screen Shot 2019-10-21 at 3.25.40 PM.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23231) ReplicationSource do not update metrics after refresh
[
https://issues.apache.org/jira/browse/HBASE-23231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964511#comment-16964511
]
Hudson commented on HBASE-23231:
Results for branch branch-2.2
[build #679 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> ReplicationSource do not update metrics after refresh
> -
>
> Key: HBASE-23231
> URL: https://issues.apache.org/jira/browse/HBASE-23231
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.2.2
>Reporter: Lijin Bin
>Assignee: Lijin Bin
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.2.2, 2.1.8
>
>
> When replication refresh to new state, it will create a new source and
> terminate the old source and replace the old source with new source.
> {code}
> public void refreshSources(String peerId) throws IOException {
> String terminateMessage = "Peer " + peerId +
> " state or config changed. Will close the previous replication source
> and open a new one";
> ReplicationPeer peer = replicationPeers.getPeer(peerId);
> ReplicationSourceInterface src = createSource(peerId, peer);
> // synchronized on latestPaths to avoid missing the new log
> synchronized (this.latestPaths) {
> ReplicationSourceInterface toRemove = this.sources.put(peerId, src);
> if (toRemove != null) {
> LOG.info("Terminate replication source for " + toRemove.getPeerId());
> toRemove.terminate(terminateMessage);
> }
> for (NavigableSet walsByGroup : walsById.get(peerId).values()) {
> walsByGroup.forEach(wal -> src.enqueueLog(new Path(this.logDir,
> wal)));
> }
> }
> LOG.info("Startup replication source for " + src.getPeerId());
> src.startup();
> {code}
> terminate replication source will remove all metrics, current terminate
> replication source be called after create new source which do init metrics,
> so the result is there is no corresponding metrics after refresh replication
> source.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23192) CatalogJanitor consistencyCheck does not log problematic row on exception
[
https://issues.apache.org/jira/browse/HBASE-23192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964507#comment-16964507
]
Hudson commented on HBASE-23192:
Results for branch branch-2.2
[build #679 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> CatalogJanitor consistencyCheck does not log problematic row on exception
> -
>
> Key: HBASE-23192
> URL: https://issues.apache.org/jira/browse/HBASE-23192
> Project: HBase
> Issue Type: Bug
> Components: hbck2
>Affects Versions: 2.1.7, 2.2.2
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Minor
> Fix For: 3.0.0, 2.3.0, 2.3.3
>
>
> Small stuff. Trying to fix a cluser, cleared an info:server field. Damaged
> hbase:meta for CatalogJanitor; when it should have just logged and skipped
> the bad entity when doing consistency check, instead CJ crashed. Also doesn't
> log bad raw which would help debugging.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22739) ArrayIndexOutOfBoundsException when balance
[
https://issues.apache.org/jira/browse/HBASE-22739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964508#comment-16964508
]
Hudson commented on HBASE-22739:
Results for branch branch-2.2
[build #679 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> ArrayIndexOutOfBoundsException when balance
> ---
>
> Key: HBASE-22739
> URL: https://issues.apache.org/jira/browse/HBASE-22739
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Reporter: casuallc
>Assignee: Lijin Bin
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3
>
>
>
> {code:java}
> 2019-07-25 15:19:59,828 ERROR [master/nna:16000.Chore.1]
> hbase.ScheduledChore: Caught error
> java.lang.ArrayIndexOutOfBoundsException: 3171
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.removeRegion(BaseLoadBalancer.java:873)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.doAction(BaseLoadBalancer.java:716)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:407)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:318)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1650)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1567)
> at
> org.apache.hadoop.hbase.master.balancer.BalancerChore.chore(BalancerChore.java:49)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> should check if the regionIndex is valid when removeRegion,
> java:
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
> {code:java}
> int[] removeRegion(int[] regions, int regionIndex) {
> //TODO: this maybe costly. Consider using linked lists
> int[] newRegions = new int[regions.length - 1];
> int i = 0;
> for (i = 0; i < regions.length; i++) {
> if (regions[i] == regionIndex) {
> break;
> }
> if (i == regions.length - 1) {
> return Arrays.copyOf(regions, regions.length);
> }
> newRegions[i] = regions[i];
> }
> System.arraycopy(regions, i+1, newRegions, i, newRegions.length - i);
> return newRegions;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-20827) Add pause when retrying after CallQueueTooBigException for reportRegionStateTransition
[
https://issues.apache.org/jira/browse/HBASE-20827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964504#comment-16964504
]
Hudson commented on HBASE-20827:
Results for branch branch-2.2
[build #679 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/679//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add pause when retrying after CallQueueTooBigException for
> reportRegionStateTransition
> ---
>
> Key: HBASE-20827
> URL: https://issues.apache.org/jira/browse/HBASE-20827
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3
>
> Attachments: HBASE-20827.master.001.patch, HBASE-20827.patch
>
>
> We should be adding a pause during retry of CallQueueTooBigException to avoid
> flooding master.
> {code}
> org.apache.hadoop.hbase.CallQueueTooBigException: Call queue is full on
> ctr-e138-1518143905142-387782-01-03.hwx.site,2,1530317245246, too
> many items queued ?
> at sun.reflect.GeneratedConstructorAccessor108.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100)
> at
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:359)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:336)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportRegionStateTransition(HRegionServer.java:2259)
> at
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:120)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException):
> Call queue is full on
> ctr-e138-1518143905142-387782-01-03.hwx.site,2,1530317245246, too
> many items queued ?
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:387)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)
> at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)
> at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:161)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:191)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
>
[jira] [Commented] (HBASE-23238) Additional test and checks for null references on ScannerCallableWithReplicas
[
https://issues.apache.org/jira/browse/HBASE-23238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964495#comment-16964495
]
HBase QA commented on HBASE-23238:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
29s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
4s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
37s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
36s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
45s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
26s{color} | {color:red} hbase-server: The patch generated 1 new + 19 unchanged
- 0 fixed = 20 total (was 19) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
34s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
17m 6s{color} | {color:green} Patch does not cause any errors with Hadoop
2.8.5 2.9.2 or 3.1.2. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
30s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}310m 42s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
58s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}379m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.client.TestSnapshotTemporaryDirectoryWithRegionReplicas |
| | hadoop.hbase.client.TestFromClientSideWithCoprocessor |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 base:
https://builds.apache.org/job/PreCommit-HBASE-Build/984/artifact/patchprocess/Dockerfile
|
| JIRA Issue | HBASE-23238 |
| JIRA Patch URL |
[jira] [Commented] (HBASE-23219) Re-enable ZKLess tests for branch-1 (Revert HBASE-14622)
[ https://issues.apache.org/jira/browse/HBASE-23219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964446#comment-16964446 ] Andrew Kyle Purtell commented on HBASE-23219: - Thank you for submitting these patches. Looking now. > Re-enable ZKLess tests for branch-1 (Revert HBASE-14622) > > > Key: HBASE-23219 > URL: https://issues.apache.org/jira/browse/HBASE-23219 > Project: HBase > Issue Type: Task > Components: test >Affects Versions: 1.3.6 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Trivial > Fix For: 1.4.12, 1.3.7 > > Attachments: HBASE-23219.branch-1.001.patch, > HBASE-23219.branch-1.3.001.patch, HBASE-23219.branch-1.4.001.patch > > > Since we are using zkless in our production setup, we would like to enable > these tests back in apache on branch-1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23082) Backport low-latency snapshot tracking for space quotas to 2.x
[ https://issues.apache.org/jira/browse/HBASE-23082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-23082: --- Attachment: HBASE-23082.004.branch-2.patch > Backport low-latency snapshot tracking for space quotas to 2.x > -- > > Key: HBASE-23082 > URL: https://issues.apache.org/jira/browse/HBASE-23082 > Project: HBase > Issue Type: Improvement > Components: Quotas >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 2.3.0, 2.1.8, 2.2.3 > > Attachments: HBASE-23082.001.patch, HBASE-23082.002.branch-2.patch, > HBASE-23082.002.patch, HBASE-23082.003.branch-2.patch, > HBASE-23082.004.branch-2.patch, HBASE-23082.branch-2.1.002.patch > > > Some rebase'ing at $dayjob found that HBASE-18133 and HBASE-18135 never made > it to any branch-2's. > There's no good reason that I can think of as to why we shouldn't backport > these (will make quotas and snapshots much more accurate). As memory serves > me, they just were landing when we were branching for early 2.0.0 releases. > Let's get backports for HBASE-18133, HBASE-18135, and HBASE-20531 (and > implicitly HBASE-20439 and HBASE-20440) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23185) High cpu usage because getTable()#put() gets config value every time
[ https://issues.apache.org/jira/browse/HBASE-23185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964421#comment-16964421 ] Michael Stack commented on HBASE-23185: --- I merged the commit. WIll check nightly to see if the two tests Sean identified start failing again. Thanks for the fix [~lineyshinya] > High cpu usage because getTable()#put() gets config value every time > > > Key: HBASE-23185 > URL: https://issues.apache.org/jira/browse/HBASE-23185 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 1.5.0, 1.4.10, 1.2.12, 1.3.5 >Reporter: Shinya Yoshida >Assignee: Shinya Yoshida >Priority: Major > Labels: performance > Fix For: 1.6.0 > > Attachments: Screenshot from 2019-10-18 12-38-14.png, Screenshot from > 2019-10-18 13-03-24.png > > > When we analyzed the performance of our hbase application with many puts, we > found that Configuration methods use many CPU resources: > !Screenshot from 2019-10-18 12-38-14.png|width=460,height=205! > As you can see, getTable().put() is calling Configuration methods which cause > regex or synchronization by Hashtable. > This should not happen in 0.99.2 because > https://issues.apache.org/jira/browse/HBASE-12128 addressed such an issue. > However, it's reproducing nowadays by bugs or leakages after many code > evoluations between 0.9x and 1.x. > # > [https://github.com/apache/hbase/blob/dd9eadb00f9dcd071a246482a11dfc7d63845f00/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java#L369-L374] > ** finishSetup is called every new HTable() e.g. every con.getTable() > ** So getInt is called everytime and it does regex > # > [https://github.com/apache/hbase/blob/dd9eadb00f9dcd071a246482a11dfc7d63845f00/hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java#L115] > ** BufferedMutatorImpl is created every first put for HTable e.g. > con.getTable().put() > ** Create ConnectionConf every time in BufferedMutatorImpl constructor > ** ConnectionConf gets config value in the constructor > # > [https://github.com/apache/hbase/blob/dd9eadb00f9dcd071a246482a11dfc7d63845f00/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java#L326] > ** AsyncProcess is created in BufferedMutatorImpl constructor, so new > AsyncProcess is created by con.getTable().put() > ** AsyncProcess parse many configurations > So, con.getTable().put() is heavy operation for CPU because of getting config > value. > > With in-house patch for this issue, we observed about 10% improvement on > max-throughput (e.g. CPU usage) at client-side: > !Screenshot from 2019-10-18 13-03-24.png|width=508,height=223! > > Seems branch-2 is not affected because client implementation has been changed > dramatically. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack merged pull request #748: HBASE-23185 Fix high cpu usage because getTable()#put() gets config value every time [Take2]
saintstack merged pull request #748: HBASE-23185 Fix high cpu usage because getTable()#put() gets config value every time [Take2] URL: https://github.com/apache/hbase/pull/748 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #775: HBASE-23230 Enforce member visibility in HRegionServer
Apache-HBase commented on issue #775: HBASE-23230 Enforce member visibility in HRegionServer URL: https://github.com/apache/hbase/pull/775#issuecomment-548590712 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 0m 0s | Docker mode activated. | | :broken_heart: | patch | 0m 6s | https://github.com/apache/hbase/pull/775 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/775 | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-775/2/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on issue #775: HBASE-23230 Enforce member visibility in HRegionServer
ndimiduk commented on issue #775: HBASE-23230 Enforce member visibility in HRegionServer URL: https://github.com/apache/hbase/pull/775#issuecomment-548588201 * rebase onto master * address PR comments * fix new checkstyle warnings * fix broken `TestRegionServerMetrics` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member visibility in HRegionServer
ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member visibility in HRegionServer URL: https://github.com/apache/hbase/pull/775#discussion_r341376288 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java ## @@ -3284,45 +3215,7 @@ protected boolean closeRegion(String encodedName, final boolean abort, final Ser return true; } - /** - * Close and offline the region for split or merge - * - * @param regionEncodedName the name of the region(s) to close - * @return true if closed the region successfully. - * @throws IOException - */ - protected boolean closeAndOfflineRegionForSplitOrMerge(final List regionEncodedName) Review comment: Static analysis says it's not used. deleted. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member visibility in HRegionServer
ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member visibility in HRegionServer URL: https://github.com/apache/hbase/pull/775#discussion_r341361136 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java ## @@ -331,47 +328,36 @@ // Go down hard. Used if file system becomes unavailable and also in // debugging and unit tests. private volatile boolean abortRequested; - public static final String ABORT_TIMEOUT = "hbase.regionserver.abort.timeout"; + static final String ABORT_TIMEOUT = "hbase.regionserver.abort.timeout"; // Default abort timeout is 1200 seconds for safe private static final long DEFAULT_ABORT_TIMEOUT = 120; // Will run this task when abort timeout - public static final String ABORT_TIMEOUT_TASK = "hbase.regionserver.abort.timeout.task"; - - ConcurrentMap rowlocks = new ConcurrentHashMap<>(); Review comment: This field is flagged as unused by static analysis. It's possible that something in the scope of `HIA.TOOLS` is reaching in for it, but if the region server itself never uses it, then it's a variable that could be local to the tool that's using it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member visibility in HRegionServer
ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member
visibility in HRegionServer
URL: https://github.com/apache/hbase/pull/775#discussion_r341377173
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
##
@@ -1291,7 +1295,8 @@ protected RpcServerInterface createRpcServer(Server
server, Configuration conf,
}
protected Class getRpcSchedulerFactoryClass() {
-return
this.regionServer.conf.getClass(REGION_SERVER_RPC_SCHEDULER_FACTORY_CLASS,
Review comment:
True. Breaking up these compound statements makes it easier to see NPEs when
they happen. I guess such stylistic choices will become obsolete once Java14
lands.
https://bugs.openjdk.java.net/browse/JDK-8218628
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member visibility in HRegionServer
ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member visibility in HRegionServer URL: https://github.com/apache/hbase/pull/775#discussion_r341363875 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java ## @@ -331,47 +328,36 @@ // Go down hard. Used if file system becomes unavailable and also in // debugging and unit tests. private volatile boolean abortRequested; - public static final String ABORT_TIMEOUT = "hbase.regionserver.abort.timeout"; + static final String ABORT_TIMEOUT = "hbase.regionserver.abort.timeout"; // Default abort timeout is 1200 seconds for safe private static final long DEFAULT_ABORT_TIMEOUT = 120; // Will run this task when abort timeout - public static final String ABORT_TIMEOUT_TASK = "hbase.regionserver.abort.timeout.task"; - - ConcurrentMap rowlocks = new ConcurrentHashMap<>(); + static final String ABORT_TIMEOUT_TASK = "hbase.regionserver.abort.timeout.task"; // A state before we go into stopped state. At this stage we're closing user // space regions. private boolean stopping = false; - - volatile boolean killed = false; - + private volatile boolean killed = false; private volatile boolean shutDown = false; protected final Configuration conf; - private Path rootDir; + private Path dataRootDir; private Path walRootDir; - protected final ReentrantReadWriteLock lock = new ReentrantReadWriteLock(); - - final int numRetries; - protected final int threadWakeFrequency; - protected final int msgInterval; + private final int threadWakeFrequency; + final int msgInterval; private static final String PERIOD_COMPACTION = "hbase.regionserver.compaction.check.period"; private final int compactionCheckFrequency; private static final String PERIOD_FLUSH = "hbase.regionserver.flush.check.period"; private final int flushCheckFrequency; - protected final int numRegionsToReport; - // Stub to do region server status calls against the master. private volatile RegionServerStatusService.BlockingInterface rssStub; private volatile LockService.BlockingInterface lockStub; // RPC client. Used to make the stub above that does region server status checking. - RpcClient rpcClient; - - private RpcControllerFactory rpcControllerFactory; Review comment: Another field assigned but never used. `RpcControllerFactory` instances do existing the codebase, they're just not pulled from `HRegionServer` any more. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member visibility in HRegionServer
ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member visibility in HRegionServer URL: https://github.com/apache/hbase/pull/775#discussion_r341361774 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java ## @@ -331,47 +328,36 @@ // Go down hard. Used if file system becomes unavailable and also in // debugging and unit tests. private volatile boolean abortRequested; - public static final String ABORT_TIMEOUT = "hbase.regionserver.abort.timeout"; + static final String ABORT_TIMEOUT = "hbase.regionserver.abort.timeout"; // Default abort timeout is 1200 seconds for safe private static final long DEFAULT_ABORT_TIMEOUT = 120; // Will run this task when abort timeout - public static final String ABORT_TIMEOUT_TASK = "hbase.regionserver.abort.timeout.task"; - - ConcurrentMap rowlocks = new ConcurrentHashMap<>(); + static final String ABORT_TIMEOUT_TASK = "hbase.regionserver.abort.timeout.task"; // A state before we go into stopped state. At this stage we're closing user // space regions. private boolean stopping = false; - - volatile boolean killed = false; - + private volatile boolean killed = false; private volatile boolean shutDown = false; protected final Configuration conf; - private Path rootDir; + private Path dataRootDir; private Path walRootDir; - protected final ReentrantReadWriteLock lock = new ReentrantReadWriteLock(); - - final int numRetries; Review comment: Yep, a value is set in the constructor and then never read. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member visibility in HRegionServer
ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member
visibility in HRegionServer
URL: https://github.com/apache/hbase/pull/775#discussion_r341364066
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
##
@@ -582,8 +569,6 @@ public HRegionServer(Configuration conf) throws
IOException {
// Disable usage of meta replicas in the regionserver
this.conf.setBoolean(HConstants.USE_META_REPLICAS, false);
// Config'ed params
- this.numRetries =
this.conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
- HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
Review comment:
Yep, never used.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #773: HBASE-23212 : Dynamically reload configs for Region Recovery chore
Apache-HBase commented on issue #773: HBASE-23212 : Dynamically reload configs for Region Recovery chore URL: https://github.com/apache/hbase/pull/773#issuecomment-548570647 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 1m 17s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 0s | No case conflicting files found. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ master Compile Tests _ | | :blue_heart: | mvndep | 0m 36s | Maven dependency ordering for branch | | :green_heart: | mvninstall | 5m 36s | master passed | | :green_heart: | compile | 3m 3s | master passed | | :green_heart: | checkstyle | 2m 32s | master passed | | :blue_heart: | refguide | 5m 25s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | :green_heart: | shadedjars | 4m 37s | branch has no errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 3m 45s | master passed | | :blue_heart: | spotbugs | 3m 36s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :green_heart: | findbugs | 18m 3s | master passed | ||| _ Patch Compile Tests _ | | :blue_heart: | mvndep | 0m 15s | Maven dependency ordering for patch | | :green_heart: | mvninstall | 4m 53s | the patch passed | | :green_heart: | compile | 2m 59s | the patch passed | | :green_heart: | javac | 2m 59s | the patch passed | | :green_heart: | checkstyle | 2m 28s | the patch passed | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :blue_heart: | refguide | 5m 43s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | :green_heart: | shadedjars | 4m 35s | patch has no errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 15m 39s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | :green_heart: | javadoc | 3m 47s | the patch passed | | :green_heart: | findbugs | 18m 41s | the patch passed | ||| _ Other Tests _ | | :broken_heart: | unit | 299m 6s | root in the patch failed. | | :green_heart: | asflicense | 1m 38s | The patch does not generate ASF License warnings. | | | | 411m 40s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.client.TestSnapshotTemporaryDirectoryWithRegionReplicas | | | hadoop.hbase.master.TestSplitWALManager | | | hadoop.hbase.util.TestFromClientSide3WoUnsafe | | | hadoop.hbase.client.TestFromClientSide | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-773/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/773 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile refguide | | uname | Linux b1268a42ab57 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-773/out/precommit/personality/provided.sh | | git revision | master / e1b4a2a895 | | Default Java | 1.8.0_181 | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-773/4/artifact/out/branch-site/book.html | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-773/4/artifact/out/patch-site/book.html | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-773/4/artifact/out/patch-unit-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-773/4/testReport/ | | Max. process+thread count | 4795 (vs. ulimit of 1) | | modules | C: hbase-common hbase-server . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-773/4/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to
[jira] [Commented] (HBASE-23209) Simplify logic in DefaultMobStoreCompactor
[ https://issues.apache.org/jira/browse/HBASE-23209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964331#comment-16964331 ] Vladimir Rodionov commented on HBASE-23209: --- Changed was pushed to parents PR branch. > Simplify logic in DefaultMobStoreCompactor > -- > > Key: HBASE-23209 > URL: https://issues.apache.org/jira/browse/HBASE-23209 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Major > > The major compaction loop is quite large and has many branches, especially in > a non-MOB mode. Consider moving MOB data only Ain a MOB compaction mode and > simplify non-MOB case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23209) Simplify logic in DefaultMobStoreCompactor
[ https://issues.apache.org/jira/browse/HBASE-23209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-23209. --- Resolution: Fixed > Simplify logic in DefaultMobStoreCompactor > -- > > Key: HBASE-23209 > URL: https://issues.apache.org/jira/browse/HBASE-23209 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Major > > The major compaction loop is quite large and has many branches, especially in > a non-MOB mode. Consider moving MOB data only Ain a MOB compaction mode and > simplify non-MOB case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23209) Simplify logic in DefaultMobStoreCompactor
[ https://issues.apache.org/jira/browse/HBASE-23209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964330#comment-16964330 ] Vladimir Rodionov commented on HBASE-23209: --- Reduced code, by leaving handling of a changed mobStoreThreshold to a MOB compaction only. Now, during regular compactions we do not check if MOB threshold was changed and do not handle this case. > Simplify logic in DefaultMobStoreCompactor > -- > > Key: HBASE-23209 > URL: https://issues.apache.org/jira/browse/HBASE-23209 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Major > > The major compaction loop is quite large and has many branches, especially in > a non-MOB mode. Consider moving MOB data only Ain a MOB compaction mode and > simplify non-MOB case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23238) Additional test and checks for null references on ScannerCallableWithReplicas
[
https://issues.apache.org/jira/browse/HBASE-23238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HBASE-23238:
-
Status: Patch Available (was: In Progress)
Attaching a branch-2 patch for the pre-commit job runs.
> Additional test and checks for null references on ScannerCallableWithReplicas
> -
>
> Key: HBASE-23238
> URL: https://issues.apache.org/jira/browse/HBASE-23238
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 1.2.12
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HBASE-23238.branch-2.patch
>
>
> One of our customers running a 1.2 based version is facing NPE when scanning
> data from a MR job. It happens when the map task is finalising:
> {noformat}
> ...
> 2019-09-10 14:17:22,238 INFO [main] org.apache.hadoop.mapred.MapTask:
> Ignoring exception during close for
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader@3a5b7d7e
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.setClose(ScannerCallableWithReplicas.java:99)
> at
> org.apache.hadoop.hbase.client.ClientScanner.close(ClientScanner.java:730)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.close(TableRecordReaderImpl.java:178)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReader.close(TableRecordReader.java:89)
> at
> org.apache.hadoop.hbase.mapreduce.MultiTableInputFormatBase$1.close(MultiTableInputFormatBase.java:112)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:529)
> at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2039)
> ...
> 2019-09-10 14:18:24,601 FATAL [IPC Server handler 5 on 35745]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
> attempt_1566832111959_6047_m_00_3 - exited :
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.setClose(ScannerCallableWithReplicas.java:99)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264)
> at
> org.apache.hadoop.hbase.client.ClientScanner.possiblyNextScanner(ClientScanner.java:248)
> at
> org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:542)
> at
> org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:371)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:222)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:147)
> at
> org.apache.hadoop.hbase.mapreduce.MultiTableInputFormatBase$1.nextKeyValue(MultiTableInputFormatBase.java:139)
> ...
> {noformat}
> After some investigation, we found out that 1.2 based deployments will
> consistently face this problem under the following conditions:
> 1) The sum of all KVs size targeted to be returned in the scan is larger than
> *max result size*;
> 2) At same time, the scan filter has exhaust, so that all remaining KVs
> should be filtered and not returned;
> We could simulate this with the UT being proposed in this PR. When checking
> newer branches, though, I could verify this specific problem is not present
> on newer branches, I believe it was indirectly sorted by changes from
> HBASE-17489.
> Nevertheless, I think it would still be useful to have this extra test and
> checks added as a safeguard measure.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23238) Additional test and checks for null references on ScannerCallableWithReplicas
[
https://issues.apache.org/jira/browse/HBASE-23238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HBASE-23238:
-
Attachment: HBASE-23238.branch-2.patch
> Additional test and checks for null references on ScannerCallableWithReplicas
> -
>
> Key: HBASE-23238
> URL: https://issues.apache.org/jira/browse/HBASE-23238
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 1.2.12
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HBASE-23238.branch-2.patch
>
>
> One of our customers running a 1.2 based version is facing NPE when scanning
> data from a MR job. It happens when the map task is finalising:
> {noformat}
> ...
> 2019-09-10 14:17:22,238 INFO [main] org.apache.hadoop.mapred.MapTask:
> Ignoring exception during close for
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader@3a5b7d7e
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.setClose(ScannerCallableWithReplicas.java:99)
> at
> org.apache.hadoop.hbase.client.ClientScanner.close(ClientScanner.java:730)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.close(TableRecordReaderImpl.java:178)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReader.close(TableRecordReader.java:89)
> at
> org.apache.hadoop.hbase.mapreduce.MultiTableInputFormatBase$1.close(MultiTableInputFormatBase.java:112)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:529)
> at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2039)
> ...
> 2019-09-10 14:18:24,601 FATAL [IPC Server handler 5 on 35745]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
> attempt_1566832111959_6047_m_00_3 - exited :
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.setClose(ScannerCallableWithReplicas.java:99)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264)
> at
> org.apache.hadoop.hbase.client.ClientScanner.possiblyNextScanner(ClientScanner.java:248)
> at
> org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:542)
> at
> org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:371)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:222)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:147)
> at
> org.apache.hadoop.hbase.mapreduce.MultiTableInputFormatBase$1.nextKeyValue(MultiTableInputFormatBase.java:139)
> ...
> {noformat}
> After some investigation, we found out that 1.2 based deployments will
> consistently face this problem under the following conditions:
> 1) The sum of all KVs size targeted to be returned in the scan is larger than
> *max result size*;
> 2) At same time, the scan filter has exhaust, so that all remaining KVs
> should be filtered and not returned;
> We could simulate this with the UT being proposed in this PR. When checking
> newer branches, though, I could verify this specific problem is not present
> on newer branches, I believe it was indirectly sorted by changes from
> HBASE-17489.
> Nevertheless, I think it would still be useful to have this extra test and
> checks added as a safeguard measure.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member visibility in HRegionServer
ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member
visibility in HRegionServer
URL: https://github.com/apache/hbase/pull/775#discussion_r341296609
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
##
@@ -1819,7 +1785,7 @@ protected void chore() {
if (r.shouldFlush(whyFlush)) {
FlushRequester requester = server.getFlushRequester();
if (requester != null) {
-long randomDelay = (long) RandomUtils.nextInt(0, rangeOfDelay) +
MIN_DELAY_TIME;
+long randomDelay = RandomUtils.nextLong(0, rangeOfDelayMs) +
MIN_DELAY_TIME;
Review comment:
No I don't believe it will. I think `int` was used down here because the
value is read out as an `int` using `conf.getInt()` -- which it still uses. The
value will remain as whatever was read from the config. I'm simply converting
the value to a `long` slightly earlier.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member visibility in HRegionServer
ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member
visibility in HRegionServer
URL: https://github.com/apache/hbase/pull/775#discussion_r341293775
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
##
@@ -393,9 +393,13 @@ public MasterRpcServices(HMaster m) throws IOException {
}
@Override
- protected RpcServerInterface createRpcServer(Server server, Configuration
conf,
- RpcSchedulerFactory rpcSchedulerFactory, InetSocketAddress bindAddress,
String name)
- throws IOException {
+ protected RpcServerInterface createRpcServer(
Review comment:
@saintstack @binlijin Here's an example of this style with a throws clause.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member visibility in HRegionServer
ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member
visibility in HRegionServer
URL: https://github.com/apache/hbase/pull/775#discussion_r341292482
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
##
@@ -2285,42 +2245,49 @@ public void postOpenDeployTasks(final
PostOpenDeployContext context) throws IOEx
LOG.debug("Finished post open deploy task for " +
r.getRegionInfo().getRegionNameAsString());
}
- @Override
- public boolean reportRegionStateTransition(final
RegionStateTransitionContext context) {
-TransitionCode code = context.getCode();
-long openSeqNum = context.getOpenSeqNum();
+ /**
+ * Helper method for use in tests. Skip the region transition report when
there's no master
+ * around to receive it.
+ */
+ private boolean skipReportingTransition(final RegionStateTransitionContext
context) {
+final TransitionCode code = context.getCode();
+final long openSeqNum = context.getOpenSeqNum();
long masterSystemTime = context.getMasterSystemTime();
-RegionInfo[] hris = context.getHris();
-long[] procIds = context.getProcIds();
-
-if (TEST_SKIP_REPORTING_TRANSITION) {
- // This is for testing only in case there is no master
- // to handle the region transition report at all.
- if (code == TransitionCode.OPENED) {
-Preconditions.checkArgument(hris != null && hris.length == 1);
-if (hris[0].isMetaRegion()) {
- try {
-MetaTableLocator.setMetaLocation(getZooKeeper(), serverName,
-hris[0].getReplicaId(),State.OPEN);
- } catch (KeeperException e) {
-LOG.info("Failed to update meta location", e);
-return false;
- }
-} else {
- try {
-
MetaTableAccessor.updateRegionLocation(asyncClusterConnection.toConnection(),
hris[0],
+final RegionInfo[] hris = context.getHris();
+
+if (code == TransitionCode.OPENED) {
+ Preconditions.checkArgument(hris != null && hris.length == 1);
+ if (hris[0].isMetaRegion()) {
+try {
+ MetaTableLocator.setMetaLocation(getZooKeeper(), serverName,
+ hris[0].getReplicaId(),State.OPEN);
+} catch (KeeperException e) {
+ LOG.info("Failed to update meta location", e);
+ return false;
+}
+ } else {
+try {
+
MetaTableAccessor.updateRegionLocation(asyncClusterConnection.toConnection(),
hris[0],
serverName, openSeqNum, masterSystemTime);
- } catch (IOException e) {
-LOG.info("Failed to update meta", e);
-return false;
- }
+} catch (IOException e) {
+ LOG.info("Failed to update meta", e);
+ return false;
}
}
- return true;
}
+return true;
+ }
+
+ private ReportRegionStateTransitionRequest
createReportRegionStateTransitionRequest(
+ final RegionStateTransitionContext context
+ ) {
Review comment:
@binlijin mentioned the same about my newline habits. You don't like the `)
{` on their own line? I like this style because (1) it unclutters the argument
list, when they would need to line wrap, to have each on their own line; (2)
moving down the `) {` makes room for a `throws` clause with its list of
exceptions; and (3) keeps indentation levels clear re: what is argument list
and what is method body.
Example with throws arguments:
```java
private ReportRegionStateTransitionRequest
createReportRegionStateTransitionRequest(
final RegionStateTransitionContext context
) throws ThisException, ThatException {
// method body
}
```
I don't want to rock the boat too badly for you lads, let me undo these
radical, divergent styles ;)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #777: HBASE-23171 Ensure region state error messages include server making report (branch-2.1)
saintstack commented on a change in pull request #777: HBASE-23171 Ensure region state error messages include server making report (branch-2.1) URL: https://github.com/apache/hbase/pull/777#discussion_r341273252 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java ## @@ -991,8 +991,8 @@ void checkOnlineRegionsReportForMeta(final ServerName serverName, final Set
[GitHub] [hbase] saintstack commented on a change in pull request #777: HBASE-23171 Ensure region state error messages include server making report (branch-2.1)
saintstack commented on a change in pull request #777: HBASE-23171 Ensure region state error messages include server making report (branch-2.1) URL: https://github.com/apache/hbase/pull/777#discussion_r341273007 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java ## @@ -991,8 +991,8 @@ void checkOnlineRegionsReportForMeta(final ServerName serverName, final Set
[GitHub] [hbase] saintstack commented on a change in pull request #777: HBASE-23171 Ensure region state error messages include server making report (branch-2.1)
saintstack commented on a change in pull request #777: HBASE-23171 Ensure region state error messages include server making report (branch-2.1) URL: https://github.com/apache/hbase/pull/777#discussion_r341273365 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java ## @@ -1035,7 +1035,8 @@ void checkOnlineRegionsReport(final ServerStateNode serverNode, final Set
[jira] [Work started] (HBASE-23239) Reporting on status of backing MOB files from client-facing cells
[ https://issues.apache.org/jira/browse/HBASE-23239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-23239 started by Sean Busbey. --- > Reporting on status of backing MOB files from client-facing cells > - > > Key: HBASE-23239 > URL: https://issues.apache.org/jira/browse/HBASE-23239 > Project: HBase > Issue Type: Improvement > Components: mapreduce, mob, Operability >Affects Versions: 2.1.0, 2.0.0, 2.2.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > > We should be able to do some correctness testing of the mob files referenced > by current cells in a mob enabled table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] wchevreuil merged pull request #780: HBASE-23238 Additional test and checks for null references on Scanner…
wchevreuil merged pull request #780: HBASE-23238 Additional test and checks for null references on Scanner… URL: https://github.com/apache/hbase/pull/780 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (HBASE-23239) Reporting on status of backing MOB files from client-facing cells
Sean Busbey created HBASE-23239: --- Summary: Reporting on status of backing MOB files from client-facing cells Key: HBASE-23239 URL: https://issues.apache.org/jira/browse/HBASE-23239 Project: HBase Issue Type: Improvement Components: mapreduce, mob, Operability Affects Versions: 2.2.0, 2.0.0, 2.1.0 Reporter: Sean Busbey Assignee: Sean Busbey We should be able to do some correctness testing of the mob files referenced by current cells in a mob enabled table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #202: HBASE-22280 Separate read/write handler for priority request(especial…
Apache-HBase commented on issue #202: HBASE-22280 Separate read/write handler for priority request(especial… URL: https://github.com/apache/hbase/pull/202#issuecomment-548471975 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 0m 33s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 1s | No case conflicting files found. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ master Compile Tests _ | | :green_heart: | mvninstall | 5m 30s | master passed | | :green_heart: | compile | 0m 56s | master passed | | :green_heart: | checkstyle | 1m 17s | master passed | | :green_heart: | shadedjars | 4m 36s | branch has no errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 0m 37s | master passed | | :blue_heart: | spotbugs | 3m 55s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :green_heart: | findbugs | 3m 52s | master passed | ||| _ Patch Compile Tests _ | | :green_heart: | mvninstall | 4m 54s | the patch passed | | :green_heart: | compile | 0m 56s | the patch passed | | :green_heart: | javac | 0m 56s | the patch passed | | :green_heart: | checkstyle | 1m 19s | the patch passed | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :green_heart: | shadedjars | 4m 37s | patch has no errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 15m 33s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | :green_heart: | javadoc | 0m 36s | the patch passed | | :green_heart: | findbugs | 4m 7s | the patch passed | ||| _ Other Tests _ | | :broken_heart: | unit | 162m 49s | hbase-server in the patch failed. | | :green_heart: | asflicense | 0m 34s | The patch does not generate ASF License warnings. | | | | 219m 13s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.master.assignment.TestReportOnlineRegionsRace | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-202/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/202 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 0e66ee070ec5 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-202/out/precommit/personality/provided.sh | | git revision | master / e1b4a2a895 | | Default Java | 1.8.0_181 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-202/1/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-202/1/testReport/ | | Max. process+thread count | 4924 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-202/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member visibility in HRegionServer
ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member
visibility in HRegionServer
URL: https://github.com/apache/hbase/pull/775#discussion_r341227551
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
##
@@ -3807,10 +3688,14 @@ public SecureBulkLoadManager
getSecureBulkLoadManager() {
}
@Override
- public EntityLock regionLock(List regionInfos, String
description, Abortable abort)
- throws IOException {
-return new LockServiceClient(conf, lockStub,
asyncClusterConnection.getNonceGenerator())
- .regionLock(regionInfos, description, abort);
+ public EntityLock regionLock(
+ final List regionInfo,
+ final String description,
+ final Abortable abort
+ ) {
Review comment:
I'm using the code formatter in `dev-support/hbase_eclipse_formatter.xml`
and checkstyle doesn't complain about it. The newlines between method arguments
make for simpler reading and shorter diffs in the long run. My rule of thumb is
that if the parameter list need to wrap, they should probably be broken out
into one line each.
What formatter do you use? It would be great if we didn't have to talk about
this at all, if there was a tool like rust or go that just does it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #780: HBASE-23238 Additional test and checks for null references on Scanner…
Apache-HBase commented on issue #780: HBASE-23238 Additional test and checks for null references on Scanner… URL: https://github.com/apache/hbase/pull/780#issuecomment-548447243 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 1m 27s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 0s | No case conflicting files found. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ branch-1 Compile Tests _ | | :blue_heart: | mvndep | 1m 20s | Maven dependency ordering for branch | | :green_heart: | mvninstall | 7m 36s | branch-1 passed | | :green_heart: | compile | 1m 1s | branch-1 passed with JDK v1.8.0_232 | | :green_heart: | compile | 1m 13s | branch-1 passed with JDK v1.7.0_242 | | :green_heart: | checkstyle | 2m 23s | branch-1 passed | | :green_heart: | shadedjars | 3m 10s | branch has no errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 0m 53s | branch-1 passed with JDK v1.8.0_232 | | :green_heart: | javadoc | 1m 8s | branch-1 passed with JDK v1.7.0_242 | | :blue_heart: | spotbugs | 2m 52s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :green_heart: | findbugs | 4m 22s | branch-1 passed | ||| _ Patch Compile Tests _ | | :blue_heart: | mvndep | 0m 14s | Maven dependency ordering for patch | | :green_heart: | mvninstall | 2m 7s | the patch passed | | :green_heart: | compile | 1m 0s | the patch passed with JDK v1.8.0_232 | | :green_heart: | javac | 1m 0s | the patch passed | | :green_heart: | compile | 1m 10s | the patch passed with JDK v1.7.0_242 | | :green_heart: | javac | 1m 10s | the patch passed | | :green_heart: | checkstyle | 2m 22s | the patch passed | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :green_heart: | shadedjars | 3m 10s | patch has no errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 5m 21s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | :green_heart: | javadoc | 0m 51s | the patch passed with JDK v1.8.0_232 | | :green_heart: | javadoc | 1m 8s | the patch passed with JDK v1.7.0_242 | | :green_heart: | findbugs | 4m 42s | the patch passed | ||| _ Other Tests _ | | :green_heart: | unit | 2m 38s | hbase-client in the patch passed. | | :green_heart: | unit | 148m 20s | hbase-server in the patch passed. | | :green_heart: | asflicense | 0m 44s | The patch does not generate ASF License warnings. | | | | 202m 33s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-780/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/780 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 8e102a11e2be 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-780/out/precommit/personality/provided.sh | | git revision | branch-1 / 4bcc397 | | Default Java | 1.7.0_242 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_232 /usr/lib/jvm/zulu-7-amd64:1.7.0_242 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-780/1/testReport/ | | Max. process+thread count | 4250 (vs. ulimit of 1) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-780/1/console | | versions | git=1.9.1 maven=3.0.5 findbugs=3.0.1 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member visibility in HRegionServer
ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member
visibility in HRegionServer
URL: https://github.com/apache/hbase/pull/775#discussion_r341225818
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
##
@@ -1797,18 +1762,19 @@ protected void chore() {
}
}
- static class PeriodicMemStoreFlusher extends ScheduledChore {
-final HRegionServer server;
-final static int RANGE_OF_DELAY = 5 * 60; // 5 min in seconds
-final static int MIN_DELAY_TIME = 0; // millisec
+ private static class PeriodicMemStoreFlusher extends ScheduledChore {
Review comment:
Yeah perhaps in a subsequent patch. Trying to keep this scope narrow on "OOP
101" stuff.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member visibility in HRegionServer
ndimiduk commented on a change in pull request #775: HBASE-23230 Enforce member
visibility in HRegionServer
URL: https://github.com/apache/hbase/pull/775#discussion_r341225386
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
##
@@ -549,28 +534,30 @@
* means it needs to just come up and make do without a Master to talk to:
e.g. in test or
* HRegionServer is doing other than its usual duties: e.g. as an
hollowed-out host whose only
* purpose is as a Replication-stream sink; see HBASE-18846 for more.
+ * TODO: can this replace {@link #TEST_SKIP_REPORTING_TRANSITION} ?
*/
private final boolean masterless;
- static final String MASTERLESS_CONFIG_NAME = "hbase.masterless";
+ private static final String MASTERLESS_CONFIG_NAME = "hbase.masterless";
/**regionserver codec list **/
- public static final String REGIONSERVER_CODEC = "hbase.regionserver.codecs";
+ private static final String REGIONSERVER_CODEC = "hbase.regionserver.codecs";
// A timer to shutdown the process if abort takes too long
private Timer abortMonitor;
/**
- * Starts a HRegionServer at the default location
+ * Starts a HRegionServer at the default location.
+ *
+ * Don't start any services or managers in here in the Constructor.
+ * Defer till after we register with the Master as much as possible. See
{@link #startServices}.
*/
- // Don't start any services or managers in here in the Constructor.
- // Defer till after we register with the Master as much as possible. See
#startServices.
- public HRegionServer(Configuration conf) throws IOException {
+ public HRegionServer(final Configuration conf) throws IOException {
Review comment:
`final` on params makes me feel better at night knowing we're not trying to
use "out" parameters. Overall I find that pushing toward a more functional
style leads to more reliable and maintainable code, and making as many things
final as possible makes for good general hygiene.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #749: HBASE-23205 Correctly update the position of WALs currently being replicated
Apache-HBase commented on issue #749: HBASE-23205 Correctly update the position of WALs currently being replicated URL: https://github.com/apache/hbase/pull/749#issuecomment-548427010 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 0m 35s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 0s | No case conflicting files found. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | ||| _ branch-1 Compile Tests _ | | :green_heart: | mvninstall | 8m 26s | branch-1 passed | | :green_heart: | compile | 0m 42s | branch-1 passed with JDK v1.8.0_232 | | :green_heart: | compile | 0m 48s | branch-1 passed with JDK v1.7.0_242 | | :green_heart: | checkstyle | 1m 47s | branch-1 passed | | :green_heart: | shadedjars | 3m 9s | branch has no errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 0m 36s | branch-1 passed with JDK v1.8.0_232 | | :green_heart: | javadoc | 0m 42s | branch-1 passed with JDK v1.7.0_242 | | :blue_heart: | spotbugs | 3m 3s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :green_heart: | findbugs | 3m 1s | branch-1 passed | ||| _ Patch Compile Tests _ | | :green_heart: | mvninstall | 2m 5s | the patch passed | | :green_heart: | compile | 0m 41s | the patch passed with JDK v1.8.0_232 | | :green_heart: | javac | 0m 41s | the patch passed | | :green_heart: | compile | 0m 46s | the patch passed with JDK v1.7.0_242 | | :green_heart: | javac | 0m 46s | the patch passed | | :green_heart: | checkstyle | 1m 41s | hbase-server: The patch generated 0 new + 42 unchanged - 12 fixed = 42 total (was 54) | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :green_heart: | shadedjars | 3m 7s | patch has no errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 5m 11s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | :green_heart: | javadoc | 0m 30s | the patch passed with JDK v1.8.0_232 | | :green_heart: | javadoc | 0m 42s | the patch passed with JDK v1.7.0_242 | | :green_heart: | findbugs | 3m 6s | the patch passed | ||| _ Other Tests _ | | :green_heart: | unit | 118m 17s | hbase-server in the patch passed. | | :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. | | | | 159m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-749/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/749 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux f4a65f785f70 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-749/out/precommit/personality/provided.sh | | git revision | branch-1 / 4bcc397 | | Default Java | 1.7.0_242 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_232 /usr/lib/jvm/zulu-7-amd64:1.7.0_242 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-749/5/testReport/ | | Max. process+thread count | 3765 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-749/5/console | | versions | git=1.9.1 maven=3.0.5 findbugs=3.0.1 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22917) Proc-WAL roll fails always saying someone else has already created log
[
https://issues.apache.org/jira/browse/HBASE-22917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964113#comment-16964113
]
Michael Stack commented on HBASE-22917:
---
Reverted from branch-2.2+. The PR has been closed as merged. Make a new one
[~pankaj2461] after we have answers for [~zhangduo] query above? Thanks sir.
> Proc-WAL roll fails always saying someone else has already created log
> --
>
> Key: HBASE-22917
> URL: https://issues.apache.org/jira/browse/HBASE-22917
> Project: HBase
> Issue Type: Bug
> Components: proc-v2, wal
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 3.0.0, 2.3.0, 2.2.3
>
>
> Recently we met a weird scenario where Procedure WAL roll fails as it is
> already created by someone else.
> Later while going through the logs and code, observed that during Proc-WAL
> roll it failed to write the header. On failure file stream is just closed,
> {code}
> try {
> ProcedureWALFormat.writeHeader(newStream, header);
> startPos = newStream.getPos();
> } catch (IOException ioe) {
> LOG.warn("Encountered exception writing header", ioe);
> newStream.close();
> return false;
> }
> {code}
> Since we don't delete the corrupted file or increment the *flushLogId*, so on
> each retry it is trying to create the same *flushLogId* file. However Hmaster
> failover will resolve this issue, but we should handle it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22917) Proc-WAL roll fails always saying someone else has already created log
[
https://issues.apache.org/jira/browse/HBASE-22917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964107#comment-16964107
]
Michael Stack commented on HBASE-22917:
---
Reasonable. Let me revert.
> Proc-WAL roll fails always saying someone else has already created log
> --
>
> Key: HBASE-22917
> URL: https://issues.apache.org/jira/browse/HBASE-22917
> Project: HBase
> Issue Type: Bug
> Components: proc-v2, wal
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 3.0.0, 2.3.0, 2.2.3
>
>
> Recently we met a weird scenario where Procedure WAL roll fails as it is
> already created by someone else.
> Later while going through the logs and code, observed that during Proc-WAL
> roll it failed to write the header. On failure file stream is just closed,
> {code}
> try {
> ProcedureWALFormat.writeHeader(newStream, header);
> startPos = newStream.getPos();
> } catch (IOException ioe) {
> LOG.warn("Encountered exception writing header", ioe);
> newStream.close();
> return false;
> }
> {code}
> Since we don't delete the corrupted file or increment the *flushLogId*, so on
> each retry it is trying to create the same *flushLogId* file. However Hmaster
> failover will resolve this issue, but we should handle it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23136) PartionedMobFileCompactor bulkloaded files shouldn't get replicated (addressing buklload replication related issue raised in HBASE-22380)
[ https://issues.apache.org/jira/browse/HBASE-23136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-23136: - Fix Version/s: 2.1.8 > PartionedMobFileCompactor bulkloaded files shouldn't get replicated > (addressing buklload replication related issue raised in HBASE-22380) > - > > Key: HBASE-23136 > URL: https://issues.apache.org/jira/browse/HBASE-23136 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0, 2.2.2, 2.1.8 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Critical > Labels: bulkload, mob > Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3 > > Attachments: HBASE-23136.branch-2.1.patch, > HBASE-23136.branch-2.2.patch, HBASE-23136.branch-2.patch > > > Following bulkload replication fixes started in HBASE-22380, this addresses > [~javaman_chen] observation regarding *PartitionedMobCompactor* and bulk > loaded. As noted by [~javaman_chen], *PartitionedMobCompactor* uses bulkload > feature to update resulting hfile into region hstore. This file, however, > shouldn't get replicated in any condition. This PR adds required changes and > extra test for this situation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23136) PartionedMobFileCompactor bulkloaded files shouldn't get replicated (addressing buklload replication related issue raised in HBASE-22380)
[ https://issues.apache.org/jira/browse/HBASE-23136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-23136: - Resolution: Fixed Status: Resolved (was: Patch Available) > PartionedMobFileCompactor bulkloaded files shouldn't get replicated > (addressing buklload replication related issue raised in HBASE-22380) > - > > Key: HBASE-23136 > URL: https://issues.apache.org/jira/browse/HBASE-23136 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0, 2.2.2, 2.1.8 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Critical > Labels: bulkload, mob > Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3 > > Attachments: HBASE-23136.branch-2.1.patch, > HBASE-23136.branch-2.2.patch, HBASE-23136.branch-2.patch > > > Following bulkload replication fixes started in HBASE-22380, this addresses > [~javaman_chen] observation regarding *PartitionedMobCompactor* and bulk > loaded. As noted by [~javaman_chen], *PartitionedMobCompactor* uses bulkload > feature to update resulting hfile into region hstore. This file, however, > shouldn't get replicated in any condition. This PR adds required changes and > extra test for this situation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23136) PartionedMobFileCompactor bulkloaded files shouldn't get replicated (addressing buklload replication related issue raised in HBASE-22380)
[ https://issues.apache.org/jira/browse/HBASE-23136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964074#comment-16964074 ] Wellington Chevreuil commented on HBASE-23136: -- branch-2.1 patch test failure is unrelated, got it passing locally. Pushing this commit to branch-2.1. > PartionedMobFileCompactor bulkloaded files shouldn't get replicated > (addressing buklload replication related issue raised in HBASE-22380) > - > > Key: HBASE-23136 > URL: https://issues.apache.org/jira/browse/HBASE-23136 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0, 2.2.2, 2.1.8 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Critical > Labels: bulkload, mob > Fix For: 3.0.0, 2.3.0, 2.2.3 > > Attachments: HBASE-23136.branch-2.1.patch, > HBASE-23136.branch-2.2.patch, HBASE-23136.branch-2.patch > > > Following bulkload replication fixes started in HBASE-22380, this addresses > [~javaman_chen] observation regarding *PartitionedMobCompactor* and bulk > loaded. As noted by [~javaman_chen], *PartitionedMobCompactor* uses bulkload > feature to update resulting hfile into region hstore. This file, however, > shouldn't get replicated in any condition. This PR adds required changes and > extra test for this situation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23221) Polish the WAL interface after HBASE-23181
[ https://issues.apache.org/jira/browse/HBASE-23221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-23221. --- Hadoop Flags: Reviewed Resolution: Fixed Cherry-picked to branch-2.1+. > Polish the WAL interface after HBASE-23181 > -- > > Key: HBASE-23221 > URL: https://issues.apache.org/jira/browse/HBASE-23221 > Project: HBase > Issue Type: Improvement > Components: regionserver, wal >Reporter: Duo Zhang >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3 > > > We have a closeRegion flag which seems to be redundant with the marker > WALEdit. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22917) Proc-WAL roll fails always saying someone else has already created log
[
https://issues.apache.org/jira/browse/HBASE-22917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964050#comment-16964050
]
Hudson commented on HBASE-22917:
Results for branch master
[build #1522 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1522/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1522//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1522//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/master/1522//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Proc-WAL roll fails always saying someone else has already created log
> --
>
> Key: HBASE-22917
> URL: https://issues.apache.org/jira/browse/HBASE-22917
> Project: HBase
> Issue Type: Bug
> Components: proc-v2, wal
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 3.0.0, 2.3.0, 2.2.3
>
>
> Recently we met a weird scenario where Procedure WAL roll fails as it is
> already created by someone else.
> Later while going through the logs and code, observed that during Proc-WAL
> roll it failed to write the header. On failure file stream is just closed,
> {code}
> try {
> ProcedureWALFormat.writeHeader(newStream, header);
> startPos = newStream.getPos();
> } catch (IOException ioe) {
> LOG.warn("Encountered exception writing header", ioe);
> newStream.close();
> return false;
> }
> {code}
> Since we don't delete the corrupted file or increment the *flushLogId*, so on
> each retry it is trying to create the same *flushLogId* file. However Hmaster
> failover will resolve this issue, but we should handle it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23192) CatalogJanitor consistencyCheck does not log problematic row on exception
[
https://issues.apache.org/jira/browse/HBASE-23192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964051#comment-16964051
]
Hudson commented on HBASE-23192:
Results for branch master
[build #1522 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1522/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1522//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1522//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/master/1522//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> CatalogJanitor consistencyCheck does not log problematic row on exception
> -
>
> Key: HBASE-23192
> URL: https://issues.apache.org/jira/browse/HBASE-23192
> Project: HBase
> Issue Type: Bug
> Components: hbck2
>Affects Versions: 2.1.7, 2.2.2
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Minor
> Fix For: 3.0.0, 2.3.0, 2.3.3
>
>
> Small stuff. Trying to fix a cluser, cleared an info:server field. Damaged
> hbase:meta for CatalogJanitor; when it should have just logged and skipped
> the bad entity when doing consistency check, instead CJ crashed. Also doesn't
> log bad raw which would help debugging.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22739) ArrayIndexOutOfBoundsException when balance
[
https://issues.apache.org/jira/browse/HBASE-22739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964052#comment-16964052
]
Hudson commented on HBASE-22739:
Results for branch master
[build #1522 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1522/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1522//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1522//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/master/1522//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> ArrayIndexOutOfBoundsException when balance
> ---
>
> Key: HBASE-22739
> URL: https://issues.apache.org/jira/browse/HBASE-22739
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Reporter: casuallc
>Assignee: Lijin Bin
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3
>
>
>
> {code:java}
> 2019-07-25 15:19:59,828 ERROR [master/nna:16000.Chore.1]
> hbase.ScheduledChore: Caught error
> java.lang.ArrayIndexOutOfBoundsException: 3171
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.removeRegion(BaseLoadBalancer.java:873)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.doAction(BaseLoadBalancer.java:716)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:407)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:318)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1650)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1567)
> at
> org.apache.hadoop.hbase.master.balancer.BalancerChore.chore(BalancerChore.java:49)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> should check if the regionIndex is valid when removeRegion,
> java:
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
> {code:java}
> int[] removeRegion(int[] regions, int regionIndex) {
> //TODO: this maybe costly. Consider using linked lists
> int[] newRegions = new int[regions.length - 1];
> int i = 0;
> for (i = 0; i < regions.length; i++) {
> if (regions[i] == regionIndex) {
> break;
> }
> if (i == regions.length - 1) {
> return Arrays.copyOf(regions, regions.length);
> }
> newRegions[i] = regions[i];
> }
> System.arraycopy(regions, i+1, newRegions, i, newRegions.length - i);
> return newRegions;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23191) Log spams on Replication
[
https://issues.apache.org/jira/browse/HBASE-23191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964049#comment-16964049
]
Hudson commented on HBASE-23191:
Results for branch master
[build #1522 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1522/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1522//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1522//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/master/1522//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Log spams on Replication
>
>
> Key: HBASE-23191
> URL: https://issues.apache.org/jira/browse/HBASE-23191
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 3.0.0
>Reporter: Karthik Palanisamy
>Assignee: Karthik Palanisamy
>Priority: Trivial
> Fix For: 3.0.0, 2.3.0, 2.2.3
>
>
> If no new active writes in WAL edit, then *WALEntryStream#hasNext ->
> ReaderBase -> ProtobufLogReader#readNext* will reach end of file. It would be
> a good idea for changing the log level from INFO to DEBUG.
>
> {code:java}
> 2019-10-18 22:25:03,572 INFO
> [RS_REFRESH_PEER-regionserver/apache303:16020-0.replicationSource,p1hdp314.replicationSource.wal-reader.apache303.openstacklocal%2C16020%2C1571383146790,p1hdp314]
> wal.ProtobufLogReader: Reached the end of file at position 83
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23175) Yarn unable to acquire delegation token for HBase Spark jobs
[
https://issues.apache.org/jira/browse/HBASE-23175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964053#comment-16964053
]
Hudson commented on HBASE-23175:
Results for branch master
[build #1522 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1522/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1522//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1522//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/master/1522//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Yarn unable to acquire delegation token for HBase Spark jobs
>
>
> Key: HBASE-23175
> URL: https://issues.apache.org/jira/browse/HBASE-23175
> Project: HBase
> Issue Type: Bug
> Components: security, spark
>Affects Versions: 2.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3
>
> Attachments: HBASE-23175.master.001.patch
>
>
> Spark rely on the TokenUtil.obtainToken(conf) API which is removed in
> HBase-2.0, though it has been fixed in SPARK-26432 to use the new API but
> planned for Spark-3.0, hence we need the fix in HBase until they release it
> and we upgrade it
> {code}
> 18/03/20 20:39:07 ERROR ApplicationMaster: User class threw exception:
> org.apache.hadoop.hbase.HBaseIOException:
> com.google.protobuf.ServiceException: Error calling method
> hbase.pb.AuthenticationService.GetAuthenticationToken
> org.apache.hadoop.hbase.HBaseIOException:
> com.google.protobuf.ServiceException: Error calling method
> hbase.pb.AuthenticationService.GetAuthenticationToken
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:360)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:346)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:86)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil$1.run(TokenUtil.java:121)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil$1.run(TokenUtil.java:118)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:313)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:118)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:272)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:533)
> at
> org.apache.hadoop.hbase.spark.HBaseContext.(HBaseContext.scala:73)
> at
> org.apache.hadoop.hbase.spark.JavaHBaseContext.(JavaHBaseContext.scala:46)
> at
> org.apache.hadoop.hbase.spark.example.hbasecontext.JavaHBaseBulkDeleteExample.main(JavaHBaseBulkDeleteExample.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:706)
> Caused by: com.google.protobuf.ServiceException: Error calling method
> hbase.pb.AuthenticationService.GetAuthenticationToken
> at
> org.apache.hadoop.hbase.client.SyncCoprocessorRpcChannel.callBlockingMethod(SyncCoprocessorRpcChannel.java:71)
> at
> org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService$BlockingStub.getAuthenticationToken(AuthenticationProtos.java:4512)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:81)
> ... 17 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-20827) Add pause when retrying after CallQueueTooBigException for reportRegionStateTransition
[
https://issues.apache.org/jira/browse/HBASE-20827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964048#comment-16964048
]
Hudson commented on HBASE-20827:
Results for branch master
[build #1522 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1522/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1522//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1522//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/master/1522//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add pause when retrying after CallQueueTooBigException for
> reportRegionStateTransition
> ---
>
> Key: HBASE-20827
> URL: https://issues.apache.org/jira/browse/HBASE-20827
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3
>
> Attachments: HBASE-20827.master.001.patch, HBASE-20827.patch
>
>
> We should be adding a pause during retry of CallQueueTooBigException to avoid
> flooding master.
> {code}
> org.apache.hadoop.hbase.CallQueueTooBigException: Call queue is full on
> ctr-e138-1518143905142-387782-01-03.hwx.site,2,1530317245246, too
> many items queued ?
> at sun.reflect.GeneratedConstructorAccessor108.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100)
> at
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:359)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:336)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportRegionStateTransition(HRegionServer.java:2259)
> at
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:120)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException):
> Call queue is full on
> ctr-e138-1518143905142-387782-01-03.hwx.site,2,1530317245246, too
> many items queued ?
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:387)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)
> at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)
> at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:161)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:191)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
> at
>
[jira] [Commented] (HBASE-23231) ReplicationSource do not update metrics after refresh
[
https://issues.apache.org/jira/browse/HBASE-23231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964054#comment-16964054
]
Hudson commented on HBASE-23231:
Results for branch master
[build #1522 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1522/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1522//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1522//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/master/1522//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> ReplicationSource do not update metrics after refresh
> -
>
> Key: HBASE-23231
> URL: https://issues.apache.org/jira/browse/HBASE-23231
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.2.2
>Reporter: Lijin Bin
>Assignee: Lijin Bin
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.2.2, 2.1.8
>
>
> When replication refresh to new state, it will create a new source and
> terminate the old source and replace the old source with new source.
> {code}
> public void refreshSources(String peerId) throws IOException {
> String terminateMessage = "Peer " + peerId +
> " state or config changed. Will close the previous replication source
> and open a new one";
> ReplicationPeer peer = replicationPeers.getPeer(peerId);
> ReplicationSourceInterface src = createSource(peerId, peer);
> // synchronized on latestPaths to avoid missing the new log
> synchronized (this.latestPaths) {
> ReplicationSourceInterface toRemove = this.sources.put(peerId, src);
> if (toRemove != null) {
> LOG.info("Terminate replication source for " + toRemove.getPeerId());
> toRemove.terminate(terminateMessage);
> }
> for (NavigableSet walsByGroup : walsById.get(peerId).values()) {
> walsByGroup.forEach(wal -> src.enqueueLog(new Path(this.logDir,
> wal)));
> }
> }
> LOG.info("Startup replication source for " + src.getPeerId());
> src.startup();
> {code}
> terminate replication source will remove all metrics, current terminate
> replication source be called after create new source which do init metrics,
> so the result is there is no corresponding metrics after refresh replication
> source.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] wchevreuil commented on issue #780: HBASE-23238 Additional test and checks for null references on Scanner…
wchevreuil commented on issue #780: HBASE-23238 Additional test and checks for null references on Scanner… URL: https://github.com/apache/hbase/pull/780#issuecomment-548386690 Thanks @busbey ! > would the test also get forward ported to branches-2/master? or is there something providing equivalent coverage already? I was planning to add the test in all branches, as I don't see any existing one checking on this particular condition. The null checks would also apply to branch-2, but not to master, so for master we should probably have a patch with the test only, but yeah, let's wait for qabot. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #746: HBASE-23195 FSDataInputStreamWrapper unbuffer can NOT invoke the clas…
Apache9 commented on a change in pull request #746: HBASE-23195
FSDataInputStreamWrapper unbuffer can NOT invoke the clas…
URL: https://github.com/apache/hbase/pull/746#discussion_r341147831
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/FSDataInputStreamWrapper.java
##
@@ -270,39 +269,23 @@ public void unbuffer() {
if (this.instanceOfCanUnbuffer == null) {
// To ensure we compute whether the stream is instance of CanUnbuffer
only once.
this.instanceOfCanUnbuffer = false;
-Class[] streamInterfaces = streamClass.getInterfaces();
-for (Class c : streamInterfaces) {
- if
(c.getCanonicalName().toString().equals("org.apache.hadoop.fs.CanUnbuffer")) {
-try {
- this.unbuffer = streamClass.getDeclaredMethod("unbuffer");
-} catch (NoSuchMethodException | SecurityException e) {
- if (isLogTraceEnabled) {
-LOG.trace("Failed to find 'unbuffer' method in class " +
streamClass
-+ " . So there may be a TCP socket connection "
-+ "left open in CLOSE_WAIT state.", e);
- }
- return;
-}
-this.instanceOfCanUnbuffer = true;
-break;
- }
+if (CanUnbuffer.class.isAssignableFrom(streamClass)) {
+ this.unbuffer = (CanUnbuffer) wrappedStream;
+ this.instanceOfCanUnbuffer = true;
}
}
if (this.instanceOfCanUnbuffer) {
try {
- this.unbuffer.invoke(wrappedStream);
-} catch (IllegalAccessException | IllegalArgumentException |
InvocationTargetException e) {
+ this.unbuffer.unbuffer();
+} catch (UnsupportedOperationException e){
Review comment:
So it is possible that a stream implement CanUnbuffer but just throws an
UnsupportedOperationException? Do we have an example?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #746: HBASE-23195 FSDataInputStreamWrapper unbuffer can NOT invoke the clas…
Apache9 commented on a change in pull request #746: HBASE-23195
FSDataInputStreamWrapper unbuffer can NOT invoke the clas…
URL: https://github.com/apache/hbase/pull/746#discussion_r341146952
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/FSDataInputStreamWrapper.java
##
@@ -270,22 +271,18 @@ public void unbuffer() {
if (this.instanceOfCanUnbuffer == null) {
// To ensure we compute whether the stream is instance of CanUnbuffer
only once.
this.instanceOfCanUnbuffer = false;
-Class[] streamInterfaces = streamClass.getInterfaces();
-for (Class c : streamInterfaces) {
- if
(c.getCanonicalName().toString().equals("org.apache.hadoop.fs.CanUnbuffer")) {
-try {
- this.unbuffer = streamClass.getDeclaredMethod("unbuffer");
-} catch (NoSuchMethodException | SecurityException e) {
- if (isLogTraceEnabled) {
-LOG.trace("Failed to find 'unbuffer' method in class " +
streamClass
-+ " . So there may be a TCP socket connection "
-+ "left open in CLOSE_WAIT state.", e);
- }
- return;
+if (CanUnbuffer.class.isAssignableFrom(streamClass)) {
Review comment:
I think why we use isAssignableFrom is for hadoop-2.7-? Why not just use` if
(wrappedStream instanceof CanUnbuffer)`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #773: HBASE-23212 : Dynamically reload configs for Region Recovery chore
virajjasani commented on a change in pull request #773: HBASE-23212 : Dynamically reload configs for Region Recovery chore URL: https://github.com/apache/hbase/pull/773#discussion_r341058494 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionsRecoveryChore.java ## @@ -171,4 +167,20 @@ private void prepareTableToReopenRegionsMap( } + // hashcode/equals implementation to ensure at-most one object of RegionsRecoveryChore Review comment: Sounds good but ideally this chore should have multiple instances running anyways. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #773: HBASE-23212 : Dynamically reload configs for Region Recovery chore
virajjasani commented on a change in pull request #773: HBASE-23212 : Dynamically reload configs for Region Recovery chore URL: https://github.com/apache/hbase/pull/773#discussion_r341058494 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionsRecoveryChore.java ## @@ -171,4 +167,20 @@ private void prepareTableToReopenRegionsMap( } + // hashcode/equals implementation to ensure at-most one object of RegionsRecoveryChore Review comment: Sounds good but ideally this chore should not have multiple instances running anyways. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23073) Add an optional costFunction to balance regions according to a capacity rule
[ https://issues.apache.org/jira/browse/HBASE-23073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964034#comment-16964034 ] Wellington Chevreuil commented on HBASE-23073: -- Thanks [~PierreZ]! Just pressed the *submit-patch* button to have pre-commit jobs running on it. > Add an optional costFunction to balance regions according to a capacity rule > > > Key: HBASE-23073 > URL: https://issues.apache.org/jira/browse/HBASE-23073 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 3.0.0 >Reporter: Pierre Zemb >Assignee: Pierre Zemb >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > Attachments: HBASE-23073.branch-1.001.patch > > > Based on the work in > [HBASE-22618|https://issues.apache.org/jira/browse/HBASE-22618], users can > now load custom costFunctions inside the main balancer used by HBase. As an > example, we like like to add upstream an optional cost function called > HeterogeneousRegionCountCostFunction that will deal with our issue: how to > balance regions according to the capacity of a RS instead of using the > RegionCountSkewCostFunction that is trying to avoid skew. > A rule file is loaded from HDFS before balancing. It contains lines of rules. > A rule is composed of a regexp for hostname, and a limit. For example, we > could have: > * rs[0-9] 200 > * rs1[0-9] 50 > RegionServers with hostname matching the first rules will have a limit of > 200, and the others 50. If there's no match, a default is set. > Thanks to the rule, we have two informations: the max number of regions for > this cluster, and the rules for each servers. HeterogeneousBalancer will try > to balance regions according to their capacity. > Let's take an example. Let's say that we have 20 RS: > 10 RS, named through rs0 to rs9 loaded with 60 regions each, and each can > handle 200 regions. > 10 RS, named through rs10 to rs19 loaded with 60 regions each, and each > can support 50 regions. > Based on the following rules: > rs[0-9] 200 > rs1[0-9] 50 > The second group is overloaded, whereas the first group has plenty of space. > Moving a region from the first group to the second should provide a lower > cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23073) Add an optional costFunction to balance regions according to a capacity rule
[ https://issues.apache.org/jira/browse/HBASE-23073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-23073: - Status: Patch Available (was: Open) > Add an optional costFunction to balance regions according to a capacity rule > > > Key: HBASE-23073 > URL: https://issues.apache.org/jira/browse/HBASE-23073 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 3.0.0 >Reporter: Pierre Zemb >Assignee: Pierre Zemb >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > Attachments: HBASE-23073.branch-1.001.patch > > > Based on the work in > [HBASE-22618|https://issues.apache.org/jira/browse/HBASE-22618], users can > now load custom costFunctions inside the main balancer used by HBase. As an > example, we like like to add upstream an optional cost function called > HeterogeneousRegionCountCostFunction that will deal with our issue: how to > balance regions according to the capacity of a RS instead of using the > RegionCountSkewCostFunction that is trying to avoid skew. > A rule file is loaded from HDFS before balancing. It contains lines of rules. > A rule is composed of a regexp for hostname, and a limit. For example, we > could have: > * rs[0-9] 200 > * rs1[0-9] 50 > RegionServers with hostname matching the first rules will have a limit of > 200, and the others 50. If there's no match, a default is set. > Thanks to the rule, we have two informations: the max number of regions for > this cluster, and the rules for each servers. HeterogeneousBalancer will try > to balance regions according to their capacity. > Let's take an example. Let's say that we have 20 RS: > 10 RS, named through rs0 to rs9 loaded with 60 regions each, and each can > handle 200 regions. > 10 RS, named through rs10 to rs19 loaded with 60 regions each, and each > can support 50 regions. > Based on the following rules: > rs[0-9] 200 > rs1[0-9] 50 > The second group is overloaded, whereas the first group has plenty of space. > Moving a region from the first group to the second should provide a lower > cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22917) Proc-WAL roll fails always saying someone else has already created log
[
https://issues.apache.org/jira/browse/HBASE-22917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964006#comment-16964006
]
Duo Zhang commented on HBASE-22917:
---
I do not think it is fine to merge this back.
This is the feedback on the PR
{quote}
IIRC two master can't write proc-WAL together, do we have such corner scenario?
However here we are trying to cleanup when rollWriter fails when write header
throws IOE.
{quote}
Actually it is our magic on the file id which prevents two masters write the
proc-WAL together, so we should not use this as a assumption to implement our
file id logic, totally wrong.
Now we just increase the file id by one if we failed to delete the old file,
but this is an rpc call right? It could happen that on the NN side, the file
has been deleted successfully but at client side we get an error, and then we
increase the file id by 1, and then there will be a whole, what if another
master tries to write new file id but just fill in the whole? Then we have two
'live' masters which could both write proc wal(at least there be a small
overlap due to the aysnc behavior on zk session expire processing). This will
lead to inconsistency and mess up everything.
So my suggestion is that, unless we have a clear explaination that the above
scenario can not happen, then the safest way is to just abort the HMaster if we
fail to roll the writer. And maybe it is safe to just increase the file id
without deleting the broken proc wal file(this is a typical solution in WAL
based system), but anyway, usually deleting a wal file is not a good idea...
Thanks.
> Proc-WAL roll fails always saying someone else has already created log
> --
>
> Key: HBASE-22917
> URL: https://issues.apache.org/jira/browse/HBASE-22917
> Project: HBase
> Issue Type: Bug
> Components: proc-v2, wal
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 3.0.0, 2.3.0, 2.2.3
>
>
> Recently we met a weird scenario where Procedure WAL roll fails as it is
> already created by someone else.
> Later while going through the logs and code, observed that during Proc-WAL
> roll it failed to write the header. On failure file stream is just closed,
> {code}
> try {
> ProcedureWALFormat.writeHeader(newStream, header);
> startPos = newStream.getPos();
> } catch (IOException ioe) {
> LOG.warn("Encountered exception writing header", ioe);
> newStream.close();
> return false;
> }
> {code}
> Since we don't delete the corrupted file or increment the *flushLogId*, so on
> each retry it is trying to create the same *flushLogId* file. However Hmaster
> failover will resolve this issue, but we should handle it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (HBASE-22917) Proc-WAL roll fails always saying someone else has already created log
[
https://issues.apache.org/jira/browse/HBASE-22917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang reopened HBASE-22917:
---
> Proc-WAL roll fails always saying someone else has already created log
> --
>
> Key: HBASE-22917
> URL: https://issues.apache.org/jira/browse/HBASE-22917
> Project: HBase
> Issue Type: Bug
> Components: proc-v2, wal
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 3.0.0, 2.3.0, 2.2.3
>
>
> Recently we met a weird scenario where Procedure WAL roll fails as it is
> already created by someone else.
> Later while going through the logs and code, observed that during Proc-WAL
> roll it failed to write the header. On failure file stream is just closed,
> {code}
> try {
> ProcedureWALFormat.writeHeader(newStream, header);
> startPos = newStream.getPos();
> } catch (IOException ioe) {
> LOG.warn("Encountered exception writing header", ioe);
> newStream.close();
> return false;
> }
> {code}
> Since we don't delete the corrupted file or increment the *flushLogId*, so on
> each retry it is trying to create the same *flushLogId* file. However Hmaster
> failover will resolve this issue, but we should handle it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23073) Add an optional costFunction to balance regions according to a capacity rule
[ https://issues.apache.org/jira/browse/HBASE-23073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16963992#comment-16963992 ] Pierre Zemb commented on HBASE-23073: - I added a patch for branch-1 [~wchevreuil] > Add an optional costFunction to balance regions according to a capacity rule > > > Key: HBASE-23073 > URL: https://issues.apache.org/jira/browse/HBASE-23073 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 3.0.0 >Reporter: Pierre Zemb >Assignee: Pierre Zemb >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > Attachments: HBASE-23073.branch-1.001.patch > > > Based on the work in > [HBASE-22618|https://issues.apache.org/jira/browse/HBASE-22618], users can > now load custom costFunctions inside the main balancer used by HBase. As an > example, we like like to add upstream an optional cost function called > HeterogeneousRegionCountCostFunction that will deal with our issue: how to > balance regions according to the capacity of a RS instead of using the > RegionCountSkewCostFunction that is trying to avoid skew. > A rule file is loaded from HDFS before balancing. It contains lines of rules. > A rule is composed of a regexp for hostname, and a limit. For example, we > could have: > * rs[0-9] 200 > * rs1[0-9] 50 > RegionServers with hostname matching the first rules will have a limit of > 200, and the others 50. If there's no match, a default is set. > Thanks to the rule, we have two informations: the max number of regions for > this cluster, and the rules for each servers. HeterogeneousBalancer will try > to balance regions according to their capacity. > Let's take an example. Let's say that we have 20 RS: > 10 RS, named through rs0 to rs9 loaded with 60 regions each, and each can > handle 200 regions. > 10 RS, named through rs10 to rs19 loaded with 60 regions each, and each > can support 50 regions. > Based on the following rules: > rs[0-9] 200 > rs1[0-9] 50 > The second group is overloaded, whereas the first group has plenty of space. > Moving a region from the first group to the second should provide a lower > cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
