[GitHub] [hbase] Apache-HBase commented on pull request #4106: HBASE-26545 Implement tracing of scan
Apache-HBase commented on pull request #4106: URL: https://github.com/apache/hbase/pull/4106#issuecomment-1061533398 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 51s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 51s | master passed | | +1 :green_heart: | compile | 1m 44s | master passed | | +1 :green_heart: | shadedjars | 3m 38s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 14s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 34s | the patch passed | | +1 :green_heart: | compile | 1m 45s | the patch passed | | +1 :green_heart: | javac | 1m 45s | the patch passed | | +1 :green_heart: | shadedjars | 3m 35s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 21s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 431m 45s | root in the patch passed. | | | | 457m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4106/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4106 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9020ebc7aa57 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / bcd9a9acef | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4106/6/testReport/ | | Max. process+thread count | 2674 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4106/6/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26042) WAL lockup on 'sync failed' org.apache.hbase.thirdparty.io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
[
https://issues.apache.org/jira/browse/HBASE-26042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoit Sigoure updated HBASE-26042:
---
Affects Version/s: 2.4.8
> WAL lockup on 'sync failed'
> org.apache.hbase.thirdparty.io.netty.channel.unix.Errors$NativeIoException:
> readAddress(..) failed: Connection reset by peer
>
>
> Key: HBASE-26042
> URL: https://issues.apache.org/jira/browse/HBASE-26042
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.5, 2.4.8
>Reporter: Michael Stack
>Priority: Major
> Attachments: HBASE-26042-test-repro.patch, debug-dump.txt,
> hbase-cvp-regionserver-cvp328.sjc.aristanetworks.com.log, js1, js2
>
>
> Making note of issue seen in production cluster.
> Node had been struggling under load for a few days with slow syncs up to 10
> seconds, a few STUCK MVCCs from which it recovered and some java pauses up to
> three seconds in length.
> Then the below happened:
> {code:java}
> 2021-06-27 13:41:27,604 WARN [AsyncFSWAL-0-hdfs://:8020/hbase]
> wal.AsyncFSWAL: sync
> failedorg.apache.hbase.thirdparty.io.netty.channel.unix.Errors$NativeIoException:
> readAddress(..) failed: Connection reset by peer {code}
> ... and WAL turned dead in the water. Scanners start expiring. RPC prints
> text versions of requests complaining requestsTooSlow. Then we start to see
> these:
> {code:java}
> org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync
> result after 30 ms for txid=552128301, WAL system stuck? {code}
> Whats supposed to happen when other side goes away like this is that we will
> roll the WAL – go set up a new one. You can see it happening if you run
> {code:java}
> mvn test
> -Dtest=org.apache.hadoop.hbase.regionserver.wal.TestAsyncFSWAL#testBrokenWriter
> {code}
> I tried hacking the test to repro the above hang by throwing same exception
> in above test (on linux because need epoll to repro) but all just worked.
> Thread dumps of the hungup WAL subsystem are a little odd. The log roller is
> stuck w/o timeout trying to write a long on the WAL header:
>
> {code:java}
> Thread 9464: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=175 (Compiled frame)
> - java.util.concurrent.CompletableFuture$Signaller.block() @bci=19,
> line=1707 (Compiled frame)
> -
> java.util.concurrent.ForkJoinPool.managedBlock(java.util.concurrent.ForkJoinPool$ManagedBlocker)
> @bci=119, line=3323 (Compiled frame)
> - java.util.concurrent.CompletableFuture.waitingGet(boolean) @bci=115,
> line=1742 (Compiled frame)
> - java.util.concurrent.CompletableFuture.get() @bci=11, line=1908 (Compiled
> frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(java.util.function.Consumer)
> @bci=16, line=189 (Compiled frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeMagicAndWALHeader(byte[],
> org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos$WALHeader)
> @bci=9, line=202 (Compiled frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(org.apache.hadoop.fs.FileSystem,
> org.apache.hadoop.fs.Path, org.apache.hadoop.conf.Configuration, boolean,
> long) @bci=107, line=170 (Compiled frame)
> -
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(org.apache.hadoop.conf.Configuration,
> org.apache.hadoop.fs.FileSystem, org.apache.hadoop.fs.Path, boolean, long,
> org.apache.hbase.thirdparty.io.netty.channel.EventLoopGroup, java.lang.Class)
> @bci=61, line=113 (Compiled frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(org.apache.hadoop.fs.Path)
> @bci=22, line=651 (Compiled frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(org.apache.hadoop.fs.Path)
> @bci=2, line=128 (Compiled frame)
> - org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(boolean)
> @bci=101, line=797 (Compiled frame)
> - org.apache.hadoop.hbase.wal.AbstractWALRoller$RollController.rollWal(long)
> @bci=18, line=263 (Compiled frame)
> - org.apache.hadoop.hbase.wal.AbstractWALRoller.run() @bci=198, line=179
> (Compiled frame) {code}
>
> Other threads are BLOCKED trying to append the WAL w/ flush markers etc.
> unable to add the ringbuffer:
>
> {code:java}
> Thread 9465: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.parkNanos(long) @bci=11, line=338
> (Compiled frame)
> - com.lmax.disruptor.MultiProducerSequen
[jira] [Updated] (HBASE-26042) WAL lockup on 'sync failed' org.apache.hbase.thirdparty.io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
[
https://issues.apache.org/jira/browse/HBASE-26042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoit Sigoure updated HBASE-26042:
---
Attachment: debug-dump.txt
> WAL lockup on 'sync failed'
> org.apache.hbase.thirdparty.io.netty.channel.unix.Errors$NativeIoException:
> readAddress(..) failed: Connection reset by peer
>
>
> Key: HBASE-26042
> URL: https://issues.apache.org/jira/browse/HBASE-26042
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.5
>Reporter: Michael Stack
>Priority: Major
> Attachments: HBASE-26042-test-repro.patch, debug-dump.txt,
> hbase-cvp-regionserver-cvp328.sjc.aristanetworks.com.log, js1, js2
>
>
> Making note of issue seen in production cluster.
> Node had been struggling under load for a few days with slow syncs up to 10
> seconds, a few STUCK MVCCs from which it recovered and some java pauses up to
> three seconds in length.
> Then the below happened:
> {code:java}
> 2021-06-27 13:41:27,604 WARN [AsyncFSWAL-0-hdfs://:8020/hbase]
> wal.AsyncFSWAL: sync
> failedorg.apache.hbase.thirdparty.io.netty.channel.unix.Errors$NativeIoException:
> readAddress(..) failed: Connection reset by peer {code}
> ... and WAL turned dead in the water. Scanners start expiring. RPC prints
> text versions of requests complaining requestsTooSlow. Then we start to see
> these:
> {code:java}
> org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync
> result after 30 ms for txid=552128301, WAL system stuck? {code}
> Whats supposed to happen when other side goes away like this is that we will
> roll the WAL – go set up a new one. You can see it happening if you run
> {code:java}
> mvn test
> -Dtest=org.apache.hadoop.hbase.regionserver.wal.TestAsyncFSWAL#testBrokenWriter
> {code}
> I tried hacking the test to repro the above hang by throwing same exception
> in above test (on linux because need epoll to repro) but all just worked.
> Thread dumps of the hungup WAL subsystem are a little odd. The log roller is
> stuck w/o timeout trying to write a long on the WAL header:
>
> {code:java}
> Thread 9464: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=175 (Compiled frame)
> - java.util.concurrent.CompletableFuture$Signaller.block() @bci=19,
> line=1707 (Compiled frame)
> -
> java.util.concurrent.ForkJoinPool.managedBlock(java.util.concurrent.ForkJoinPool$ManagedBlocker)
> @bci=119, line=3323 (Compiled frame)
> - java.util.concurrent.CompletableFuture.waitingGet(boolean) @bci=115,
> line=1742 (Compiled frame)
> - java.util.concurrent.CompletableFuture.get() @bci=11, line=1908 (Compiled
> frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(java.util.function.Consumer)
> @bci=16, line=189 (Compiled frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeMagicAndWALHeader(byte[],
> org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos$WALHeader)
> @bci=9, line=202 (Compiled frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(org.apache.hadoop.fs.FileSystem,
> org.apache.hadoop.fs.Path, org.apache.hadoop.conf.Configuration, boolean,
> long) @bci=107, line=170 (Compiled frame)
> -
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(org.apache.hadoop.conf.Configuration,
> org.apache.hadoop.fs.FileSystem, org.apache.hadoop.fs.Path, boolean, long,
> org.apache.hbase.thirdparty.io.netty.channel.EventLoopGroup, java.lang.Class)
> @bci=61, line=113 (Compiled frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(org.apache.hadoop.fs.Path)
> @bci=22, line=651 (Compiled frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(org.apache.hadoop.fs.Path)
> @bci=2, line=128 (Compiled frame)
> - org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(boolean)
> @bci=101, line=797 (Compiled frame)
> - org.apache.hadoop.hbase.wal.AbstractWALRoller$RollController.rollWal(long)
> @bci=18, line=263 (Compiled frame)
> - org.apache.hadoop.hbase.wal.AbstractWALRoller.run() @bci=198, line=179
> (Compiled frame) {code}
>
> Other threads are BLOCKED trying to append the WAL w/ flush markers etc.
> unable to add the ringbuffer:
>
> {code:java}
> Thread 9465: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.parkNanos(long) @bci=11, line=338
> (Compiled frame)
> - com.lmax.disruptor.MultiProducerSequencer.n
[jira] [Updated] (HBASE-26042) WAL lockup on 'sync failed' org.apache.hbase.thirdparty.io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
[
https://issues.apache.org/jira/browse/HBASE-26042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoit Sigoure updated HBASE-26042:
---
Attachment: hbase-cvp-regionserver-cvp328.sjc.aristanetworks.com.log
> WAL lockup on 'sync failed'
> org.apache.hbase.thirdparty.io.netty.channel.unix.Errors$NativeIoException:
> readAddress(..) failed: Connection reset by peer
>
>
> Key: HBASE-26042
> URL: https://issues.apache.org/jira/browse/HBASE-26042
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.5
>Reporter: Michael Stack
>Priority: Major
> Attachments: HBASE-26042-test-repro.patch, debug-dump.txt,
> hbase-cvp-regionserver-cvp328.sjc.aristanetworks.com.log, js1, js2
>
>
> Making note of issue seen in production cluster.
> Node had been struggling under load for a few days with slow syncs up to 10
> seconds, a few STUCK MVCCs from which it recovered and some java pauses up to
> three seconds in length.
> Then the below happened:
> {code:java}
> 2021-06-27 13:41:27,604 WARN [AsyncFSWAL-0-hdfs://:8020/hbase]
> wal.AsyncFSWAL: sync
> failedorg.apache.hbase.thirdparty.io.netty.channel.unix.Errors$NativeIoException:
> readAddress(..) failed: Connection reset by peer {code}
> ... and WAL turned dead in the water. Scanners start expiring. RPC prints
> text versions of requests complaining requestsTooSlow. Then we start to see
> these:
> {code:java}
> org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync
> result after 30 ms for txid=552128301, WAL system stuck? {code}
> Whats supposed to happen when other side goes away like this is that we will
> roll the WAL – go set up a new one. You can see it happening if you run
> {code:java}
> mvn test
> -Dtest=org.apache.hadoop.hbase.regionserver.wal.TestAsyncFSWAL#testBrokenWriter
> {code}
> I tried hacking the test to repro the above hang by throwing same exception
> in above test (on linux because need epoll to repro) but all just worked.
> Thread dumps of the hungup WAL subsystem are a little odd. The log roller is
> stuck w/o timeout trying to write a long on the WAL header:
>
> {code:java}
> Thread 9464: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=175 (Compiled frame)
> - java.util.concurrent.CompletableFuture$Signaller.block() @bci=19,
> line=1707 (Compiled frame)
> -
> java.util.concurrent.ForkJoinPool.managedBlock(java.util.concurrent.ForkJoinPool$ManagedBlocker)
> @bci=119, line=3323 (Compiled frame)
> - java.util.concurrent.CompletableFuture.waitingGet(boolean) @bci=115,
> line=1742 (Compiled frame)
> - java.util.concurrent.CompletableFuture.get() @bci=11, line=1908 (Compiled
> frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(java.util.function.Consumer)
> @bci=16, line=189 (Compiled frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeMagicAndWALHeader(byte[],
> org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos$WALHeader)
> @bci=9, line=202 (Compiled frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(org.apache.hadoop.fs.FileSystem,
> org.apache.hadoop.fs.Path, org.apache.hadoop.conf.Configuration, boolean,
> long) @bci=107, line=170 (Compiled frame)
> -
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(org.apache.hadoop.conf.Configuration,
> org.apache.hadoop.fs.FileSystem, org.apache.hadoop.fs.Path, boolean, long,
> org.apache.hbase.thirdparty.io.netty.channel.EventLoopGroup, java.lang.Class)
> @bci=61, line=113 (Compiled frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(org.apache.hadoop.fs.Path)
> @bci=22, line=651 (Compiled frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(org.apache.hadoop.fs.Path)
> @bci=2, line=128 (Compiled frame)
> - org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(boolean)
> @bci=101, line=797 (Compiled frame)
> - org.apache.hadoop.hbase.wal.AbstractWALRoller$RollController.rollWal(long)
> @bci=18, line=263 (Compiled frame)
> - org.apache.hadoop.hbase.wal.AbstractWALRoller.run() @bci=198, line=179
> (Compiled frame) {code}
>
> Other threads are BLOCKED trying to append the WAL w/ flush markers etc.
> unable to add the ringbuffer:
>
> {code:java}
> Thread 9465: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.parkNanos(long) @bci=11, line=338
> (Compiled frame)
> - c
[jira] [Commented] (HBASE-26042) WAL lockup on 'sync failed' org.apache.hbase.thirdparty.io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
[
https://issues.apache.org/jira/browse/HBASE-26042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17502810#comment-17502810
]
Benoit Sigoure commented on HBASE-26042:
We've run into this issue on a test cluster with HBase 2.4.8.
Let me know if I can collect anything else to help you, as things are still
stuck right now and we can probably keep it untouched for another day or two as
it's a test cluster.
> WAL lockup on 'sync failed'
> org.apache.hbase.thirdparty.io.netty.channel.unix.Errors$NativeIoException:
> readAddress(..) failed: Connection reset by peer
>
>
> Key: HBASE-26042
> URL: https://issues.apache.org/jira/browse/HBASE-26042
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.5, 2.4.8
>Reporter: Michael Stack
>Priority: Major
> Attachments: HBASE-26042-test-repro.patch, debug-dump.txt,
> hbase-cvp-regionserver-cvp328.sjc.aristanetworks.com.log, js1, js2
>
>
> Making note of issue seen in production cluster.
> Node had been struggling under load for a few days with slow syncs up to 10
> seconds, a few STUCK MVCCs from which it recovered and some java pauses up to
> three seconds in length.
> Then the below happened:
> {code:java}
> 2021-06-27 13:41:27,604 WARN [AsyncFSWAL-0-hdfs://:8020/hbase]
> wal.AsyncFSWAL: sync
> failedorg.apache.hbase.thirdparty.io.netty.channel.unix.Errors$NativeIoException:
> readAddress(..) failed: Connection reset by peer {code}
> ... and WAL turned dead in the water. Scanners start expiring. RPC prints
> text versions of requests complaining requestsTooSlow. Then we start to see
> these:
> {code:java}
> org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync
> result after 30 ms for txid=552128301, WAL system stuck? {code}
> Whats supposed to happen when other side goes away like this is that we will
> roll the WAL – go set up a new one. You can see it happening if you run
> {code:java}
> mvn test
> -Dtest=org.apache.hadoop.hbase.regionserver.wal.TestAsyncFSWAL#testBrokenWriter
> {code}
> I tried hacking the test to repro the above hang by throwing same exception
> in above test (on linux because need epoll to repro) but all just worked.
> Thread dumps of the hungup WAL subsystem are a little odd. The log roller is
> stuck w/o timeout trying to write a long on the WAL header:
>
> {code:java}
> Thread 9464: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=175 (Compiled frame)
> - java.util.concurrent.CompletableFuture$Signaller.block() @bci=19,
> line=1707 (Compiled frame)
> -
> java.util.concurrent.ForkJoinPool.managedBlock(java.util.concurrent.ForkJoinPool$ManagedBlocker)
> @bci=119, line=3323 (Compiled frame)
> - java.util.concurrent.CompletableFuture.waitingGet(boolean) @bci=115,
> line=1742 (Compiled frame)
> - java.util.concurrent.CompletableFuture.get() @bci=11, line=1908 (Compiled
> frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(java.util.function.Consumer)
> @bci=16, line=189 (Compiled frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeMagicAndWALHeader(byte[],
> org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos$WALHeader)
> @bci=9, line=202 (Compiled frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(org.apache.hadoop.fs.FileSystem,
> org.apache.hadoop.fs.Path, org.apache.hadoop.conf.Configuration, boolean,
> long) @bci=107, line=170 (Compiled frame)
> -
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(org.apache.hadoop.conf.Configuration,
> org.apache.hadoop.fs.FileSystem, org.apache.hadoop.fs.Path, boolean, long,
> org.apache.hbase.thirdparty.io.netty.channel.EventLoopGroup, java.lang.Class)
> @bci=61, line=113 (Compiled frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(org.apache.hadoop.fs.Path)
> @bci=22, line=651 (Compiled frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(org.apache.hadoop.fs.Path)
> @bci=2, line=128 (Compiled frame)
> - org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(boolean)
> @bci=101, line=797 (Compiled frame)
> - org.apache.hadoop.hbase.wal.AbstractWALRoller$RollController.rollWal(long)
> @bci=18, line=263 (Compiled frame)
> - org.apache.hadoop.hbase.wal.AbstractWALRoller.run() @bci=198, line=179
> (Compiled frame) {code}
>
> Other threads are BLOCKED trying to append the WAL w/ flush markers etc.
> unable to add the ringbuffer:
>
> {code:java}
>
[GitHub] [hbase] Apache-HBase commented on pull request #4118: HBASE-26582 Prune use of Random and SecureRandom objects
Apache-HBase commented on pull request #4118: URL: https://github.com/apache/hbase/pull/4118#issuecomment-1061550659 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 48s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 50s | master passed | | +1 :green_heart: | compile | 3m 53s | master passed | | +1 :green_heart: | shadedjars | 4m 48s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 31s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 10s | the patch passed | | +1 :green_heart: | compile | 4m 27s | the patch passed | | +1 :green_heart: | javac | 4m 27s | the patch passed | | +1 :green_heart: | shadedjars | 4m 23s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 15s | hbase-common generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 27s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 0m 20s | hbase-metrics in the patch passed. | | +1 :green_heart: | unit | 0m 49s | hbase-hadoop-compat in the patch passed. | | +1 :green_heart: | unit | 1m 56s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 0m 40s | hbase-zookeeper in the patch passed. | | +1 :green_heart: | unit | 0m 30s | hbase-replication in the patch passed. | | +1 :green_heart: | unit | 18m 57s | hbase-balancer in the patch passed. | | +1 :green_heart: | unit | 0m 52s | hbase-http in the patch passed. | | +1 :green_heart: | unit | 1m 12s | hbase-asyncfs in the patch passed. | | +1 :green_heart: | unit | 2m 21s | hbase-procedure in the patch passed. | | -1 :x: | unit | 345m 46s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 20m 20s | hbase-mapreduce in the patch passed. | | +1 :green_heart: | unit | 1m 34s | hbase-testing-util in the patch passed. | | +1 :green_heart: | unit | 0m 44s | hbase-it in the patch passed. | | +1 :green_heart: | unit | 5m 34s | hbase-rest in the patch passed. | | | | 440m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4118/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4118 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 03c24ab5ceb9 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / bcd9a9acef | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javadoc | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4118/3/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-common.txt | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4118/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4118/3/testReport/ | | Max. process+thread count | 3479 (vs. ulimit of 3) | | modules | C: hbase-common hbase-metrics hbase-hadoop-compat hbase-client hbase-zookeeper hbase-replication hbase-balancer hbase-http hbase-asyncfs hbase-procedure hbase-server hbase-mapreduce hbase-testing-util hbase-it hbase-rest U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4118/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26042) WAL lockup on 'sync failed'
[
https://issues.apache.org/jira/browse/HBASE-26042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoit Sigoure updated HBASE-26042:
---
Summary: WAL lockup on 'sync failed' (was: WAL lockup on 'sync failed'
org.apache.hbase.thirdparty.io.netty.channel.unix.Errors$NativeIoException:
readAddress(..) failed: Connection reset by peer)
> WAL lockup on 'sync failed'
> ---
>
> Key: HBASE-26042
> URL: https://issues.apache.org/jira/browse/HBASE-26042
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.5, 2.4.8
>Reporter: Michael Stack
>Priority: Major
> Attachments: HBASE-26042-test-repro.patch, debug-dump.txt,
> hbase-cvp-regionserver-cvp328.sjc.aristanetworks.com.log, js1, js2
>
>
> Making note of issue seen in production cluster.
> Node had been struggling under load for a few days with slow syncs up to 10
> seconds, a few STUCK MVCCs from which it recovered and some java pauses up to
> three seconds in length.
> Then the below happened:
> {code:java}
> 2021-06-27 13:41:27,604 WARN [AsyncFSWAL-0-hdfs://:8020/hbase]
> wal.AsyncFSWAL: sync
> failedorg.apache.hbase.thirdparty.io.netty.channel.unix.Errors$NativeIoException:
> readAddress(..) failed: Connection reset by peer {code}
> ... and WAL turned dead in the water. Scanners start expiring. RPC prints
> text versions of requests complaining requestsTooSlow. Then we start to see
> these:
> {code:java}
> org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync
> result after 30 ms for txid=552128301, WAL system stuck? {code}
> Whats supposed to happen when other side goes away like this is that we will
> roll the WAL – go set up a new one. You can see it happening if you run
> {code:java}
> mvn test
> -Dtest=org.apache.hadoop.hbase.regionserver.wal.TestAsyncFSWAL#testBrokenWriter
> {code}
> I tried hacking the test to repro the above hang by throwing same exception
> in above test (on linux because need epoll to repro) but all just worked.
> Thread dumps of the hungup WAL subsystem are a little odd. The log roller is
> stuck w/o timeout trying to write a long on the WAL header:
>
> {code:java}
> Thread 9464: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=175 (Compiled frame)
> - java.util.concurrent.CompletableFuture$Signaller.block() @bci=19,
> line=1707 (Compiled frame)
> -
> java.util.concurrent.ForkJoinPool.managedBlock(java.util.concurrent.ForkJoinPool$ManagedBlocker)
> @bci=119, line=3323 (Compiled frame)
> - java.util.concurrent.CompletableFuture.waitingGet(boolean) @bci=115,
> line=1742 (Compiled frame)
> - java.util.concurrent.CompletableFuture.get() @bci=11, line=1908 (Compiled
> frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(java.util.function.Consumer)
> @bci=16, line=189 (Compiled frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeMagicAndWALHeader(byte[],
> org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos$WALHeader)
> @bci=9, line=202 (Compiled frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(org.apache.hadoop.fs.FileSystem,
> org.apache.hadoop.fs.Path, org.apache.hadoop.conf.Configuration, boolean,
> long) @bci=107, line=170 (Compiled frame)
> -
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(org.apache.hadoop.conf.Configuration,
> org.apache.hadoop.fs.FileSystem, org.apache.hadoop.fs.Path, boolean, long,
> org.apache.hbase.thirdparty.io.netty.channel.EventLoopGroup, java.lang.Class)
> @bci=61, line=113 (Compiled frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(org.apache.hadoop.fs.Path)
> @bci=22, line=651 (Compiled frame)
> -
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(org.apache.hadoop.fs.Path)
> @bci=2, line=128 (Compiled frame)
> - org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(boolean)
> @bci=101, line=797 (Compiled frame)
> - org.apache.hadoop.hbase.wal.AbstractWALRoller$RollController.rollWal(long)
> @bci=18, line=263 (Compiled frame)
> - org.apache.hadoop.hbase.wal.AbstractWALRoller.run() @bci=198, line=179
> (Compiled frame) {code}
>
> Other threads are BLOCKED trying to append the WAL w/ flush markers etc.
> unable to add the ringbuffer:
>
> {code:java}
> Thread 9465: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.parkNanos(long) @bci=11, line=338
> (Compiled frame)
> - com.lmax.disruptor.MultiProducerSequencer.next(int) @bci=82, line=136
> (Compiled frame)
> - com.lmax.disruptor.Multi
[GitHub] [hbase] Apache-HBase commented on pull request #4182: HBASE-26811 Secondary replica may be disabled for read forever
Apache-HBase commented on pull request #4182: URL: https://github.com/apache/hbase/pull/4182#issuecomment-1061570087 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 58s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 23s | master passed | | +1 :green_heart: | compile | 1m 13s | master passed | | +1 :green_heart: | shadedjars | 4m 16s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 53s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 18s | the patch passed | | +1 :green_heart: | compile | 1m 13s | the patch passed | | +1 :green_heart: | javac | 1m 13s | the patch passed | | +1 :green_heart: | shadedjars | 4m 20s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 44s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 59s | hbase-client in the patch passed. | | -1 :x: | unit | 317m 12s | hbase-server in the patch failed. | | | | 341m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4182/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4182 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b0b091cb5a8d 5.4.0-1043-aws #45~18.04.1-Ubuntu SMP Fri Apr 9 23:32:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 39ecaa1975 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4182/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4182/1/testReport/ | | Max. process+thread count | 3312 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4182/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4182: HBASE-26811 Secondary replica may be disabled for read forever
Apache-HBase commented on pull request #4182: URL: https://github.com/apache/hbase/pull/4182#issuecomment-1061572803 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 40s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 45s | Maven dependency ordering for branch | | -1 :x: | mvninstall | 1m 52s | root in master failed. | | +1 :green_heart: | compile | 0m 49s | master passed | | +1 :green_heart: | shadedjars | 3m 50s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 34s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 21s | the patch passed | | +1 :green_heart: | compile | 0m 49s | the patch passed | | +1 :green_heart: | javac | 0m 49s | the patch passed | | +1 :green_heart: | shadedjars | 3m 49s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 34s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 33s | hbase-client in the patch passed. | | -1 :x: | unit | 325m 40s | hbase-server in the patch failed. | | | | 344m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4182/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4182 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6274825de575 5.4.0-1054-aws #57~18.04.1-Ubuntu SMP Thu Jul 15 03:21:36 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 39ecaa1975 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | mvninstall | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4182/1/artifact/yetus-jdk8-hadoop3-check/output/branch-mvninstall-root.txt | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4182/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4182/1/testReport/ | | Max. process+thread count | 2977 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4182/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] jojochuang commented on a change in pull request #4182: HBASE-26811 Secondary replica may be disabled for read forever
jojochuang commented on a change in pull request #4182:
URL: https://github.com/apache/hbase/pull/4182#discussion_r821475575
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicaWaitForPrimaryFlushConf.java
##
@@ -0,0 +1,120 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtil;
+import org.apache.hadoop.hbase.StartTestingClusterOption;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.TableNameTestRule;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.executor.ExecutorType;
+import org.apache.hadoop.hbase.regionserver.Region.Operation;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testclassification.RegionServerTests;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hbase.util.ServerRegionReplicaUtil;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+@Category({ RegionServerTests.class, MediumTests.class })
+public class TestRegionReplicaWaitForPrimaryFlushConf {
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+ HBaseClassTestRule.forClass(TestReplicateToReplica.class);
+
+ private static byte[] FAMILY = Bytes.toBytes("family");
+
+ private TableName tableName;
+
+ @Rule
+ public final TableNameTestRule name = new TableNameTestRule();
+ private static final HBaseTestingUtil HTU = new HBaseTestingUtil();
+
+ @BeforeClass
+ public static void setUpBeforeClass() throws Exception {
+Configuration conf = HTU.getConfiguration();
+
conf.setBoolean(ServerRegionReplicaUtil.REGION_REPLICA_REPLICATION_CONF_KEY,
true);
+
conf.setBoolean(ServerRegionReplicaUtil.REGION_REPLICA_WAIT_FOR_PRIMARY_FLUSH_CONF_KEY,
false);
+
HTU.startMiniCluster(StartTestingClusterOption.builder().numRegionServers(2).build());
+
+ }
+
+ @AfterClass
+ public static void tearDownAfterClass() throws Exception {
+HTU.shutdownMiniCluster();
+ }
+
+ @Test
+ public void test() throws Exception {
+tableName = name.getTableName();
+TableDescriptor tableDescriptor =
TableDescriptorBuilder.newBuilder(tableName)
+
.setRegionReplication(2).setColumnFamily(ColumnFamilyDescriptorBuilder.of(FAMILY))
+.setRegionMemStoreReplication(true).build();
+HTU.getAdmin().createTable(tableDescriptor);
+
+final ArrayList> regionAndRegionServers =
+new ArrayList>(Arrays.asList(null, null));
+
+for (int i = 0; i < 2; i++) {
+ HRegionServer rs = HTU.getMiniHBaseCluster().getRegionServer(i);
+ List onlineRegions = rs.getRegions(tableName);
+ for (HRegion region : onlineRegions) {
+int replicaId = region.getRegionInfo().getReplicaId();
+assertTrue(regionAndRegionServers.get(replicaId) == null);
Review comment:
use assertNull()
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicaWaitForPrimaryFlushConf.java
##
@@ -0,0 +1,120 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+
[GitHub] [hbase] comnetwork commented on a change in pull request #4182: HBASE-26811 Secondary replica may be disabled for read forever
comnetwork commented on a change in pull request #4182:
URL: https://github.com/apache/hbase/pull/4182#discussion_r821492296
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicaWaitForPrimaryFlushConf.java
##
@@ -0,0 +1,120 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtil;
+import org.apache.hadoop.hbase.StartTestingClusterOption;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.TableNameTestRule;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.executor.ExecutorType;
+import org.apache.hadoop.hbase.regionserver.Region.Operation;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testclassification.RegionServerTests;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hbase.util.ServerRegionReplicaUtil;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+@Category({ RegionServerTests.class, MediumTests.class })
+public class TestRegionReplicaWaitForPrimaryFlushConf {
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+ HBaseClassTestRule.forClass(TestReplicateToReplica.class);
+
+ private static byte[] FAMILY = Bytes.toBytes("family");
+
+ private TableName tableName;
+
+ @Rule
+ public final TableNameTestRule name = new TableNameTestRule();
+ private static final HBaseTestingUtil HTU = new HBaseTestingUtil();
+
+ @BeforeClass
+ public static void setUpBeforeClass() throws Exception {
+Configuration conf = HTU.getConfiguration();
+
conf.setBoolean(ServerRegionReplicaUtil.REGION_REPLICA_REPLICATION_CONF_KEY,
true);
+
conf.setBoolean(ServerRegionReplicaUtil.REGION_REPLICA_WAIT_FOR_PRIMARY_FLUSH_CONF_KEY,
false);
+
HTU.startMiniCluster(StartTestingClusterOption.builder().numRegionServers(2).build());
+
+ }
+
+ @AfterClass
+ public static void tearDownAfterClass() throws Exception {
+HTU.shutdownMiniCluster();
+ }
+
+ @Test
+ public void test() throws Exception {
+tableName = name.getTableName();
+TableDescriptor tableDescriptor =
TableDescriptorBuilder.newBuilder(tableName)
+
.setRegionReplication(2).setColumnFamily(ColumnFamilyDescriptorBuilder.of(FAMILY))
+.setRegionMemStoreReplication(true).build();
+HTU.getAdmin().createTable(tableDescriptor);
+
+final ArrayList> regionAndRegionServers =
+new ArrayList>(Arrays.asList(null, null));
+
+for (int i = 0; i < 2; i++) {
+ HRegionServer rs = HTU.getMiniHBaseCluster().getRegionServer(i);
+ List onlineRegions = rs.getRegions(tableName);
+ for (HRegion region : onlineRegions) {
+int replicaId = region.getRegionInfo().getReplicaId();
+assertTrue(regionAndRegionServers.get(replicaId) == null);
+regionAndRegionServers.set(replicaId, new Pair(region, rs));
+ }
+}
+for (Pair pair : regionAndRegionServers) {
+ assertNotNull(pair);
+}
+
+HRegionServer secondaryRs = regionAndRegionServers.get(1).getSecond();
+
+try {
+ secondaryRs.getExecutorService()
+ .getExecutorThreadPool(ExecutorType.RS_REGION_REPLICA_FLUSH_OPS);
+ fail();
+} catch (NullPointerException e) {
+ assertTrue(e != null);
Review comment:
Yes, it is just suppress compiler style warn.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific co
[GitHub] [hbase] comnetwork commented on a change in pull request #4182: HBASE-26811 Secondary replica may be disabled for read forever
comnetwork commented on a change in pull request #4182:
URL: https://github.com/apache/hbase/pull/4182#discussion_r821534629
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicaWaitForPrimaryFlushConf.java
##
@@ -0,0 +1,120 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtil;
+import org.apache.hadoop.hbase.StartTestingClusterOption;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.TableNameTestRule;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.executor.ExecutorType;
+import org.apache.hadoop.hbase.regionserver.Region.Operation;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testclassification.RegionServerTests;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hbase.util.ServerRegionReplicaUtil;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+@Category({ RegionServerTests.class, MediumTests.class })
+public class TestRegionReplicaWaitForPrimaryFlushConf {
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+ HBaseClassTestRule.forClass(TestReplicateToReplica.class);
+
+ private static byte[] FAMILY = Bytes.toBytes("family");
+
+ private TableName tableName;
+
+ @Rule
+ public final TableNameTestRule name = new TableNameTestRule();
+ private static final HBaseTestingUtil HTU = new HBaseTestingUtil();
+
+ @BeforeClass
+ public static void setUpBeforeClass() throws Exception {
+Configuration conf = HTU.getConfiguration();
+
conf.setBoolean(ServerRegionReplicaUtil.REGION_REPLICA_REPLICATION_CONF_KEY,
true);
+
conf.setBoolean(ServerRegionReplicaUtil.REGION_REPLICA_WAIT_FOR_PRIMARY_FLUSH_CONF_KEY,
false);
+
HTU.startMiniCluster(StartTestingClusterOption.builder().numRegionServers(2).build());
+
+ }
+
+ @AfterClass
+ public static void tearDownAfterClass() throws Exception {
+HTU.shutdownMiniCluster();
+ }
+
+ @Test
+ public void test() throws Exception {
Review comment:
@jojochuang ,fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] comnetwork commented on a change in pull request #4182: HBASE-26811 Secondary replica may be disabled for read forever
comnetwork commented on a change in pull request #4182:
URL: https://github.com/apache/hbase/pull/4182#discussion_r821535557
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicaWaitForPrimaryFlushConf.java
##
@@ -0,0 +1,120 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtil;
+import org.apache.hadoop.hbase.StartTestingClusterOption;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.TableNameTestRule;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.executor.ExecutorType;
+import org.apache.hadoop.hbase.regionserver.Region.Operation;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testclassification.RegionServerTests;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hbase.util.ServerRegionReplicaUtil;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+@Category({ RegionServerTests.class, MediumTests.class })
+public class TestRegionReplicaWaitForPrimaryFlushConf {
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+ HBaseClassTestRule.forClass(TestReplicateToReplica.class);
+
+ private static byte[] FAMILY = Bytes.toBytes("family");
+
+ private TableName tableName;
+
+ @Rule
+ public final TableNameTestRule name = new TableNameTestRule();
+ private static final HBaseTestingUtil HTU = new HBaseTestingUtil();
+
+ @BeforeClass
+ public static void setUpBeforeClass() throws Exception {
+Configuration conf = HTU.getConfiguration();
+
conf.setBoolean(ServerRegionReplicaUtil.REGION_REPLICA_REPLICATION_CONF_KEY,
true);
+
conf.setBoolean(ServerRegionReplicaUtil.REGION_REPLICA_WAIT_FOR_PRIMARY_FLUSH_CONF_KEY,
false);
+
HTU.startMiniCluster(StartTestingClusterOption.builder().numRegionServers(2).build());
+
+ }
+
+ @AfterClass
+ public static void tearDownAfterClass() throws Exception {
+HTU.shutdownMiniCluster();
+ }
+
+ @Test
+ public void test() throws Exception {
+tableName = name.getTableName();
+TableDescriptor tableDescriptor =
TableDescriptorBuilder.newBuilder(tableName)
+
.setRegionReplication(2).setColumnFamily(ColumnFamilyDescriptorBuilder.of(FAMILY))
+.setRegionMemStoreReplication(true).build();
+HTU.getAdmin().createTable(tableDescriptor);
+
+final ArrayList> regionAndRegionServers =
+new ArrayList>(Arrays.asList(null, null));
+
+for (int i = 0; i < 2; i++) {
+ HRegionServer rs = HTU.getMiniHBaseCluster().getRegionServer(i);
+ List onlineRegions = rs.getRegions(tableName);
+ for (HRegion region : onlineRegions) {
+int replicaId = region.getRegionInfo().getReplicaId();
+assertTrue(regionAndRegionServers.get(replicaId) == null);
+regionAndRegionServers.set(replicaId, new Pair(region, rs));
+ }
+}
+for (Pair pair : regionAndRegionServers) {
+ assertNotNull(pair);
+}
+
+HRegionServer secondaryRs = regionAndRegionServers.get(1).getSecond();
+
+try {
+ secondaryRs.getExecutorService()
+ .getExecutorThreadPool(ExecutorType.RS_REGION_REPLICA_FLUSH_OPS);
+ fail();
+} catch (NullPointerException e) {
+ assertTrue(e != null);
+}
+
+HRegion secondaryRegion = regionAndRegionServers.get(1).getFirst();
+assertTrue(
Review comment:
@jojochuang ,fix it
--
This is an automated message from the Apache Git Service.
To respond to the messag
[GitHub] [hbase] comnetwork commented on a change in pull request #4182: HBASE-26811 Secondary replica may be disabled for read forever
comnetwork commented on a change in pull request #4182:
URL: https://github.com/apache/hbase/pull/4182#discussion_r821536045
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicaWaitForPrimaryFlushConf.java
##
@@ -0,0 +1,120 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtil;
+import org.apache.hadoop.hbase.StartTestingClusterOption;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.TableNameTestRule;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.executor.ExecutorType;
+import org.apache.hadoop.hbase.regionserver.Region.Operation;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testclassification.RegionServerTests;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hbase.util.ServerRegionReplicaUtil;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+@Category({ RegionServerTests.class, MediumTests.class })
+public class TestRegionReplicaWaitForPrimaryFlushConf {
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+ HBaseClassTestRule.forClass(TestReplicateToReplica.class);
+
+ private static byte[] FAMILY = Bytes.toBytes("family");
+
+ private TableName tableName;
+
+ @Rule
+ public final TableNameTestRule name = new TableNameTestRule();
+ private static final HBaseTestingUtil HTU = new HBaseTestingUtil();
+
+ @BeforeClass
+ public static void setUpBeforeClass() throws Exception {
+Configuration conf = HTU.getConfiguration();
+
conf.setBoolean(ServerRegionReplicaUtil.REGION_REPLICA_REPLICATION_CONF_KEY,
true);
+
conf.setBoolean(ServerRegionReplicaUtil.REGION_REPLICA_WAIT_FOR_PRIMARY_FLUSH_CONF_KEY,
false);
+
HTU.startMiniCluster(StartTestingClusterOption.builder().numRegionServers(2).build());
+
+ }
+
+ @AfterClass
+ public static void tearDownAfterClass() throws Exception {
+HTU.shutdownMiniCluster();
+ }
+
+ @Test
+ public void test() throws Exception {
+tableName = name.getTableName();
+TableDescriptor tableDescriptor =
TableDescriptorBuilder.newBuilder(tableName)
+
.setRegionReplication(2).setColumnFamily(ColumnFamilyDescriptorBuilder.of(FAMILY))
+.setRegionMemStoreReplication(true).build();
+HTU.getAdmin().createTable(tableDescriptor);
+
+final ArrayList> regionAndRegionServers =
+new ArrayList>(Arrays.asList(null, null));
+
+for (int i = 0; i < 2; i++) {
+ HRegionServer rs = HTU.getMiniHBaseCluster().getRegionServer(i);
+ List onlineRegions = rs.getRegions(tableName);
+ for (HRegion region : onlineRegions) {
+int replicaId = region.getRegionInfo().getReplicaId();
+assertTrue(regionAndRegionServers.get(replicaId) == null);
Review comment:
@jojochuang , fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4182: HBASE-26811 Secondary replica may be disabled for read forever
Apache-HBase commented on pull request #4182: URL: https://github.com/apache/hbase/pull/4182#issuecomment-1061672233 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 35s | master passed | | +1 :green_heart: | compile | 2m 53s | master passed | | +1 :green_heart: | checkstyle | 0m 57s | master passed | | +1 :green_heart: | spotbugs | 1m 56s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 24s | the patch passed | | +1 :green_heart: | compile | 2m 50s | the patch passed | | +1 :green_heart: | javac | 2m 50s | the patch passed | | +1 :green_heart: | checkstyle | 0m 53s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 33s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotbugs | 2m 7s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 19s | The patch does not generate ASF License warnings. | | | | 34m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4182/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4182 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux c56a424e72f9 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 39ecaa1975 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 60 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4182/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26760) LICENSE handling should not allow non-aggregated "apache-2.0"
[
https://issues.apache.org/jira/browse/HBASE-26760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17502879#comment-17502879
]
Hudson commented on HBASE-26760:
Results for branch branch-2
[build #477 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/477/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/477/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/477/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/477/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/477//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> LICENSE handling should not allow non-aggregated "apache-2.0"
> -
>
> Key: HBASE-26760
> URL: https://issues.apache.org/jira/browse/HBASE-26760
> Project: HBase
> Issue Type: Task
> Components: community
>Affects Versions: 2.4.3, 2.4.4, 2.4.5, 2.4.6, 2.4.7, 2.4.8, 2.4.9, 2.4.10
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.11
>
>
> our non-aggregated license check currently allows the malformed name
> "apache-2.0". All ALv2 licenses should be aggregated.
> - remove "apache-2.0" from the {{non_aggregate_fine}} list
> - figure out what dependencies are referring to this license and correct them
> in {{supplemental-models.xml}}
> nice to have: maybe add a precommit check that complains if the
> {{non_aggregate_fine}} set is changed?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-25709) Close region may stuck when region is compacting and skipped most cells read
[
https://issues.apache.org/jira/browse/HBASE-25709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17502877#comment-17502877
]
Hudson commented on HBASE-25709:
Results for branch branch-2
[build #477 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/477/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/477/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/477/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/477/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/477//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Close region may stuck when region is compacting and skipped most cells read
>
>
> Key: HBASE-25709
> URL: https://issues.apache.org/jira/browse/HBASE-25709
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Affects Versions: 1.7.1, 3.0.0-alpha-2, 2.4.10
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.11
>
> Attachments: Master-UI-RIT.png, RS-region-state.png
>
>
> We found in our cluster about stop region stuck. The region is compacting,
> and its store files has many TTL expired cells. Close region state
> marker(HRegion#writestate.writesEnabled) is not checked in compaction,
> because most cells were skipped.
> !RS-region-state.png|width=698,height=310!
>
> !Master-UI-RIT.png|width=693,height=157!
>
> HBASE-23968 has encountered similar problem, but the solution in it is outer
> the method
> InternalScanner#next(List result, ScannerContext scannerContext), which
> will not return if there are many skipped cells, for current compaction
> scanner context. As a result, we need to return in time in the next method,
> and then check the stop marker.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26552) Introduce retry to logroller to avoid abort
[
https://issues.apache.org/jira/browse/HBASE-26552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17502878#comment-17502878
]
Hudson commented on HBASE-26552:
Results for branch branch-2
[build #477 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/477/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/477/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/477/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/477/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/477//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Introduce retry to logroller to avoid abort
> ---
>
> Key: HBASE-26552
> URL: https://issues.apache.org/jira/browse/HBASE-26552
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 3.0.0-alpha-2, 2.4.10
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.11
>
>
> When calling RollController#rollWal in AbstractWALRoller, the regionserver
> may abort when encounters exception,
> {code:java}
> ...
> } catch (FailedLogCloseException | ConnectException e) {
> abort("Failed log close in log roller", e);
> } catch (IOException ex) {
> // Abort if we get here. We probably won't recover an IOE. HBASE-1132
> abort("IOE in log roller",
> ex instanceof RemoteException ? ((RemoteException)
> ex).unwrapRemoteException() : ex);
> } catch (Exception ex) {
> LOG.error("Log rolling failed", ex);
> abort("Log rolling failed", ex);
> } {code}
> I think we should support retry of rollWal here to avoid recovering the
> service by killing regionserver. The restart of regionserver is costly and
> very not friendly to the availability.
> I find that when creating new writer for the WAL in
> FanOutOneBlockAsyncDFSOutputHelper#createOutput, it supports retry to
> addBlock by setting this config "hbase.fs.async.create.retries". The idea of
> retry to roll WAL is similar to it, they both try best to make roll WAL
> succeed.
> But the initialization of new WAL writer also includes flushing the write
> buffer flush and waiting until it is completed by
> AsyncProtobufLogWriter#writeMagicAndWALHeader, which can also fail by some
> hardware reasons. The regionserver connected to the datanodes after addBlock,
> but that not means the magic and header can be flushed successfully.
> {code:java}
> protected long writeMagicAndWALHeader(byte[] magic, WALHeader header) throws
> IOException {
> return write(future -> {
> output.write(magic);
> try {
> header.writeDelimitedTo(asyncOutputWrapper);
> } catch (IOException e) {
> // should not happen
> throw new AssertionError(e);
> }
> addListener(output.flush(false), (len, error) -> {
> if (error != null) {
> future.completeExceptionally(error);
> } else {
> future.complete(len);
> }
> });
> });
> }{code}
> We have found that in our production clusters, there exists aborting of
> regionservers that caused by "IOE in log roller". And the practice in our
> clusters is that just one more retry of rollWal can make the WAL roll
> complete and continue serving.
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26784) Use HIGH_QOS for ResultScanner.close requests
[ https://issues.apache.org/jira/browse/HBASE-26784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17502914#comment-17502914 ] Bryan Beaudreault commented on HBASE-26784: --- Thanks for merging [~Xiaolin Ha] ! > Use HIGH_QOS for ResultScanner.close requests > - > > Key: HBASE-26784 > URL: https://issues.apache.org/jira/browse/HBASE-26784 > Project: HBase > Issue Type: Improvement >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Major > Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3 > > > Active scanners take resources on the RegionServer, and scanner leaks can > have a big impact on GC. Under high load, ResultScanner.close() calls might > get rejected or timed out before executing. Exceptions thrown by close() are > currently swallowed, so clients will not retry. This leads to a leak of > active scanners on the server side. > These calls should be relatively free to execute, and executing them will > free up the resources which can have an impact on overall RegionServer > performance. As such it can be beneficial to execute all > ResultScanner.close() requests with HIGH_QOS regardless of the priority of > the underlying Scan. > In a scan-heavy stress test I ran, after a couple minutes the Full GC time > (using G1GC) starts increasing and we start seeing 1s pauses, due to > increasing number of active scanners (into the 10s of thousands). Re-running > the stress test with HIGH_QOS for ResultScanner.close() eliminates the leak > and GC time remains stable. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #4181: HBASE-26810 Add dynamic configuration support for system coprocessors
Apache-HBase commented on pull request #4181: URL: https://github.com/apache/hbase/pull/4181#issuecomment-1061752850 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 42s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 32s | master passed | | +1 :green_heart: | compile | 5m 51s | master passed | | +1 :green_heart: | checkstyle | 1m 7s | master passed | | +0 :ok: | refguide | 1m 54s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotbugs | 7m 57s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 14s | the patch passed | | +1 :green_heart: | compile | 5m 53s | the patch passed | | +1 :green_heart: | javac | 5m 53s | the patch passed | | +1 :green_heart: | checkstyle | 1m 6s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +0 :ok: | refguide | 1m 32s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | hadoopcheck | 11m 5s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotbugs | 8m 6s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 17s | The patch does not generate ASF License warnings. | | | | 55m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4181/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4181 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile refguide | | uname | Linux 0e5381768656 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 39ecaa1975 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | refguide | https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-4181/2/yetus-general-check/output/branch-site/book.html | | refguide | https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-4181/2/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 87 (vs. ulimit of 3) | | modules | C: hbase-server . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4181/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #4182: HBASE-26811 Secondary replica may be disabled for read forever
Apache9 commented on a change in pull request #4182:
URL: https://github.com/apache/hbase/pull/4182#discussion_r821644223
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java
##
@@ -1335,11 +1335,14 @@ public boolean hasRegionMemStoreReplication() {
* @return the modifyable TD
*/
public ModifyableTableDescriptor setRegionMemStoreReplication(boolean
memstoreReplication) {
- setValue(REGION_MEMSTORE_REPLICATION_KEY,
Boolean.toString(memstoreReplication));
- // If the memstore replication is setup, we do not have to wait for
observing a flush event
+ ModifyableTableDescriptor returnDesc =
+ setValue(REGION_MEMSTORE_REPLICATION_KEY,
Boolean.toString(memstoreReplication));
+ // If the memstore replication not setup, we do not have to wait for
observing a flush event
// from primary before starting to serve reads, because gaps from
replication is not applicable
- return setValue(REGION_REPLICA_WAIT_FOR_PRIMARY_FLUSH_CONF_KEY,
- Boolean.toString(memstoreReplication));
+ if (!memstoreReplication) {
Review comment:
This is a behavior change? If memstoreReplication is true, we will not
set REGION_REPLICA_WAIT_FOR_PRIMARY_FLUSH_CONF_KEY to true?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #4166: HBASE-26802 Backport the log4j2 changes to branch-2
Apache9 commented on pull request #4166: URL: https://github.com/apache/hbase/pull/4166#issuecomment-1061761627 Let me try to execute the UTs locally to see what is the problem. Seems we always hit the timeout. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #4164: HBASE-25465 Use javac --release option for supporting cross version c…
Apache9 commented on pull request #4164: URL: https://github.com/apache/hbase/pull/4164#issuecomment-1061762866 The result is overall good. Let me reply on the mailing list. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4180: HBASE-26807 Unify CallQueueTooBigException special pause with CallDroppedException
Apache-HBase commented on pull request #4180: URL: https://github.com/apache/hbase/pull/4180#issuecomment-1061779760 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4180: HBASE-26807 Unify CallQueueTooBigException special pause with CallDroppedException
Apache-HBase commented on pull request #4180: URL: https://github.com/apache/hbase/pull/4180#issuecomment-1061780327 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | patch | 0m 2s | https://github.com/apache/hbase/pull/4180 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/4180 | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4180/3/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] comnetwork commented on a change in pull request #4182: HBASE-26811 Secondary replica may be disabled for read forever
comnetwork commented on a change in pull request #4182:
URL: https://github.com/apache/hbase/pull/4182#discussion_r821669668
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java
##
@@ -1335,11 +1335,14 @@ public boolean hasRegionMemStoreReplication() {
* @return the modifyable TD
*/
public ModifyableTableDescriptor setRegionMemStoreReplication(boolean
memstoreReplication) {
- setValue(REGION_MEMSTORE_REPLICATION_KEY,
Boolean.toString(memstoreReplication));
- // If the memstore replication is setup, we do not have to wait for
observing a flush event
+ ModifyableTableDescriptor returnDesc =
+ setValue(REGION_MEMSTORE_REPLICATION_KEY,
Boolean.toString(memstoreReplication));
+ // If the memstore replication not setup, we do not have to wait for
observing a flush event
// from primary before starting to serve reads, because gaps from
replication is not applicable
- return setValue(REGION_REPLICA_WAIT_FOR_PRIMARY_FLUSH_CONF_KEY,
- Boolean.toString(memstoreReplication));
+ if (!memstoreReplication) {
Review comment:
@Apache9 , yes, the fix of the problem described on jira is here, if we
explicitly set memstoreReplication to true at table level, we do not change the
REGION_REPLICA_WAIT_FOR_PRIMARY_FLUSH_CONF_KEY, keep its original config(the
default value is true). If REGION_REPLICA_WAIT_FOR_PRIMARY_FLUSH_CONF_KEY is
false and memstoreReplication is true, the secondary region is enabled for read
when open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] comnetwork commented on a change in pull request #4182: HBASE-26811 Secondary replica may be disabled for read forever
comnetwork commented on a change in pull request #4182:
URL: https://github.com/apache/hbase/pull/4182#discussion_r821669668
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java
##
@@ -1335,11 +1335,14 @@ public boolean hasRegionMemStoreReplication() {
* @return the modifyable TD
*/
public ModifyableTableDescriptor setRegionMemStoreReplication(boolean
memstoreReplication) {
- setValue(REGION_MEMSTORE_REPLICATION_KEY,
Boolean.toString(memstoreReplication));
- // If the memstore replication is setup, we do not have to wait for
observing a flush event
+ ModifyableTableDescriptor returnDesc =
+ setValue(REGION_MEMSTORE_REPLICATION_KEY,
Boolean.toString(memstoreReplication));
+ // If the memstore replication not setup, we do not have to wait for
observing a flush event
// from primary before starting to serve reads, because gaps from
replication is not applicable
- return setValue(REGION_REPLICA_WAIT_FOR_PRIMARY_FLUSH_CONF_KEY,
- Boolean.toString(memstoreReplication));
+ if (!memstoreReplication) {
Review comment:
@Apache9 , yes, the fix of the problem described on jira is here, if we
explicitly set memstoreReplication to true at table level(the default value is
true), we do not change the REGION_REPLICA_WAIT_FOR_PRIMARY_FLUSH_CONF_KEY,
keep its original config(the default value is also true). If
REGION_REPLICA_WAIT_FOR_PRIMARY_FLUSH_CONF_KEY is false and memstoreReplication
is true, the secondary region is enabled for read when open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-26522) Improve documentation of hbase 1.x to 2.x potential incompatibilities
[
https://issues.apache.org/jira/browse/HBASE-26522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17502973#comment-17502973
]
Bryan Beaudreault commented on HBASE-26522:
---
The most disruptive manifestation of that check (before we disabled it, so we
never tested prod) was https://issues.apache.org/jira/browse/HBASE-26575. I
agree 2.5.0 is an opportunity, but I think I'd have to do some pretty thorough
load testing to determine what a reasonable default for this is. I have it on a
list of things for my team to circle back to post-upgrade, but not sure if I
can fit that in near term. As mentioned in that issue, I think one of the
biggest problems with that feature is:
{quote}At this point I'll say that this in general seems overly aggressive,
especially since the StoreHotnessProtector doesn't actually do any checks for
actual load on the RS. You could have a totally idle RegionServer and submit a
single batch of 100 Puts with 101 columns each – if you don't have at least 5
retries configured, the batch will fail.
{quote}
I'm not entirely sure how to measure load on the RS at that level. Perhaps if
the counters it uses are only trending upwards for a period of time, rather
than going up then down as expected. Meaning writes are backing up. Would need
to look further.
> Improve documentation of hbase 1.x to 2.x potential incompatibilities
> -
>
> Key: HBASE-26522
> URL: https://issues.apache.org/jira/browse/HBASE-26522
> Project: HBase
> Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Minor
>
> We're working on a major upgrade of almost 900 tables across 100 production
> clusters (and corresponding QA environment clusters). We've upgraded about
> 25% of our QA environment and run into a series of incompatibilities along
> the way. Most of them have been easy to get around, but I wanted to create
> this Jira to collect them so that we can make an update to the docs for
> future upgraders.
> My plan is to periodically edit this description to add to the list. If
> anyone else has anything to contribute, feel free to edit as well or add a
> comment.
> Incompatibilities to document:
> - HBASE-15676 changed the serialized byte string used for the fuzzy mask.
> FuzzyRowFilters created by older clients will not match any rows in an hbase2
> cluster. This was fixed in HBASE-26537 but should be documented in our
> upgrade guide.
> - CDH5 try/catches bad HTableDescriptor.getDurability calls and returns
> USE_DEFAULT. In hbase2, if someone creates a table with a bad durability
> (i.e. DEFAULT instead of USE_DEFAULT), it results in a failure which causes
> the CreateTableProcedure to infinitely retries with no backoff. This rapid
> retry caused a bunch of pain on the cluster that encountered it, backing up
> datanode's ability to keep up with the millions of calls to create and delete
> .regioninfo files.
> - This isn't quite an incompatibility, but HBASE-19389 introduced a
> concurrency mitigation which may have surprising results coming from older
> versions. The defaults are pretty conservative – when writing more than 100
> columns, no more than 10 concurrent writes or 20 pending writes at once.
> - Increments sent from branch-1 clients may get erroneously stored with a
> timestamp of 0 on hbase2+ clusters: HBASE-26713
> - CheckAndMutate with a "null" compare value used to ignore CompareOp. Fixed
> in HBASE-26742, checkAndMutate affects may change between versions.
> - client will not know how to handle dangling rep_barrier rows in meta:
> HBASE-26797
> - the default hbase split policy is SteppingSplitPolicy. This is overall a
> good policy which is more likely to split small tables to ensure they are
> spread across more servers. If you upgrade, you may notice your tables
> suddenly getting split more than you're used to. This may be an issue if you
> use a row key prefix, because hbase isn't aware of your prefix and may mess
> up your splits. You can get around this by defining a RegionSplitRestriction.
> See HBASE-25766
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #4180: HBASE-26807 Unify CallQueueTooBigException special pause with CallDroppedException
Apache-HBase commented on pull request #4180: URL: https://github.com/apache/hbase/pull/4180#issuecomment-1061850046 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 58s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 18s | master passed | | +1 :green_heart: | compile | 4m 34s | master passed | | +1 :green_heart: | checkstyle | 1m 32s | master passed | | +0 :ok: | refguide | 3m 27s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotbugs | 3m 28s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 17s | the patch passed | | +1 :green_heart: | compile | 4m 25s | the patch passed | | +1 :green_heart: | javac | 4m 25s | the patch passed | | +1 :green_heart: | checkstyle | 1m 29s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +0 :ok: | refguide | 2m 21s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | hadoopcheck | 15m 32s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotbugs | 3m 29s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. | | | | 56m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4180/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4180 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile refguide xml | | uname | Linux 296992526bee 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 39ecaa1975 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | refguide | https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-4180/4/yetus-general-check/output/branch-site/book.html | | refguide | https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-4180/4/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 60 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4180/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bbeaudreault opened a new pull request #4183: HBASE-26784 Addendum: Close scanner request should properly inherit original timeout and priority
bbeaudreault opened a new pull request #4183: URL: https://github.com/apache/hbase/pull/4183 Fixes a bug in the original implementation in https://github.com/apache/hbase/pull/4163, wherein we attempted to inherit the original controller priority/timeout but accidentally was using the same HBaseRpcController for both reading and writing the fields. Improves tests to ensure that we are properly inheriting these fields. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26784) Use HIGH_QOS for ResultScanner.close requests
[ https://issues.apache.org/jira/browse/HBASE-26784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503008#comment-17503008 ] Bryan Beaudreault commented on HBASE-26784: --- Sorry all, in porting this change to our internal fork I noticed a small bug in the branch-2 PR. I wish I had noticed this earlier of course, but I've submitted an addendum PR to fix: https://github.com/apache/hbase/pull/4183 > Use HIGH_QOS for ResultScanner.close requests > - > > Key: HBASE-26784 > URL: https://issues.apache.org/jira/browse/HBASE-26784 > Project: HBase > Issue Type: Improvement >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Major > Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3 > > > Active scanners take resources on the RegionServer, and scanner leaks can > have a big impact on GC. Under high load, ResultScanner.close() calls might > get rejected or timed out before executing. Exceptions thrown by close() are > currently swallowed, so clients will not retry. This leads to a leak of > active scanners on the server side. > These calls should be relatively free to execute, and executing them will > free up the resources which can have an impact on overall RegionServer > performance. As such it can be beneficial to execute all > ResultScanner.close() requests with HIGH_QOS regardless of the priority of > the underlying Scan. > In a scan-heavy stress test I ran, after a couple minutes the Full GC time > (using G1GC) starts increasing and we start seeing 1s pauses, due to > increasing number of active scanners (into the 10s of thousands). Re-running > the stress test with HIGH_QOS for ResultScanner.close() eliminates the leak > and GC time remains stable. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Reopened] (HBASE-26784) Use HIGH_QOS for ResultScanner.close requests
[ https://issues.apache.org/jira/browse/HBASE-26784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault reopened HBASE-26784: --- Reopening for addendum > Use HIGH_QOS for ResultScanner.close requests > - > > Key: HBASE-26784 > URL: https://issues.apache.org/jira/browse/HBASE-26784 > Project: HBase > Issue Type: Improvement >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Major > Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3 > > > Active scanners take resources on the RegionServer, and scanner leaks can > have a big impact on GC. Under high load, ResultScanner.close() calls might > get rejected or timed out before executing. Exceptions thrown by close() are > currently swallowed, so clients will not retry. This leads to a leak of > active scanners on the server side. > These calls should be relatively free to execute, and executing them will > free up the resources which can have an impact on overall RegionServer > performance. As such it can be beneficial to execute all > ResultScanner.close() requests with HIGH_QOS regardless of the priority of > the underlying Scan. > In a scan-heavy stress test I ran, after a couple minutes the Full GC time > (using G1GC) starts increasing and we start seeing 1s pauses, due to > increasing number of active scanners (into the 10s of thousands). Re-running > the stress test with HIGH_QOS for ResultScanner.close() eliminates the leak > and GC time remains stable. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] ndimiduk commented on pull request #4173: Backport "HBASE-26782 Minor code cleanup in and around RpcExecutor" to branch-2
ndimiduk commented on pull request #4173: URL: https://github.com/apache/hbase/pull/4173#issuecomment-1061872425 I wouldn't want to make any interface compatibility changes to IA.LimitedPrivate classes on a patch release, only minor release. Thank again @stoty for taking the time! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #4176: HBASE-26531 Trace coprocessor exec endpoints
ndimiduk commented on pull request #4176: URL: https://github.com/apache/hbase/pull/4176#issuecomment-1061881121 Failure in `TestCoprocessorEndpointTracing.traceSyncTableEndpoint` demonstrates the bit I haven't sorted yet. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4183: HBASE-26784 Addendum: Close scanner request should properly inherit original timeout and priority
Apache-HBase commented on pull request #4183: URL: https://github.com/apache/hbase/pull/4183#issuecomment-1061890361 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 0s | branch-2 passed | | +1 :green_heart: | compile | 0m 22s | branch-2 passed | | +1 :green_heart: | shadedjars | 3m 31s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 47s | the patch passed | | +1 :green_heart: | compile | 0m 21s | the patch passed | | +1 :green_heart: | javac | 0m 21s | the patch passed | | +1 :green_heart: | shadedjars | 3m 29s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 19s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 3m 24s | hbase-client in the patch passed. | | | | 19m 13s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4183/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4183 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1f89d90ca5eb 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 5bae04e20c | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4183/1/testReport/ | | Max. process+thread count | 157 (vs. ulimit of 12500) | | modules | C: hbase-client U: hbase-client | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4183/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4173: Backport "HBASE-26782 Minor code cleanup in and around RpcExecutor" to branch-2
Apache-HBase commented on pull request #4173: URL: https://github.com/apache/hbase/pull/4173#issuecomment-1061916840 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 27s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 39s | branch-2 passed | | +1 :green_heart: | compile | 2m 51s | branch-2 passed | | +1 :green_heart: | checkstyle | 0m 44s | branch-2 passed | | +1 :green_heart: | spotbugs | 1m 35s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 9s | the patch passed | | +1 :green_heart: | compile | 2m 57s | the patch passed | | +1 :green_heart: | javac | 2m 57s | the patch passed | | +1 :green_heart: | checkstyle | 0m 45s | hbase-server: The patch generated 0 new + 1 unchanged - 14 fixed = 1 total (was 15) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 9m 18s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 1m 57s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 35m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4173/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4173 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 76305228b76a 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 5bae04e20c | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 60 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4173/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bbeaudreault commented on pull request #4183: HBASE-26784 Addendum: Close scanner request should properly inherit original timeout and priority
bbeaudreault commented on pull request #4183: URL: https://github.com/apache/hbase/pull/4183#issuecomment-1061921442 Build failed due to "no space left on device". Force pushed to re-run tests, hopefully I'll get a different host. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bbeaudreault edited a comment on pull request #4183: HBASE-26784 Addendum: Close scanner request should properly inherit original timeout and priority
bbeaudreault edited a comment on pull request #4183: URL: https://github.com/apache/hbase/pull/4183#issuecomment-1061921442 Build failed due to "no space left on device". Force pushed to re-run build, hopefully I'll get a different host. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4183: HBASE-26784 Addendum: Close scanner request should properly inherit original timeout and priority
Apache-HBase commented on pull request #4183: URL: https://github.com/apache/hbase/pull/4183#issuecomment-1061954949 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 39s | branch-2 passed | | +1 :green_heart: | compile | 0m 22s | branch-2 passed | | +1 :green_heart: | shadedjars | 3m 29s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 19s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 49s | the patch passed | | +1 :green_heart: | compile | 0m 21s | the patch passed | | +1 :green_heart: | javac | 0m 21s | the patch passed | | +1 :green_heart: | shadedjars | 3m 29s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 19s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 3m 27s | hbase-client in the patch passed. | | | | 18m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4183/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4183 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a230ca7c0e87 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 5bae04e20c | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4183/2/testReport/ | | Max. process+thread count | 157 (vs. ulimit of 12500) | | modules | C: hbase-client U: hbase-client | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4183/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4182: HBASE-26811 Secondary replica may be disabled for read forever
Apache-HBase commented on pull request #4182: URL: https://github.com/apache/hbase/pull/4182#issuecomment-1061959292 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 23s | master passed | | +1 :green_heart: | compile | 0m 47s | master passed | | +1 :green_heart: | shadedjars | 3m 45s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 34s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 18s | the patch passed | | +1 :green_heart: | compile | 0m 47s | the patch passed | | +1 :green_heart: | javac | 0m 47s | the patch passed | | +1 :green_heart: | shadedjars | 3m 43s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 33s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 31s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 318m 37s | hbase-server in the patch passed. | | | | 338m 15s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4182/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4182 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 458a5978ed7a 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 39ecaa1975 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4182/2/testReport/ | | Max. process+thread count | 3356 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4182/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Reopened] (HBASE-25844) Fix Jersey for hbase-server processes
[
https://issues.apache.org/jira/browse/HBASE-25844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk reopened HBASE-25844:
--
Actually, I'm not sure. I see that we still have {{javax.ws.rs-api}} as a
dependency in our hbase-http pom. If, for example, the {{ClientBuilder}} from
this jar is used, it'll instantiate whatever non-shaded jersey client is on the
class path.
> Fix Jersey for hbase-server processes
> -
>
> Key: HBASE-25844
> URL: https://issues.apache.org/jira/browse/HBASE-25844
> Project: HBase
> Issue Type: Task
> Components: master, regionserver, thirdparty
>Affects Versions: 3.0.0-alpha-1, 2.5.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
>
> I spent some time trying to use Jersey from within the Master and it's not
> working. To summarize, we have unshaded resources from both
> jersey-server-1.19 and jersey-server-2.32 on the hbase-server classpath.
> Jersey's initialization uses ServiceLoader to look up concrete implementation
> classes of {{javax.ws.rs}} classes at runtime. Because we do not shade
> {{javax.ws.rs}} in hbase-thirdparty-jersey, an attempt to use shaded
> jersey-2.x still results in loading unshaded jersey-1.x jars, leading to an
> error like this
> {noformat}
> java.lang.AbstractMethodError:
> javax.ws.rs.core.UriBuilder.uri(Ljava/lang/String;)Ljavax/ws/rs/core/UriBuilder;
> at javax.ws.rs.core.UriBuilder.fromUri(UriBuilder.java:96)
> at
> org.apache.hbase.thirdparty.org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:275)
> at
> org.apache.hbase.thirdparty.org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
> at
> org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
> at
> org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
> at
> org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:112)
> {noformat}
> We cannot override what version of these classes are loaded at runtime via
> Java property because Jersey's load order implementation checks system
> properties as a last resort, not first thing as is claimed by javadoc.
> So I can think of two solutions.
> # One is to shade {{javax.ws.rs}} in hbase-thirdparty-jersey. This would
> shade both the interfaces and the resource files that are referenced at
> runtime, allowing for an entirely isolated jersey container instantiate.
> # Another idea is to add a custom {{ClassLoader}} that is inserted before
> jersey is initialized. This would filter out resources that are "banned",
> allowing our desired implementation through.
> Between these, I think (1) is better, but I don't know what else might break.
> I've made an effort of both, but with neither approach can I get a jersey
> environment to response from my resource class... either because the solution
> is incomplete, or because I don't have the jersey environment configured
> properly.
> See also some discussion that happened over on Slack,
> https://apache-hbase.slack.com/archives/C13K8NVAM/p1618857521051700
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] taklwu commented on a change in pull request #4125: HBASE-26666 Add native TLS encryption support to RPC server/client
taklwu commented on a change in pull request #4125:
URL: https://github.com/apache/hbase/pull/4125#discussion_r821955359
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java
##
@@ -181,4 +204,80 @@ public int getNumOpenConnections() {
// allChannels also contains the server channel, so exclude that from the
count.
return channelsCount > 0 ? channelsCount - 1 : channelsCount;
}
+
+ private synchronized void initSSL(ChannelPipeline p, boolean
supportPlaintext) throws
+X509Exception {
+SslContext nettySslContext;
+
+SSLContextAndOptions sslContextAndOptions =
x509Util.getDefaultSSLContextAndOptions();
+nettySslContext = sslContextAndOptions
+ .createNettyJdkSslContext(sslContextAndOptions.getSSLContext(), false);
+
+if (supportPlaintext) {
Review comment:
[nit] does this `supportPlaintext` is always `true` ? should we just
support the plain text mode? or if you would like to have a configuration for
it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-26812) ShortCircuitingClusterConnection fails to close RegionScanners when making short-circuited calls
[
https://issues.apache.org/jira/browse/HBASE-26812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503128#comment-17503128
]
Lars Hofhansl commented on HBASE-26812:
---
See PHOENIX-6458 and PHOENIX-6501. We (will) have a better solution in Phoenix.
> ShortCircuitingClusterConnection fails to close RegionScanners when making
> short-circuited calls
>
>
> Key: HBASE-26812
> URL: https://issues.apache.org/jira/browse/HBASE-26812
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.9
>Reporter: Lars Hofhansl
>Priority: Critical
>
> Just ran into this on the Phoenix side.
> We retrieve a Connection via
> {{{}RegionCoprocessorEnvironment.createConnection... getTable(...){}}}. And
> then call get on that table. The Get's key happens to be local. Now each call
> to table.get() leaves an open StoreScanner around forever. (verified with a
> memory profiler).
> There references are held via
> RegionScannerImpl.storeHeap.scannersForDelayedClose. Eventially the
> RegionServer goes into a GC of death and can only ended with kill -9.
> The reason appears to be that in this case there is no currentCall context.
> Some time in 2.x the Rpc handler/call was made responsible for closing open
> region scanners, but we forgot to handle {{ShortCircuitingClusterConnection}}
> It's not immediately clear how to fix this. But it does make
> ShortCircuitingClusterConnection useless and dangerous. If you use it, you
> *will* create a giant memory leak.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #4180: HBASE-26807 Unify CallQueueTooBigException special pause with CallDroppedException
Apache-HBase commented on pull request #4180: URL: https://github.com/apache/hbase/pull/4180#issuecomment-1062115683 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 42s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 50s | master passed | | +1 :green_heart: | compile | 1m 8s | master passed | | +1 :green_heart: | shadedjars | 3m 41s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 50s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 37s | the patch passed | | +1 :green_heart: | compile | 1m 9s | the patch passed | | +1 :green_heart: | javac | 1m 9s | the patch passed | | +1 :green_heart: | shadedjars | 3m 37s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 47s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 39s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 1m 39s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 306m 53s | hbase-server in the patch passed. | | | | 330m 23s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4180/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4180 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux dd4b537d24f4 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 39ecaa1975 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4180/4/testReport/ | | Max. process+thread count | 2855 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4180/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26760) LICENSE handling should not allow non-aggregated "apache-2.0"
[
https://issues.apache.org/jira/browse/HBASE-26760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503135#comment-17503135
]
Hudson commented on HBASE-26760:
Results for branch master
[build #529 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/529/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/529/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/529/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/529/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> LICENSE handling should not allow non-aggregated "apache-2.0"
> -
>
> Key: HBASE-26760
> URL: https://issues.apache.org/jira/browse/HBASE-26760
> Project: HBase
> Issue Type: Task
> Components: community
>Affects Versions: 2.4.3, 2.4.4, 2.4.5, 2.4.6, 2.4.7, 2.4.8, 2.4.9, 2.4.10
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.11
>
>
> our non-aggregated license check currently allows the malformed name
> "apache-2.0". All ALv2 licenses should be aggregated.
> - remove "apache-2.0" from the {{non_aggregate_fine}} list
> - figure out what dependencies are referring to this license and correct them
> in {{supplemental-models.xml}}
> nice to have: maybe add a precommit check that complains if the
> {{non_aggregate_fine}} set is changed?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26784) Use HIGH_QOS for ResultScanner.close requests
[
https://issues.apache.org/jira/browse/HBASE-26784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503136#comment-17503136
]
Hudson commented on HBASE-26784:
Results for branch master
[build #529 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/529/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/529/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/529/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/529/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Use HIGH_QOS for ResultScanner.close requests
> -
>
> Key: HBASE-26784
> URL: https://issues.apache.org/jira/browse/HBASE-26784
> Project: HBase
> Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3
>
>
> Active scanners take resources on the RegionServer, and scanner leaks can
> have a big impact on GC. Under high load, ResultScanner.close() calls might
> get rejected or timed out before executing. Exceptions thrown by close() are
> currently swallowed, so clients will not retry. This leads to a leak of
> active scanners on the server side.
> These calls should be relatively free to execute, and executing them will
> free up the resources which can have an impact on overall RegionServer
> performance. As such it can be beneficial to execute all
> ResultScanner.close() requests with HIGH_QOS regardless of the priority of
> the underlying Scan.
> In a scan-heavy stress test I ran, after a couple minutes the Full GC time
> (using G1GC) starts increasing and we start seeing 1s pauses, due to
> increasing number of active scanners (into the 10s of thousands). Re-running
> the stress test with HIGH_QOS for ResultScanner.close() eliminates the leak
> and GC time remains stable.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26782) Minor code cleanup in and around RpcExecutor
[
https://issues.apache.org/jira/browse/HBASE-26782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503134#comment-17503134
]
Hudson commented on HBASE-26782:
Results for branch master
[build #529 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/529/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/529/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/529/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/529/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Minor code cleanup in and around RpcExecutor
>
>
> Key: HBASE-26782
> URL: https://issues.apache.org/jira/browse/HBASE-26782
> Project: HBase
> Issue Type: Task
> Components: IPC/RPC
>Affects Versions: 2.6.0, 3.0.0-alpha-3
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 3.0.0-alpha-3
>
>
> While working on tracing, I see some minor cleanup that can be done around
> the RpcExecutor class. The implementations of the {{dispatch}} methods don't
> actually throw the exceptions defined in their signatures, and there's some
> simplification to be done and checkstyle warnings to resolve while I'm in
> there.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26764) Implement generic exception support for TraceUtil methods over Callables and Runnables
[
https://issues.apache.org/jira/browse/HBASE-26764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503142#comment-17503142
]
Hudson commented on HBASE-26764:
Results for branch branch-2.5
[build #57 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/57/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/57/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/57/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/57/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/57/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Implement generic exception support for TraceUtil methods over Callables and
> Runnables
> --
>
> Key: HBASE-26764
> URL: https://issues.apache.org/jira/browse/HBASE-26764
> Project: HBase
> Issue Type: Sub-task
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3
>
>
> {{TraceUtil.trace}} methods can be consolidated with some generics.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26760) LICENSE handling should not allow non-aggregated "apache-2.0"
[
https://issues.apache.org/jira/browse/HBASE-26760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503143#comment-17503143
]
Hudson commented on HBASE-26760:
Results for branch branch-2.5
[build #57 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/57/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/57/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/57/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/57/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/57/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> LICENSE handling should not allow non-aggregated "apache-2.0"
> -
>
> Key: HBASE-26760
> URL: https://issues.apache.org/jira/browse/HBASE-26760
> Project: HBase
> Issue Type: Task
> Components: community
>Affects Versions: 2.4.3, 2.4.4, 2.4.5, 2.4.6, 2.4.7, 2.4.8, 2.4.9, 2.4.10
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.11
>
>
> our non-aggregated license check currently allows the malformed name
> "apache-2.0". All ALv2 licenses should be aggregated.
> - remove "apache-2.0" from the {{non_aggregate_fine}} list
> - figure out what dependencies are referring to this license and correct them
> in {{supplemental-models.xml}}
> nice to have: maybe add a precommit check that complains if the
> {{non_aggregate_fine}} set is changed?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] apurtell merged pull request #4183: HBASE-26784 Addendum: Close scanner request should properly inherit original timeout and priority
apurtell merged pull request #4183: URL: https://github.com/apache/hbase/pull/4183 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26784) Use HIGH_QOS for ResultScanner.close requests
[ https://issues.apache.org/jira/browse/HBASE-26784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503149#comment-17503149 ] Andrew Kyle Purtell commented on HBASE-26784: - Merged the addendum > Use HIGH_QOS for ResultScanner.close requests > - > > Key: HBASE-26784 > URL: https://issues.apache.org/jira/browse/HBASE-26784 > Project: HBase > Issue Type: Improvement >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Major > Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3 > > > Active scanners take resources on the RegionServer, and scanner leaks can > have a big impact on GC. Under high load, ResultScanner.close() calls might > get rejected or timed out before executing. Exceptions thrown by close() are > currently swallowed, so clients will not retry. This leads to a leak of > active scanners on the server side. > These calls should be relatively free to execute, and executing them will > free up the resources which can have an impact on overall RegionServer > performance. As such it can be beneficial to execute all > ResultScanner.close() requests with HIGH_QOS regardless of the priority of > the underlying Scan. > In a scan-heavy stress test I ran, after a couple minutes the Full GC time > (using G1GC) starts increasing and we start seeing 1s pauses, due to > increasing number of active scanners (into the 10s of thousands). Re-running > the stress test with HIGH_QOS for ResultScanner.close() eliminates the leak > and GC time remains stable. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26784) Use HIGH_QOS for ResultScanner.close requests
[ https://issues.apache.org/jira/browse/HBASE-26784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-26784. - Resolution: Fixed > Use HIGH_QOS for ResultScanner.close requests > - > > Key: HBASE-26784 > URL: https://issues.apache.org/jira/browse/HBASE-26784 > Project: HBase > Issue Type: Improvement >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Major > Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3 > > > Active scanners take resources on the RegionServer, and scanner leaks can > have a big impact on GC. Under high load, ResultScanner.close() calls might > get rejected or timed out before executing. Exceptions thrown by close() are > currently swallowed, so clients will not retry. This leads to a leak of > active scanners on the server side. > These calls should be relatively free to execute, and executing them will > free up the resources which can have an impact on overall RegionServer > performance. As such it can be beneficial to execute all > ResultScanner.close() requests with HIGH_QOS regardless of the priority of > the underlying Scan. > In a scan-heavy stress test I ran, after a couple minutes the Full GC time > (using G1GC) starts increasing and we start seeing 1s pauses, due to > increasing number of active scanners (into the 10s of thousands). Re-running > the stress test with HIGH_QOS for ResultScanner.close() eliminates the leak > and GC time remains stable. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] apurtell commented on a change in pull request #4164: HBASE-25465 Use javac --release option for supporting cross version c…
apurtell commented on a change in pull request #4164:
URL: https://github.com/apache/hbase/pull/4164#discussion_r822010905
##
File path:
hbase-common/src/main/java/org/apache/hadoop/hbase/nio/SingleByteBuff.java
##
@@ -65,7 +62,7 @@ public SingleByteBuff(Recycler recycler, ByteBuffer buf) {
this.unsafeOffset = UnsafeAccess.BYTE_ARRAY_BASE_OFFSET +
buf.arrayOffset();
this.unsafeRef = buf.array();
} else {
- this.unsafeOffset = ((DirectBuffer) buf).address();
+ this.unsafeOffset = UnsafeAccess.directBufferAddress(buf);
Review comment:
Should we move all of UnsafeAccess into HBasePlatformDependent?
##
File path:
hbase-common/src/main/java/org/apache/hadoop/hbase/util/ClassSize.java
##
@@ -186,13 +186,19 @@ long sizeOfByteArray(int len) {
private byte a;
}
+private static final int ARRAY_OBJECT_INDEX_SCALE =
+ HBasePlatformDependent.arrayIndexScale(Object[].class);
+
+private static final int ARRAY_BYTE_INDEX_SCALE =
+ HBasePlatformDependent.arrayIndexScale(byte[].class);
+
public UnsafeLayout() {
}
@Override
int headerSize() {
try {
-return (int) UnsafeAccess.theUnsafe.objectFieldOffset(
+return (int) HBasePlatformDependent.objectFieldOffset(
Review comment:
This is platform dependent so should be moved into
HBasePlatformDependent. Perhaps all of ClassSize should go up in there but at
least these methods involving object layout concerns can be replaced with
external static helpers in the thirdparty module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-26522) Improve documentation of hbase 1.x to 2.x potential incompatibilities
[ https://issues.apache.org/jira/browse/HBASE-26522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503158#comment-17503158 ] Andrew Kyle Purtell commented on HBASE-26522: - I realize this is in some ways compounding the original error by changing behavior again, but we could default the feature to off, like setting hbase.region.store.parallel.put.limit to 0 as discussed on HBASE-26575. > Improve documentation of hbase 1.x to 2.x potential incompatibilities > - > > Key: HBASE-26522 > URL: https://issues.apache.org/jira/browse/HBASE-26522 > Project: HBase > Issue Type: Improvement >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Minor > > We're working on a major upgrade of almost 900 tables across 100 production > clusters (and corresponding QA environment clusters). We've upgraded about > 25% of our QA environment and run into a series of incompatibilities along > the way. Most of them have been easy to get around, but I wanted to create > this Jira to collect them so that we can make an update to the docs for > future upgraders. > My plan is to periodically edit this description to add to the list. If > anyone else has anything to contribute, feel free to edit as well or add a > comment. > Incompatibilities to document: > - HBASE-15676 changed the serialized byte string used for the fuzzy mask. > FuzzyRowFilters created by older clients will not match any rows in an hbase2 > cluster. This was fixed in HBASE-26537 but should be documented in our > upgrade guide. > - CDH5 try/catches bad HTableDescriptor.getDurability calls and returns > USE_DEFAULT. In hbase2, if someone creates a table with a bad durability > (i.e. DEFAULT instead of USE_DEFAULT), it results in a failure which causes > the CreateTableProcedure to infinitely retries with no backoff. This rapid > retry caused a bunch of pain on the cluster that encountered it, backing up > datanode's ability to keep up with the millions of calls to create and delete > .regioninfo files. > - This isn't quite an incompatibility, but HBASE-19389 introduced a > concurrency mitigation which may have surprising results coming from older > versions. The defaults are pretty conservative – when writing more than 100 > columns, no more than 10 concurrent writes or 20 pending writes at once. > - Increments sent from branch-1 clients may get erroneously stored with a > timestamp of 0 on hbase2+ clusters: HBASE-26713 > - CheckAndMutate with a "null" compare value used to ignore CompareOp. Fixed > in HBASE-26742, checkAndMutate affects may change between versions. > - client will not know how to handle dangling rep_barrier rows in meta: > HBASE-26797 > - the default hbase split policy is SteppingSplitPolicy. This is overall a > good policy which is more likely to split small tables to ensure they are > spread across more servers. If you upgrade, you may notice your tables > suddenly getting split more than you're used to. This may be an issue if you > use a row key prefix, because hbase isn't aware of your prefix and may mess > up your splits. You can get around this by defining a RegionSplitRestriction. > See HBASE-25766 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-25844) Fix Jersey for hbase-server processes
[
https://issues.apache.org/jira/browse/HBASE-25844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503160#comment-17503160
]
Andrew Kyle Purtell commented on HBASE-25844:
-
bq. we still have javax.ws.rs-api as a dependency in our hbase-http pom
So we should open another subtask for removing this and any other unshaded
dependency from hbase-http? [~ndimiduk]
> Fix Jersey for hbase-server processes
> -
>
> Key: HBASE-25844
> URL: https://issues.apache.org/jira/browse/HBASE-25844
> Project: HBase
> Issue Type: Task
> Components: master, regionserver, thirdparty
>Affects Versions: 3.0.0-alpha-1, 2.5.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
>
> I spent some time trying to use Jersey from within the Master and it's not
> working. To summarize, we have unshaded resources from both
> jersey-server-1.19 and jersey-server-2.32 on the hbase-server classpath.
> Jersey's initialization uses ServiceLoader to look up concrete implementation
> classes of {{javax.ws.rs}} classes at runtime. Because we do not shade
> {{javax.ws.rs}} in hbase-thirdparty-jersey, an attempt to use shaded
> jersey-2.x still results in loading unshaded jersey-1.x jars, leading to an
> error like this
> {noformat}
> java.lang.AbstractMethodError:
> javax.ws.rs.core.UriBuilder.uri(Ljava/lang/String;)Ljavax/ws/rs/core/UriBuilder;
> at javax.ws.rs.core.UriBuilder.fromUri(UriBuilder.java:96)
> at
> org.apache.hbase.thirdparty.org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:275)
> at
> org.apache.hbase.thirdparty.org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
> at
> org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
> at
> org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
> at
> org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:112)
> {noformat}
> We cannot override what version of these classes are loaded at runtime via
> Java property because Jersey's load order implementation checks system
> properties as a last resort, not first thing as is claimed by javadoc.
> So I can think of two solutions.
> # One is to shade {{javax.ws.rs}} in hbase-thirdparty-jersey. This would
> shade both the interfaces and the resource files that are referenced at
> runtime, allowing for an entirely isolated jersey container instantiate.
> # Another idea is to add a custom {{ClassLoader}} that is inserted before
> jersey is initialized. This would filter out resources that are "banned",
> allowing our desired implementation through.
> Between these, I think (1) is better, but I don't know what else might break.
> I've made an effort of both, but with neither approach can I get a jersey
> environment to response from my resource class... either because the solution
> is incomplete, or because I don't have the jersey environment configured
> properly.
> See also some discussion that happened over on Slack,
> https://apache-hbase.slack.com/archives/C13K8NVAM/p1618857521051700
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] apurtell commented on pull request #4118: HBASE-26582 Prune use of Random and SecureRandom objects
apurtell commented on pull request #4118: URL: https://github.com/apache/hbase/pull/4118#issuecomment-1062156088 There are valid javadoc and checkstyle warnings and an error prone finding, addressing them shortly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell closed pull request #4179: HBASE-26732 Update jackson to 2.13.1
apurtell closed pull request #4179: URL: https://github.com/apache/hbase/pull/4179 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell commented on pull request #4179: HBASE-26732 Update jackson to 2.13.1
apurtell commented on pull request #4179: URL: https://github.com/apache/hbase/pull/4179#issuecomment-1062159714 I see this duplicates changes proposed on https://github.com/apache/hbase/pull/4164 . Thats fine. I will close this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26732) Update jackson to 2.13.1
[ https://issues.apache.org/jira/browse/HBASE-26732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503166#comment-17503166 ] Andrew Kyle Purtell commented on HBASE-26732: - PR#4179 duplicates changes also proposed on PR#4164 . > Update jackson to 2.13.1 > > > Key: HBASE-26732 > URL: https://issues.apache.org/jira/browse/HBASE-26732 > Project: HBase > Issue Type: Bug > Components: security, thirdparty >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: thirdparty-4.1.0 > > > Update jackson-databind to 2.13.1 to address a raised vulnerability that > could possible DoS attack certain versions of Jackson. Please refer to > https://github.com/FasterXML/jackson-databind/issues/3328 for further info. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26732) Update jackson to 2.13.1
[ https://issues.apache.org/jira/browse/HBASE-26732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-26732: Fix Version/s: (was: 2.5.0) (was: 2.6.0) (was: 3.0.0-alpha-3) > Update jackson to 2.13.1 > > > Key: HBASE-26732 > URL: https://issues.apache.org/jira/browse/HBASE-26732 > Project: HBase > Issue Type: Bug > Components: security, thirdparty >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: thirdparty-4.1.0 > > > Update jackson-databind to 2.13.1 to address a raised vulnerability that > could possible DoS attack certain versions of Jackson. Please refer to > https://github.com/FasterXML/jackson-databind/issues/3328 for further info. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26732) Update jackson to 2.13.1
[ https://issues.apache.org/jira/browse/HBASE-26732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-26732: Resolution: Fixed Status: Resolved (was: Patch Available) > Update jackson to 2.13.1 > > > Key: HBASE-26732 > URL: https://issues.apache.org/jira/browse/HBASE-26732 > Project: HBase > Issue Type: Bug > Components: security, thirdparty >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: thirdparty-4.1.0 > > > Update jackson-databind to 2.13.1 to address a raised vulnerability that > could possible DoS attack certain versions of Jackson. Please refer to > https://github.com/FasterXML/jackson-databind/issues/3328 for further info. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] apurtell commented on pull request #4118: HBASE-26582 Prune use of Random and SecureRandom objects
apurtell commented on pull request #4118: URL: https://github.com/apache/hbase/pull/4118#issuecomment-1062176992 Updates to fix some issues I introduced in the last round. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25844) Fix Jersey for hbase-server processes
[
https://issues.apache.org/jira/browse/HBASE-25844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503173#comment-17503173
]
Nick Dimiduk commented on HBASE-25844:
--
If we can remove them, yes, I think we should.
> Fix Jersey for hbase-server processes
> -
>
> Key: HBASE-25844
> URL: https://issues.apache.org/jira/browse/HBASE-25844
> Project: HBase
> Issue Type: Task
> Components: master, regionserver, thirdparty
>Affects Versions: 3.0.0-alpha-1, 2.5.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
>
> I spent some time trying to use Jersey from within the Master and it's not
> working. To summarize, we have unshaded resources from both
> jersey-server-1.19 and jersey-server-2.32 on the hbase-server classpath.
> Jersey's initialization uses ServiceLoader to look up concrete implementation
> classes of {{javax.ws.rs}} classes at runtime. Because we do not shade
> {{javax.ws.rs}} in hbase-thirdparty-jersey, an attempt to use shaded
> jersey-2.x still results in loading unshaded jersey-1.x jars, leading to an
> error like this
> {noformat}
> java.lang.AbstractMethodError:
> javax.ws.rs.core.UriBuilder.uri(Ljava/lang/String;)Ljavax/ws/rs/core/UriBuilder;
> at javax.ws.rs.core.UriBuilder.fromUri(UriBuilder.java:96)
> at
> org.apache.hbase.thirdparty.org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:275)
> at
> org.apache.hbase.thirdparty.org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
> at
> org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
> at
> org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
> at
> org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:112)
> {noformat}
> We cannot override what version of these classes are loaded at runtime via
> Java property because Jersey's load order implementation checks system
> properties as a last resort, not first thing as is claimed by javadoc.
> So I can think of two solutions.
> # One is to shade {{javax.ws.rs}} in hbase-thirdparty-jersey. This would
> shade both the interfaces and the resource files that are referenced at
> runtime, allowing for an entirely isolated jersey container instantiate.
> # Another idea is to add a custom {{ClassLoader}} that is inserted before
> jersey is initialized. This would filter out resources that are "banned",
> allowing our desired implementation through.
> Between these, I think (1) is better, but I don't know what else might break.
> I've made an effort of both, but with neither approach can I get a jersey
> environment to response from my resource class... either because the solution
> is incomplete, or because I don't have the jersey environment configured
> properly.
> See also some discussion that happened over on Slack,
> https://apache-hbase.slack.com/archives/C13K8NVAM/p1618857521051700
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HBASE-26715) RegionServer should abort of rollWAL cannot complete in a timely manner
[
https://issues.apache.org/jira/browse/HBASE-26715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell reassigned HBASE-26715:
---
Assignee: Andrew Kyle Purtell
> RegionServer should abort of rollWAL cannot complete in a timely manner
> ---
>
> Key: HBASE-26715
> URL: https://issues.apache.org/jira/browse/HBASE-26715
> Project: HBase
> Issue Type: Bug
>Reporter: Bryan Beaudreault
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Labels: WAL
>
> Ran into an issue on hbase 2.4.6, I think related to HBASE-26679. Individual
> writes are blocking on SyncFuture, which never gets completed. Eventually
> (5m) the writes timeout and fail. But the regionserver hung on like this
> basically forever until I killed it about 14 hours later. While 26679 may fix
> the hang bug, I think we should have additional protection against such
> zombie states. In this case I think what happened is that the rollWAL was
> requested due to failed appends, but it also hung forever. See the below
> stack trace:
>
> {code:java}
> Thread 240 (regionserver/host:60020.logRoller):
> State: WAITING
> Blocked count: 38
> Waited count: 293
> Waiting on java.util.concurrent.CompletableFuture$Signaller@13342c6d
> Stack:
> java.base@11.0.5/jdk.internal.misc.Unsafe.park(Native Method)
>
> java.base@11.0.5/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
>
> java.base@11.0.5/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796)
>
> java.base@11.0.5/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128)
>
> java.base@11.0.5/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823)
>
> java.base@11.0.5/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:189)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeMagicAndWALHeader(AsyncProtobufLogWriter.java:202)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:170)
>
> app//org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:113)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:669)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:130)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:841)
>
> app//org.apache.hadoop.hbase.wal.AbstractWALRoller$RollController.rollWal(AbstractWALRoller.java:268)
>
> app//org.apache.hadoop.hbase.wal.AbstractWALRoller.run(AbstractWALRoller.java:187)
> {code}
>
> The wall roller thread was stuck on this wait seemingly forever, so it was
> never able to roll the wal and get writes working again. I think we should
> add a timeout here, and abort the regionserver if a WAL cannot be rolled in a
> timely manner.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26715) RegionServer should abort of rollWAL cannot complete in a timely manner
[
https://issues.apache.org/jira/browse/HBASE-26715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503175#comment-17503175
]
Andrew Kyle Purtell commented on HBASE-26715:
-
bq. I wonder if we could re-use hbase.regionserver.logroll.wait.timeout.ms that
was added in HBASE-26679. While that Jira used it for a different purpose, I
think the intent still remains – as a maximum time for logroll to take. It just
so happens that they tackled from the perspective of bounding retries
If we place a comment in the code to this effect it is as reasonable as any
alternative I can think of. Let me take this up.
> RegionServer should abort of rollWAL cannot complete in a timely manner
> ---
>
> Key: HBASE-26715
> URL: https://issues.apache.org/jira/browse/HBASE-26715
> Project: HBase
> Issue Type: Bug
>Reporter: Bryan Beaudreault
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Labels: WAL
>
> Ran into an issue on hbase 2.4.6, I think related to HBASE-26679. Individual
> writes are blocking on SyncFuture, which never gets completed. Eventually
> (5m) the writes timeout and fail. But the regionserver hung on like this
> basically forever until I killed it about 14 hours later. While 26679 may fix
> the hang bug, I think we should have additional protection against such
> zombie states. In this case I think what happened is that the rollWAL was
> requested due to failed appends, but it also hung forever. See the below
> stack trace:
>
> {code:java}
> Thread 240 (regionserver/host:60020.logRoller):
> State: WAITING
> Blocked count: 38
> Waited count: 293
> Waiting on java.util.concurrent.CompletableFuture$Signaller@13342c6d
> Stack:
> java.base@11.0.5/jdk.internal.misc.Unsafe.park(Native Method)
>
> java.base@11.0.5/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
>
> java.base@11.0.5/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796)
>
> java.base@11.0.5/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128)
>
> java.base@11.0.5/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823)
>
> java.base@11.0.5/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:189)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeMagicAndWALHeader(AsyncProtobufLogWriter.java:202)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:170)
>
> app//org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:113)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:669)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:130)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:841)
>
> app//org.apache.hadoop.hbase.wal.AbstractWALRoller$RollController.rollWal(AbstractWALRoller.java:268)
>
> app//org.apache.hadoop.hbase.wal.AbstractWALRoller.run(AbstractWALRoller.java:187)
> {code}
>
> The wall roller thread was stuck on this wait seemingly forever, so it was
> never able to roll the wal and get writes working again. I think we should
> add a timeout here, and abort the regionserver if a WAL cannot be rolled in a
> timely manner.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26813) Determine whether unshaded jersey jars are still needed.
[
https://issues.apache.org/jira/browse/HBASE-26813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-26813:
-
Description: I see that we still have {{javax.ws.rs-api}} as a dependency
in our hbase-http pom. If, for example, the {{ClientBuilder}}from this jar is
used, it'll instantiate whatever non-shaded jersey client is on the class path.
> Determine whether unshaded jersey jars are still needed.
>
>
> Key: HBASE-26813
> URL: https://issues.apache.org/jira/browse/HBASE-26813
> Project: HBase
> Issue Type: Sub-task
>Reporter: Nick Dimiduk
>Priority: Major
>
> I see that we still have {{javax.ws.rs-api}} as a dependency in our
> hbase-http pom. If, for example, the {{ClientBuilder}}from this jar is used,
> it'll instantiate whatever non-shaded jersey client is on the class path.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26813) Determine whether unshaded jersey jars are still needed.
Nick Dimiduk created HBASE-26813: Summary: Determine whether unshaded jersey jars are still needed. Key: HBASE-26813 URL: https://issues.apache.org/jira/browse/HBASE-26813 Project: HBase Issue Type: Sub-task Reporter: Nick Dimiduk -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26813) Determine whether unshaded jersey jars are still needed.
[
https://issues.apache.org/jira/browse/HBASE-26813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-26813:
-
Description: I see that we still have {{javax.ws.rs-api}} as a dependency
in our hbase-http pom. If, for example, the {{ClientBuilder}} from this jar is
used, it'll instantiate whatever non-shaded jersey client is on the class path.
(was: I see that we still have {{javax.ws.rs-api}} as a dependency in our
hbase-http pom. If, for example, the {{ClientBuilder}}from this jar is used,
it'll instantiate whatever non-shaded jersey client is on the class path.)
> Determine whether unshaded jersey jars are still needed.
>
>
> Key: HBASE-26813
> URL: https://issues.apache.org/jira/browse/HBASE-26813
> Project: HBase
> Issue Type: Sub-task
>Reporter: Nick Dimiduk
>Priority: Major
>
> I see that we still have {{javax.ws.rs-api}} as a dependency in our
> hbase-http pom. If, for example, the {{ClientBuilder}} from this jar is used,
> it'll instantiate whatever non-shaded jersey client is on the class path.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] ndimiduk commented on a change in pull request #4177: HBASE-25895 Implement a Cluster Metrics JSON endpoint
ndimiduk commented on a change in pull request #4177:
URL: https://github.com/apache/hbase/pull/4177#discussion_r822049148
##
File path:
hbase-http/src/main/java/org/apache/hadoop/hbase/http/gson/ByteArraySerializer.java
##
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.http.gson;
+
+import java.lang.reflect.Type;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.apache.hbase.thirdparty.com.google.gson.JsonElement;
+import org.apache.hbase.thirdparty.com.google.gson.JsonPrimitive;
+import org.apache.hbase.thirdparty.com.google.gson.JsonSerializationContext;
+import org.apache.hbase.thirdparty.com.google.gson.JsonSerializer;
+
+/**
+ * Serialize a {@code byte[]} using {@link Bytes#toString()}.
+ */
+@InterfaceAudience.Private
+public final class ByteArraySerializer implements JsonSerializer {
+
+ @Override
+ public JsonElement serialize(byte[] src, Type typeOfSrc,
JsonSerializationContext context) {
+return new JsonPrimitive(Bytes.toString(src));
Review comment:
@apurtell I cannot tell you. Actually, I'm embarrassed to see that this
commit doesn't include any tests. I've added some most basic coverage,
including a test that shows the behavior of the configured Gson instance
produced by the factory. Please advise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-26715) Blocked on SyncFuture in AsyncProtobufLogWriter#write
[
https://issues.apache.org/jira/browse/HBASE-26715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-26715:
Summary: Blocked on SyncFuture in AsyncProtobufLogWriter#write (was:
RegionServer should abort of rollWAL cannot complete in a timely manner)
> Blocked on SyncFuture in AsyncProtobufLogWriter#write
> -
>
> Key: HBASE-26715
> URL: https://issues.apache.org/jira/browse/HBASE-26715
> Project: HBase
> Issue Type: Bug
>Reporter: Bryan Beaudreault
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Labels: WAL
>
> Ran into an issue on hbase 2.4.6, I think related to HBASE-26679. Individual
> writes are blocking on SyncFuture, which never gets completed. Eventually
> (5m) the writes timeout and fail. But the regionserver hung on like this
> basically forever until I killed it about 14 hours later. While 26679 may fix
> the hang bug, I think we should have additional protection against such
> zombie states. In this case I think what happened is that the rollWAL was
> requested due to failed appends, but it also hung forever. See the below
> stack trace:
>
> {code:java}
> Thread 240 (regionserver/host:60020.logRoller):
> State: WAITING
> Blocked count: 38
> Waited count: 293
> Waiting on java.util.concurrent.CompletableFuture$Signaller@13342c6d
> Stack:
> java.base@11.0.5/jdk.internal.misc.Unsafe.park(Native Method)
>
> java.base@11.0.5/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
>
> java.base@11.0.5/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796)
>
> java.base@11.0.5/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128)
>
> java.base@11.0.5/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823)
>
> java.base@11.0.5/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:189)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeMagicAndWALHeader(AsyncProtobufLogWriter.java:202)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:170)
>
> app//org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:113)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:669)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:130)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:841)
>
> app//org.apache.hadoop.hbase.wal.AbstractWALRoller$RollController.rollWal(AbstractWALRoller.java:268)
>
> app//org.apache.hadoop.hbase.wal.AbstractWALRoller.run(AbstractWALRoller.java:187)
> {code}
>
> The wall roller thread was stuck on this wait seemingly forever, so it was
> never able to roll the wal and get writes working again. I think we should
> add a timeout here, and abort the regionserver if a WAL cannot be rolled in a
> timely manner.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] apurtell opened a new pull request #4184: HBASE-26715 Blocked on SyncFuture in AsyncProtobufLogWriter#write
apurtell opened a new pull request #4184: URL: https://github.com/apache/hbase/pull/4184 See the discussion on HBASE-26715 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26715) Blocked on SyncFuture in AsyncProtobufLogWriter#write
[
https://issues.apache.org/jira/browse/HBASE-26715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-26715:
Fix Version/s: 2.5.0
2.6.0
3.0.0-alpha-3
2.4.11
Status: Patch Available (was: Open)
> Blocked on SyncFuture in AsyncProtobufLogWriter#write
> -
>
> Key: HBASE-26715
> URL: https://issues.apache.org/jira/browse/HBASE-26715
> Project: HBase
> Issue Type: Bug
>Reporter: Bryan Beaudreault
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Labels: WAL
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.11
>
>
> Ran into an issue on hbase 2.4.6, I think related to HBASE-26679. Individual
> writes are blocking on SyncFuture, which never gets completed. Eventually
> (5m) the writes timeout and fail. But the regionserver hung on like this
> basically forever until I killed it about 14 hours later. While 26679 may fix
> the hang bug, I think we should have additional protection against such
> zombie states. In this case I think what happened is that the rollWAL was
> requested due to failed appends, but it also hung forever. See the below
> stack trace:
>
> {code:java}
> Thread 240 (regionserver/host:60020.logRoller):
> State: WAITING
> Blocked count: 38
> Waited count: 293
> Waiting on java.util.concurrent.CompletableFuture$Signaller@13342c6d
> Stack:
> java.base@11.0.5/jdk.internal.misc.Unsafe.park(Native Method)
>
> java.base@11.0.5/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
>
> java.base@11.0.5/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796)
>
> java.base@11.0.5/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128)
>
> java.base@11.0.5/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823)
>
> java.base@11.0.5/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:189)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeMagicAndWALHeader(AsyncProtobufLogWriter.java:202)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:170)
>
> app//org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:113)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:669)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:130)
>
> app//org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:841)
>
> app//org.apache.hadoop.hbase.wal.AbstractWALRoller$RollController.rollWal(AbstractWALRoller.java:268)
>
> app//org.apache.hadoop.hbase.wal.AbstractWALRoller.run(AbstractWALRoller.java:187)
> {code}
>
> The wall roller thread was stuck on this wait seemingly forever, so it was
> never able to roll the wal and get writes working again. I think we should
> add a timeout here, and abort the regionserver if a WAL cannot be rolled in a
> timely manner.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26814) Default StoreHotnessProtector to off, with logs to guide when to turn it on
Bryan Beaudreault created HBASE-26814: - Summary: Default StoreHotnessProtector to off, with logs to guide when to turn it on Key: HBASE-26814 URL: https://issues.apache.org/jira/browse/HBASE-26814 Project: HBase Issue Type: Improvement Reporter: Bryan Beaudreault As discussed in HBASE-26522 and somewhat in HBASE-26575, StoreHotnessProtector is too conservative. One can easily trigger RegionTooBusyException on an otherwise idle regionserver with relatively small amount of traffic. The idea behind this protection is good, but it needs to take into account current load in order to warrant being enabled by default. Until it's improved as such, we should change it to default off. One thing we can do to help drive usage would be to add warning logs when it would have triggered. Operators who are diagnosing performance issues will naturally look at the logs and see indication of a potential tool for mitigation. cc [~andrew.purt...@gmail.com] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26522) Improve documentation of hbase 1.x to 2.x potential incompatibilities
[ https://issues.apache.org/jira/browse/HBASE-26522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503176#comment-17503176 ] Bryan Beaudreault commented on HBASE-26522: --- That might be the best approach given current constraints. I filed https://issues.apache.org/jira/browse/HBASE-26814. When do you think you might release 2.5.0? I can try to push a quick PR before you do, unless someone else wants to take it up instead. > Improve documentation of hbase 1.x to 2.x potential incompatibilities > - > > Key: HBASE-26522 > URL: https://issues.apache.org/jira/browse/HBASE-26522 > Project: HBase > Issue Type: Improvement >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Minor > > We're working on a major upgrade of almost 900 tables across 100 production > clusters (and corresponding QA environment clusters). We've upgraded about > 25% of our QA environment and run into a series of incompatibilities along > the way. Most of them have been easy to get around, but I wanted to create > this Jira to collect them so that we can make an update to the docs for > future upgraders. > My plan is to periodically edit this description to add to the list. If > anyone else has anything to contribute, feel free to edit as well or add a > comment. > Incompatibilities to document: > - HBASE-15676 changed the serialized byte string used for the fuzzy mask. > FuzzyRowFilters created by older clients will not match any rows in an hbase2 > cluster. This was fixed in HBASE-26537 but should be documented in our > upgrade guide. > - CDH5 try/catches bad HTableDescriptor.getDurability calls and returns > USE_DEFAULT. In hbase2, if someone creates a table with a bad durability > (i.e. DEFAULT instead of USE_DEFAULT), it results in a failure which causes > the CreateTableProcedure to infinitely retries with no backoff. This rapid > retry caused a bunch of pain on the cluster that encountered it, backing up > datanode's ability to keep up with the millions of calls to create and delete > .regioninfo files. > - This isn't quite an incompatibility, but HBASE-19389 introduced a > concurrency mitigation which may have surprising results coming from older > versions. The defaults are pretty conservative – when writing more than 100 > columns, no more than 10 concurrent writes or 20 pending writes at once. > - Increments sent from branch-1 clients may get erroneously stored with a > timestamp of 0 on hbase2+ clusters: HBASE-26713 > - CheckAndMutate with a "null" compare value used to ignore CompareOp. Fixed > in HBASE-26742, checkAndMutate affects may change between versions. > - client will not know how to handle dangling rep_barrier rows in meta: > HBASE-26797 > - the default hbase split policy is SteppingSplitPolicy. This is overall a > good policy which is more likely to split small tables to ensure they are > spread across more servers. If you upgrade, you may notice your tables > suddenly getting split more than you're used to. This may be an issue if you > use a row key prefix, because hbase isn't aware of your prefix and may mess > up your splits. You can get around this by defining a RegionSplitRestriction. > See HBASE-25766 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #4181: HBASE-26810 Add dynamic configuration support for system coprocessors
Apache-HBase commented on pull request #4181: URL: https://github.com/apache/hbase/pull/4181#issuecomment-1062220164 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 23s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 34s | master passed | | +1 :green_heart: | compile | 1m 41s | master passed | | +1 :green_heart: | shadedjars | 3m 33s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 47s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 20s | the patch passed | | +1 :green_heart: | compile | 1m 41s | the patch passed | | +1 :green_heart: | javac | 1m 41s | the patch passed | | +1 :green_heart: | shadedjars | 3m 34s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 48s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 528m 15s | root in the patch passed. | | | | 549m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4181/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4181 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 73f6acf81e8f 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 39ecaa1975 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4181/2/testReport/ | | Max. process+thread count | 3172 (vs. ulimit of 3) | | modules | C: hbase-server . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4181/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4173: Backport "HBASE-26782 Minor code cleanup in and around RpcExecutor" to branch-2
Apache-HBase commented on pull request #4173: URL: https://github.com/apache/hbase/pull/4173#issuecomment-1062234277 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 18s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 11s | branch-2 passed | | +1 :green_heart: | compile | 0m 53s | branch-2 passed | | +1 :green_heart: | shadedjars | 4m 19s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 37s | the patch passed | | +1 :green_heart: | compile | 0m 55s | the patch passed | | +1 :green_heart: | javac | 0m 55s | the patch passed | | +1 :green_heart: | shadedjars | 4m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 363m 27s | hbase-server in the patch passed. | | | | 386m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4173/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4173 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1846cd652ed9 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 5bae04e20c | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4173/2/testReport/ | | Max. process+thread count | 1818 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4173/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4118: HBASE-26582 Prune use of Random and SecureRandom objects
Apache-HBase commented on pull request #4118: URL: https://github.com/apache/hbase/pull/4118#issuecomment-1062235785 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 58s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 57s | master passed | | +1 :green_heart: | compile | 7m 55s | master passed | | +1 :green_heart: | checkstyle | 3m 20s | master passed | | +1 :green_heart: | spotbugs | 6m 35s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 22s | the patch passed | | +1 :green_heart: | compile | 7m 34s | the patch passed | | -0 :warning: | javac | 0m 19s | hbase-asyncfs generated 1 new + 25 unchanged - 1 fixed = 26 total (was 26) | | -0 :warning: | javac | 2m 11s | hbase-server generated 1 new + 192 unchanged - 1 fixed = 193 total (was 193) | | -0 :warning: | javac | 0m 26s | hbase-it generated 2 new + 101 unchanged - 2 fixed = 103 total (was 103) | | -0 :warning: | checkstyle | 0m 16s | hbase-common: The patch generated 1 new + 132 unchanged - 3 fixed = 133 total (was 135) | | +1 :green_heart: | checkstyle | 0m 8s | The patch passed checkstyle in hbase-metrics | | +1 :green_heart: | checkstyle | 0m 10s | The patch passed checkstyle in hbase-hadoop-compat | | +1 :green_heart: | checkstyle | 0m 17s | hbase-client: The patch generated 0 new + 14 unchanged - 4 fixed = 14 total (was 18) | | +1 :green_heart: | checkstyle | 0m 9s | The patch passed checkstyle in hbase-zookeeper | | +1 :green_heart: | checkstyle | 0m 8s | The patch passed checkstyle in hbase-replication | | +1 :green_heart: | checkstyle | 0m 10s | The patch passed checkstyle in hbase-balancer | | +1 :green_heart: | checkstyle | 0m 9s | The patch passed checkstyle in hbase-http | | +1 :green_heart: | checkstyle | 0m 9s | The patch passed checkstyle in hbase-asyncfs | | +1 :green_heart: | checkstyle | 0m 9s | The patch passed checkstyle in hbase-procedure | | +1 :green_heart: | checkstyle | 0m 42s | hbase-server: The patch generated 0 new + 405 unchanged - 18 fixed = 405 total (was 423) | | +1 :green_heart: | checkstyle | 0m 13s | hbase-mapreduce: The patch generated 0 new + 85 unchanged - 1 fixed = 85 total (was 86) | | +1 :green_heart: | checkstyle | 0m 12s | The patch passed checkstyle in hbase-testing-util | | -0 :warning: | checkstyle | 0m 14s | hbase-it: The patch generated 2 new + 186 unchanged - 2 fixed = 188 total (was 188) | | +1 :green_heart: | checkstyle | 0m 11s | The patch passed checkstyle in hbase-rest | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 20s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotbugs | 8m 17s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 2m 13s | The patch does not generate ASF License warnings. | | | | 65m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4118/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4118 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 1ac75153ebcf 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 39ecaa1975 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4118/4/artifact/yetus-general-check/output/diff-compile-javac-hbase-asyncfs.txt | | javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4118/4/artifact/yetus-general-check/output/diff-compile-javac-hbase-server.txt | | javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4118/4/artifact/yetus-general-check/output/diff-compile-javac-hbase-it.txt | | checkstyle | https://ci-hbase.apache.
[jira] [Assigned] (HBASE-26804) Missing opentelemetry agent in hadoop-two-compat.xml
[ https://issues.apache.org/jira/browse/HBASE-26804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu reassigned HBASE-26804: Assignee: Tak-Lon (Stephen) Wu > Missing opentelemetry agent in hadoop-two-compat.xml > > > Key: HBASE-26804 > URL: https://issues.apache.org/jira/browse/HBASE-26804 > Project: HBase > Issue Type: Bug > Components: tracing >Affects Versions: 2.6.0 >Reporter: Duo Zhang >Assignee: Tak-Lon (Stephen) Wu >Priority: Blocker > > We do not have hadoop-two-compat.xml on master, this is probably why we > missed this file when backporting to branch-2. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] apurtell merged pull request #4118: HBASE-26582 Prune use of Random and SecureRandom objects
apurtell merged pull request #4118: URL: https://github.com/apache/hbase/pull/4118 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] taklwu commented on pull request #4185: HBASE-26804 Missing opentelemetry agent in hadoop-two-compat.xml
taklwu commented on pull request #4185: URL: https://github.com/apache/hbase/pull/4185#issuecomment-1062247493 see the original commit in master here https://github.com/apache/hbase/commit/8d2a0efb7a9c17e58892369b281b4af49a5f0d7e and this PR is porting the change in [hadoop-three-compat.xml](https://github.com/apache/hbase/commit/8d2a0efb7a9c17e58892369b281b4af49a5f0d7e#diff-3245893b838247ac80130558cebcbbb4d3fb8346c4290a127314be8d965b18ac) to hadoop-two-compat.xml -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Work started] (HBASE-26804) Missing opentelemetry agent in hadoop-two-compat.xml
[ https://issues.apache.org/jira/browse/HBASE-26804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-26804 started by Tak-Lon (Stephen) Wu. > Missing opentelemetry agent in hadoop-two-compat.xml > > > Key: HBASE-26804 > URL: https://issues.apache.org/jira/browse/HBASE-26804 > Project: HBase > Issue Type: Bug > Components: tracing >Affects Versions: 2.6.0 >Reporter: Duo Zhang >Assignee: Tak-Lon (Stephen) Wu >Priority: Blocker > > We do not have hadoop-two-compat.xml on master, this is probably why we > missed this file when backporting to branch-2. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26555) Fix findbugs/spotbugs findings
[ https://issues.apache.org/jira/browse/HBASE-26555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-26555: Fix Version/s: 2.5.0 2.6.0 3.0.0-alpha-3 > Fix findbugs/spotbugs findings > -- > > Key: HBASE-26555 > URL: https://issues.apache.org/jira/browse/HBASE-26555 > Project: HBase > Issue Type: Task >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Minor > Fix For: 2.5.0, 2.3.8, 2.6.0, 3.0.0-alpha-3 > > > Clean up findbugs/spotbugs findings on branch HEADs (master, branch-2, > branch-2.5, branch-2.4, branch-2.3) > For example, from recent precommits: > Possible null pointer dereference in > org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.opMeterName(Object) > due to return value of called method Dereferenced at > MetaTableMetrics.java:org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.opMeterName(Object) > due to return value of called method Dereferenced at > MetaTableMetrics.java:[line 230] > Possible null pointer dereference in > org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.opWithClientMeterName(Object) > due to return value of called method Dereferenced at > MetaTableMetrics.java:org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.opWithClientMeterName(Object) > due to return value of called method Dereferenced at > MetaTableMetrics.java:[line 210] > Possible null pointer dereference in > org.apache.hadoop.hbase.io.hfile.LruCachedBlockQueue.add(LruCachedBlock) due > to return value of called method Method invoked at > LruCachedBlockQueue.java:org.apache.hadoop.hbase.io.hfile.LruCachedBlockQueue.add(LruCachedBlock) > due to return value of called method Method invoked at > LruCachedBlockQueue.java:[line 73] > Possible null pointer dereference in > org.apache.hadoop.hbase.io.hfile.bucket.CachedEntryQueue.add(Map$Entry) due > to return value of called method Dereferenced at > CachedEntryQueue.java:org.apache.hadoop.hbase.io.hfile.bucket.CachedEntryQueue.add(Map$Entry) > due to return value of called method Dereferenced at > CachedEntryQueue.java:[line 79] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26555) Fix findbugs/spotbugs findings
[ https://issues.apache.org/jira/browse/HBASE-26555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-26555: Fix Version/s: (was: 2.5.0) (was: 2.6.0) (was: 3.0.0-alpha-3) (was: 2.4.11) > Fix findbugs/spotbugs findings > -- > > Key: HBASE-26555 > URL: https://issues.apache.org/jira/browse/HBASE-26555 > Project: HBase > Issue Type: Task >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Minor > Fix For: 2.3.8 > > > Clean up findbugs/spotbugs findings on branch HEADs (master, branch-2, > branch-2.5, branch-2.4, branch-2.3) > For example, from recent precommits: > Possible null pointer dereference in > org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.opMeterName(Object) > due to return value of called method Dereferenced at > MetaTableMetrics.java:org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.opMeterName(Object) > due to return value of called method Dereferenced at > MetaTableMetrics.java:[line 230] > Possible null pointer dereference in > org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.opWithClientMeterName(Object) > due to return value of called method Dereferenced at > MetaTableMetrics.java:org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.opWithClientMeterName(Object) > due to return value of called method Dereferenced at > MetaTableMetrics.java:[line 210] > Possible null pointer dereference in > org.apache.hadoop.hbase.io.hfile.LruCachedBlockQueue.add(LruCachedBlock) due > to return value of called method Method invoked at > LruCachedBlockQueue.java:org.apache.hadoop.hbase.io.hfile.LruCachedBlockQueue.add(LruCachedBlock) > due to return value of called method Method invoked at > LruCachedBlockQueue.java:[line 73] > Possible null pointer dereference in > org.apache.hadoop.hbase.io.hfile.bucket.CachedEntryQueue.add(Map$Entry) due > to return value of called method Dereferenced at > CachedEntryQueue.java:org.apache.hadoop.hbase.io.hfile.bucket.CachedEntryQueue.add(Map$Entry) > due to return value of called method Dereferenced at > CachedEntryQueue.java:[line 79] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #4147: HBASE-26783 ScannerCallable doubly clears meta cache on retries
Apache-HBase commented on pull request #4147: URL: https://github.com/apache/hbase/pull/4147#issuecomment-1062253225 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 49s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 58s | branch-2 passed | | +1 :green_heart: | compile | 0m 55s | branch-2 passed | | +1 :green_heart: | shadedjars | 3m 40s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 43s | the patch passed | | +1 :green_heart: | compile | 0m 54s | the patch passed | | +1 :green_heart: | javac | 0m 54s | the patch passed | | +1 :green_heart: | shadedjars | 3m 37s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 3m 30s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 315m 53s | hbase-server in the patch passed. | | | | 339m 11s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4147/11/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4147 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux fc497bf5b46b 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 5bae04e20c | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4147/11/testReport/ | | Max. process+thread count | 2148 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4147/11/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26814) Default StoreHotnessProtector to off, with logs to guide when to turn it on
[ https://issues.apache.org/jira/browse/HBASE-26814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-26814: Fix Version/s: 2.5.0 2.6.0 3.0.0-alpha-3 > Default StoreHotnessProtector to off, with logs to guide when to turn it on > --- > > Key: HBASE-26814 > URL: https://issues.apache.org/jira/browse/HBASE-26814 > Project: HBase > Issue Type: Improvement >Reporter: Bryan Beaudreault >Priority: Major > Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3 > > > As discussed in HBASE-26522 and somewhat in HBASE-26575, > StoreHotnessProtector is too conservative. One can easily trigger > RegionTooBusyException on an otherwise idle regionserver with relatively > small amount of traffic. The idea behind this protection is good, but it > needs to take into account current load in order to warrant being enabled by > default. > Until it's improved as such, we should change it to default off. One thing we > can do to help drive usage would be to add warning logs when it would have > triggered. Operators who are diagnosing performance issues will naturally > look at the logs and see indication of a potential tool for mitigation. > cc [~andrew.purt...@gmail.com] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26814) Default StoreHotnessProtector to off, with logs to guide when to turn it on
[ https://issues.apache.org/jira/browse/HBASE-26814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503196#comment-17503196 ] Andrew Kyle Purtell commented on HBASE-26814: - +1 > Default StoreHotnessProtector to off, with logs to guide when to turn it on > --- > > Key: HBASE-26814 > URL: https://issues.apache.org/jira/browse/HBASE-26814 > Project: HBase > Issue Type: Improvement >Reporter: Bryan Beaudreault >Priority: Major > Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3 > > > As discussed in HBASE-26522 and somewhat in HBASE-26575, > StoreHotnessProtector is too conservative. One can easily trigger > RegionTooBusyException on an otherwise idle regionserver with relatively > small amount of traffic. The idea behind this protection is good, but it > needs to take into account current load in order to warrant being enabled by > default. > Until it's improved as such, we should change it to default off. One thing we > can do to help drive usage would be to add warning logs when it would have > triggered. Operators who are diagnosing performance issues will naturally > look at the logs and see indication of a potential tool for mitigation. > cc [~andrew.purt...@gmail.com] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26522) Improve documentation of hbase 1.x to 2.x potential incompatibilities
[ https://issues.apache.org/jira/browse/HBASE-26522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503194#comment-17503194 ] Andrew Kyle Purtell commented on HBASE-26522: - There is time. Duo is backporting is log4j2 work. Some things Nick has been working on are being flushed out now. > Improve documentation of hbase 1.x to 2.x potential incompatibilities > - > > Key: HBASE-26522 > URL: https://issues.apache.org/jira/browse/HBASE-26522 > Project: HBase > Issue Type: Improvement >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Minor > > We're working on a major upgrade of almost 900 tables across 100 production > clusters (and corresponding QA environment clusters). We've upgraded about > 25% of our QA environment and run into a series of incompatibilities along > the way. Most of them have been easy to get around, but I wanted to create > this Jira to collect them so that we can make an update to the docs for > future upgraders. > My plan is to periodically edit this description to add to the list. If > anyone else has anything to contribute, feel free to edit as well or add a > comment. > Incompatibilities to document: > - HBASE-15676 changed the serialized byte string used for the fuzzy mask. > FuzzyRowFilters created by older clients will not match any rows in an hbase2 > cluster. This was fixed in HBASE-26537 but should be documented in our > upgrade guide. > - CDH5 try/catches bad HTableDescriptor.getDurability calls and returns > USE_DEFAULT. In hbase2, if someone creates a table with a bad durability > (i.e. DEFAULT instead of USE_DEFAULT), it results in a failure which causes > the CreateTableProcedure to infinitely retries with no backoff. This rapid > retry caused a bunch of pain on the cluster that encountered it, backing up > datanode's ability to keep up with the millions of calls to create and delete > .regioninfo files. > - This isn't quite an incompatibility, but HBASE-19389 introduced a > concurrency mitigation which may have surprising results coming from older > versions. The defaults are pretty conservative – when writing more than 100 > columns, no more than 10 concurrent writes or 20 pending writes at once. > - Increments sent from branch-1 clients may get erroneously stored with a > timestamp of 0 on hbase2+ clusters: HBASE-26713 > - CheckAndMutate with a "null" compare value used to ignore CompareOp. Fixed > in HBASE-26742, checkAndMutate affects may change between versions. > - client will not know how to handle dangling rep_barrier rows in meta: > HBASE-26797 > - the default hbase split policy is SteppingSplitPolicy. This is overall a > good policy which is more likely to split small tables to ensure they are > spread across more servers. If you upgrade, you may notice your tables > suddenly getting split more than you're used to. This may be an issue if you > use a row key prefix, because hbase isn't aware of your prefix and may mess > up your splits. You can get around this by defining a RegionSplitRestriction. > See HBASE-25766 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] apurtell commented on a change in pull request #4166: HBASE-26802 Backport the log4j2 changes to branch-2
apurtell commented on a change in pull request #4166:
URL: https://github.com/apache/hbase/pull/4166#discussion_r822116792
##
File path: bin/hbase-daemon.sh
##
@@ -155,12 +155,20 @@ JAVA=$JAVA_HOME/bin/java
export HBASE_LOG_PREFIX=hbase-$HBASE_IDENT_STRING-$command-$HOSTNAME
export HBASE_LOGFILE=$HBASE_LOG_PREFIX.log
-if [ -z "${HBASE_ROOT_LOGGER}" ]; then
-export HBASE_ROOT_LOGGER=${HBASE_ROOT_LOGGER:-"INFO,RFA"}
+if [ -z "${HBASE_ROOT_LOGGER_LEVEL}" ]; then
+export HBASE_ROOT_LOGGER_LEVEL=${HBASE_ROOT_LOGGER_LEVEL:-"INFO"}
fi
-if [ -z "${HBASE_SECURITY_LOGGER}" ]; then
Review comment:
If provided this string we could munge this string into LOGGER_APPENDER
and LOGGER_LEVEL and warn about the change in environment variable usage for
additional compatibility.
##
File path: bin/hbase-daemon.sh
##
@@ -155,12 +155,20 @@ JAVA=$JAVA_HOME/bin/java
export HBASE_LOG_PREFIX=hbase-$HBASE_IDENT_STRING-$command-$HOSTNAME
export HBASE_LOGFILE=$HBASE_LOG_PREFIX.log
-if [ -z "${HBASE_ROOT_LOGGER}" ]; then
Review comment:
If provided this string we could munge this string into LOGGER_APPENDER
and LOGGER_LEVEL and warn about the change in environment variable usage for
additional compatibility.
##
File path: bin/hbase
##
@@ -796,10 +799,6 @@ HBASE_OPTS="$HBASE_OPTS -Dhbase.log.file=$HBASE_LOGFILE"
HBASE_OPTS="$HBASE_OPTS -Dhbase.home.dir=$HBASE_HOME"
HBASE_OPTS="$HBASE_OPTS -Dhbase.id.str=$HBASE_IDENT_STRING"
HBASE_OPTS="$HBASE_OPTS -Dhbase.root.logger=${HBASE_ROOT_LOGGER:-INFO,console}"
-if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then
Review comment:
Does dropping this hunk affect native Hadoop library support?
I didn't check if this is just a removal of something redundant.
##
File path:
hbase-asyncfs/src/test/java/org/apache/hadoop/hbase/io/asyncfs/AsyncFSTestBase.java
##
@@ -96,11 +96,6 @@ protected static void startMiniDFSCluster(int servers)
throws IOException {
createDirsAndSetProperties();
Configuration conf = UTIL.getConfiguration();
-// Error level to skip some warnings specific to the minicluster. See
HBASE-4709
Review comment:
This is good cleanup.
Could also use the Log4JUtils static helpers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] apurtell commented on pull request #4166: HBASE-26802 Backport the log4j2 changes to branch-2
apurtell commented on pull request #4166: URL: https://github.com/apache/hbase/pull/4166#issuecomment-1062278983 The changes lgtm, good to commit once the test issues are analyzed and resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26802) Backport the log4j2 changes to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-26802: Fix Version/s: 2.5.0 2.6.0 > Backport the log4j2 changes to branch-2 > --- > > Key: HBASE-26802 > URL: https://issues.apache.org/jira/browse/HBASE-26802 > Project: HBase > Issue Type: Task > Components: logging >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 2.6.0 > > > One thing is that, we'd better make sure that users can still use the old > log4j.properties for logging. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26802) Backport the log4j2 changes to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503202#comment-17503202 ] Andrew Kyle Purtell commented on HBASE-26802: - Let's set this to block the 2.5 release to reflect reality. (RM's prerogative.) > Backport the log4j2 changes to branch-2 > --- > > Key: HBASE-26802 > URL: https://issues.apache.org/jira/browse/HBASE-26802 > Project: HBase > Issue Type: Task > Components: logging >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 2.5.0, 2.6.0 > > > One thing is that, we'd better make sure that users can still use the old > log4j.properties for logging. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26802) Backport the log4j2 changes to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-26802: Priority: Blocker (was: Major) > Backport the log4j2 changes to branch-2 > --- > > Key: HBASE-26802 > URL: https://issues.apache.org/jira/browse/HBASE-26802 > Project: HBase > Issue Type: Task > Components: logging >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 2.5.0, 2.6.0 > > > One thing is that, we'd better make sure that users can still use the old > log4j.properties for logging. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] apurtell commented on pull request #4184: HBASE-26715 Blocked on SyncFuture in AsyncProtobufLogWriter#write
apurtell commented on pull request #4184: URL: https://github.com/apache/hbase/pull/4184#issuecomment-1062285419 > I wonder if we should rename write to writeWALMetadata or something like that. It'd just be clearer to the reader I think, but not a huge deal. @bbeaudreault sounds good, see [c20aa7a](https://github.com/apache/hbase/pull/4184/commits/c20aa7a8efa03da157d889db60b78c9784787ca2) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bbeaudreault commented on pull request #4184: HBASE-26715 Blocked on SyncFuture in AsyncProtobufLogWriter#write
bbeaudreault commented on pull request #4184: URL: https://github.com/apache/hbase/pull/4184#issuecomment-1062295305 Thanks! Looks good. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4185: HBASE-26804 Missing opentelemetry agent in hadoop-two-compat.xml
Apache-HBase commented on pull request #4185: URL: https://github.com/apache/hbase/pull/4185#issuecomment-1062358292 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4185: HBASE-26804 Missing opentelemetry agent in hadoop-two-compat.xml
Apache-HBase commented on pull request #4185: URL: https://github.com/apache/hbase/pull/4185#issuecomment-1062360372 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 22s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 23s | branch-2 passed | | +1 :green_heart: | javadoc | 0m 13s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 13s | the patch passed | | +1 :green_heart: | javadoc | 0m 12s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 21s | hbase-assembly in the patch passed. | | | | 9m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4185/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4185 | | Optional Tests | javac javadoc unit | | uname | Linux b5a4072a3d72 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 98836fb2b4 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4185/1/testReport/ | | Max. process+thread count | 59 (vs. ulimit of 12500) | | modules | C: hbase-assembly U: hbase-assembly | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4185/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4184: HBASE-26715 Blocked on SyncFuture in AsyncProtobufLogWriter#write
Apache-HBase commented on pull request #4184: URL: https://github.com/apache/hbase/pull/4184#issuecomment-1062397998 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 24s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 5s | master passed | | +1 :green_heart: | compile | 0m 46s | master passed | | +1 :green_heart: | shadedjars | 3m 42s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 50s | the patch passed | | +1 :green_heart: | compile | 0m 45s | the patch passed | | +1 :green_heart: | javac | 0m 45s | the patch passed | | +1 :green_heart: | shadedjars | 3m 36s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 15m 25s | hbase-server in the patch failed. | | | | 32m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4184/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4184 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5c8e14ddf14b 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 10471944bd | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4184/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4184/1/testReport/ | | Max. process+thread count | 466 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4184/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4177: HBASE-25895 Implement a Cluster Metrics JSON endpoint
Apache-HBase commented on pull request #4177: URL: https://github.com/apache/hbase/pull/4177#issuecomment-1062398601 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 45s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 38s | master passed | | +1 :green_heart: | compile | 2m 30s | master passed | | +1 :green_heart: | checkstyle | 0m 41s | master passed | | +1 :green_heart: | spotbugs | 1m 33s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 18s | the patch passed | | +1 :green_heart: | compile | 2m 29s | the patch passed | | -0 :warning: | javac | 0m 17s | hbase-http generated 1 new + 25 unchanged - 1 fixed = 26 total (was 26) | | +1 :green_heart: | checkstyle | 0m 43s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 32s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotbugs | 1m 43s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 33m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4177/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4177 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 416391202bd8 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 10471944bd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4177/2/artifact/yetus-general-check/output/diff-compile-javac-hbase-http.txt | | Max. process+thread count | 60 (vs. ulimit of 3) | | modules | C: hbase-http hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4177/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26582) Prune use of Random and SecureRandom objects
[ https://issues.apache.org/jira/browse/HBASE-26582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-26582: Fix Version/s: 2.4.11 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) > Prune use of Random and SecureRandom objects > > > Key: HBASE-26582 > URL: https://issues.apache.org/jira/browse/HBASE-26582 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Minor > Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.11 > > > While looking over Findbugs/Spotbugs findings on HBASE-26555 I noticed we > have some violations of Spotbugs rules regarding lack of Random object > re-use, and generally have a lot of static Random and SecureRandom objects in > our code, especially in tests, where ThreadLocalRandom would be more than > sufficient. > This is nitty stuff but let's clean it up. It will reduce noise on the > analysis reports and SecureRandom is unnecessarily slow when used wherever we > don't need it. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] Apache9 commented on a change in pull request #4166: HBASE-26802 Backport the log4j2 changes to branch-2
Apache9 commented on a change in pull request #4166:
URL: https://github.com/apache/hbase/pull/4166#discussion_r84189
##
File path: bin/hbase
##
@@ -796,10 +799,6 @@ HBASE_OPTS="$HBASE_OPTS -Dhbase.log.file=$HBASE_LOGFILE"
HBASE_OPTS="$HBASE_OPTS -Dhbase.home.dir=$HBASE_HOME"
HBASE_OPTS="$HBASE_OPTS -Dhbase.id.str=$HBASE_IDENT_STRING"
HBASE_OPTS="$HBASE_OPTS -Dhbase.root.logger=${HBASE_ROOT_LOGGER:-INFO,console}"
-if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then
Review comment:
I can not recall why I dropped this when migrating to log4j2 on master
branch...
Will dig more and report back later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
