[GitHub] [hbase] Apache-HBase commented on pull request #1978: HBASE-24638 Edit doc on (offheap) memory management
Apache-HBase commented on pull request #1978: URL: https://github.com/apache/hbase/pull/1978#issuecomment-650021717 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 44s | master passed | | +1 :green_heart: | checkstyle | 1m 55s | master passed | | +0 :ok: | refguide | 4m 44s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotbugs | 9m 14s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 27s | the patch passed | | +1 :green_heart: | checkstyle | 1m 56s | the patch passed | | -0 :warning: | whitespace | 0m 0s | The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +0 :ok: | refguide | 4m 57s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | hadoopcheck | 11m 0s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 9m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. | | | | 59m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1978/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1978 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle refguide | | uname | Linux 7fc6f7810115 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c0461207ee | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1978/1/artifact/yetus-general-check/output/branch-site/book.html | | whitespace | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1978/1/artifact/yetus-general-check/output/whitespace-eol.txt | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1978/1/artifact/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 137 (vs. ulimit of 12500) | | modules | C: hbase-common . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1978/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Reidddddd commented on pull request #1923: HBASE-24578 [WAL] Add a parameter to config RingBufferEventHandler's SyncFuture count
Reidd commented on pull request #1923: URL: https://github.com/apache/hbase/pull/1923#issuecomment-650037495 any updates? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Reidddddd commented on a change in pull request #1923: HBASE-24578 [WAL] Add a parameter to config RingBufferEventHandler's SyncFuture count
Reidd commented on a change in pull request #1923: URL: https://github.com/apache/hbase/pull/1923#discussion_r446025854 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java ## @@ -105,6 +105,16 @@ // We use ring buffer sequence as txid of FSWALEntry and SyncFuture. private static final Logger LOG = LoggerFactory.getLogger(FSHLog.class); + private static final String TOLERABLE_LOW_REPLICATION = "hbase.regionserver.hlog.tolerable.lowreplication"; + private static final String LOW_REPLICATION_ROLL_LIMIT = "hbase.regionserver.hlog.lowreplication.rolllimit"; + private static final int DEFAULT_LOW_REPLICATION_ROLL_LIMIT = 5; + private static final String ROLL_ERRORS_TOLERATED = "hbase.regionserver.logroll.errors.tolerated"; + private static final int DEFAULT_ROLL_ERRORS_TOLERATED = 2; + private static final String SYNCER_COUNT = "hbase.regionserver.hlog.syncer.count"; + private static final int DEFAULT_SYNCER_COUNT = 5; + private static final String MAX_BATCH_COUNT = "hbase.regionserver.hlog.max.batch.count"; Review comment: After second thought, the new name should use `wal` instead of `hlog` which was the name in old version. And from the context in codes, `hbase.regionserver.wal.sync.batch.count` sounds good to me This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1970: HBASE-24625 AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length.
Apache-HBase commented on pull request #1970: URL: https://github.com/apache/hbase/pull/1970#issuecomment-650038368 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 8s | master passed | | +1 :green_heart: | compile | 1m 22s | master passed | | +1 :green_heart: | shadedjars | 6m 20s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 51s | master passed | | -0 :warning: | patch | 7m 31s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 58s | the patch passed | | +1 :green_heart: | compile | 1m 17s | the patch passed | | +1 :green_heart: | javac | 1m 17s | the patch passed | | +1 :green_heart: | shadedjars | 6m 9s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 36s | hbase-server generated 9 new + 28 unchanged - 0 fixed = 37 total (was 28) | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 41s | hbase-asyncfs in the patch passed. | | +1 :green_heart: | unit | 203m 15s | hbase-server in the patch passed. | | | | 234m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1970 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8389f6cd5833 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 84e246f9b1 | | Default Java | 1.8.0_232 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/3/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/3/testReport/ | | Max. process+thread count | 3555 (vs. ulimit of 12500) | | modules | C: hbase-asyncfs hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1970: HBASE-24625 AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length.
Apache-HBase commented on pull request #1970: URL: https://github.com/apache/hbase/pull/1970#issuecomment-650063953 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 4s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 13s | master passed | | +1 :green_heart: | checkstyle | 1m 31s | master passed | | +1 :green_heart: | spotbugs | 2m 51s | master passed | | -0 :warning: | patch | 2m 28s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 57s | the patch passed | | -0 :warning: | checkstyle | 0m 13s | hbase-asyncfs: The patch generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) | | -0 :warning: | checkstyle | 1m 15s | hbase-server: The patch generated 8 new + 38 unchanged - 0 fixed = 46 total (was 38) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 55s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 3m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. | | | | 42m 46s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1970 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 458640ec86b3 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c0461207ee | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-asyncfs.txt | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-asyncfs hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24623) SIGSEGV v ~StubRoutines::jbyte_disjoint_arraycopy
[
https://issues.apache.org/jira/browse/HBASE-24623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146151#comment-17146151
]
Anoop Sam John commented on HBASE-24623:
That will come into pic when we read from HFiles. Previously we will read into
on heap byte[] when we have to read from HDFS or from BC over SSD. From 2.3.0,
for this also, we will use the BBs form the pool.
Coming to your case, so when you see the issue in write, this comes while
replication? I can see the Mutation#add cells are backed by off heap. Those are
ByteBuffer backed KVs. When src cluster calls RPC to destination cluster, at
the destination, it will be the Netty RPC server in pic. So the req bytes will
be read into DBBs from its pool right?
> SIGSEGV v ~StubRoutines::jbyte_disjoint_arraycopy
> --
>
> Key: HBASE-24623
> URL: https://issues.apache.org/jira/browse/HBASE-24623
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Michael Stack
>Priority: Major
>
> In testing, 1% of a decent cluster went down with this seg fault in the vm:
> {code}
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x7f6659052410, pid=37208, tid=0x7f3c89453700
> #
> # JRE version: OpenJDK Runtime Environment (8.0_232-b09) (build 1.8.0_232-b09)
> # Java VM: OpenJDK 64-Bit Server VM (25.232-b09 mixed mode linux-amd64 )
> # Problematic frame:
> # v ~StubRoutines::jbyte_disjoint_arraycopy
> {code}
> Looking in the hs_err log, the crash happens in the same area. Here are a few
> of the stack traces:
> {code}
> Stack: [0x7f3c89353000,0x7f3c89454000], sp=0x7f3c89452110, free
> space=1020k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> v ~StubRoutines::jbyte_disjoint_arraycopy
> J 17674 C2
> org.apache.hadoop.hbase.util.ByteBufferUtils.copyFromBufferToArray([BLjava/nio/ByteBuffer;III)V
> (69 bytes) @ 0x7f665af000d1 [0x7f665aefffe0+0xf1]
> J 17732 C1
> org.apache.hadoop.hbase.CellUtil.copyQualifierTo(Lorg/apache/hadoop/hbase/Cell;[BI)I
> (59 bytes) @ 0x7f665bc440dc [0x7f665bc43b80+0x55c]
> j
> org.apache.hadoop.hbase.CellUtil.cloneQualifier(Lorg/apache/hadoop/hbase/Cell;)[B+12
> J 22278 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getQualifierArray()[B
> (5 bytes) @ 0x7f6659bd4784 [0x7f6659bd4760+0x24]
> j
> org.apache.hadoop.hbase.CellUtil.getCellKeyAsString(Lorg/apache/hadoop/hbase/Cell;Ljava/util/function/Function;)Ljava/lang/String;+97
> j
> org.apache.hadoop.hbase.CellUtil.getCellKeyAsString(Lorg/apache/hadoop/hbase/Cell;)Ljava/lang/String;+6
> j
> org.apache.hadoop.hbase.CellUtil.toString(Lorg/apache/hadoop/hbase/Cell;Z)Ljava/lang/String;+16
> j org.apache.hadoop.hbase.ByteBufferKeyValue.toString()Ljava/lang/String;+2
> j
> org.apache.hadoop.hbase.client.Mutation.add(Lorg/apache/hadoop/hbase/Cell;)Lorg/apache/hadoop/hbase/client/Mutation;+28
> J 22605 C2
> org.apache.hadoop.hbase.client.Put.add(Lorg/apache/hadoop/hbase/Cell;)Lorg/apache/hadoop/hbase/client/Put;
> (8 bytes) @ 0x7f665a982a04 [0x7f665a9829e0+0x24]
> J 22112 C2
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toPut(Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$MutationProto;Lorg/apache/hadoop/hbase/CellScanner;)Lorg/apache/hadoop/hbase/client/Put;
> (910 bytes) @ 0x7f665c706700 [0x7f665c706000+0x700]
> J 24084 C2
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$RegionActionResult$Builder;Lorg/apache/hadoop/hbase/regionserver/HRegion;Lorg/apache/hadoop/hbase/quotas/OperationQuota;Ljava/util/List;Lorg/apache/hadoop/hbase/CellScanner;Lorg/apache/hadoop/hbase/quotas/ActivePolicyEnforcement;Z)V
> (646 bytes) @ 0x7f665cc21100 [0x7f665cc20c80+0x480]
> J 14696 C2
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(Lorg/apache/hadoop/hbase/regionserver/HRegion;Lorg/apache/hadoop/hbase/quotas/OperationQuota;Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$RegionAction;Lorg/apache/hadoop/hbase/CellScanner;Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$RegionActionResult$Builder;Ljava/util/List;JLorg/apache/hadoop/hbase/regionserver/RSRpcServices$RegionScannersCloseCallBack;Lorg/apache/hadoop/hbase/ipc/RpcCallContext;Lorg/apache/hadoop/hbase/quotas/ActivePolicyEnforcement;)Ljava/util/List;
> (901 bytes) @ 0x7f665b722148 [0x7f665b7218e0+0x868]
> {code}
> Here's another:
> {code}
> Stack: [0x7edd015e2000,0x7edd016e3000], sp=0x7edd016e11b0, free
> space=1020k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> v ~StubRoutines::jbyte_disjoint_arraycopy
> J 18255 C2
>
[jira] [Commented] (HBASE-24616) Remove BoundedRecoveredHFilesOutputSink dependency on a TableDescriptor
[
https://issues.apache.org/jira/browse/HBASE-24616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146162#comment-17146162
]
Hudson commented on HBASE-24616:
Results for branch branch-2.3
[build #157 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/157/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/157/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/157/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/157/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/157/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove BoundedRecoveredHFilesOutputSink dependency on a TableDescriptor
>
>
> Key: HBASE-24616
> URL: https://issues.apache.org/jira/browse/HBASE-24616
> Project: HBase
> Issue Type: Bug
> Components: HFile, MTTR
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
>
> BoundedRecoveredHFilesOutputSink wants to read TableDescriptor so it writes
> the particular hfile format specified by a table's schema. Getting the table
> schema can be tough at various points of operation especially around startup.
> HBASE-23739 tried to read from the fs if unable to read TableDescriptor from
> Master. This approach works generally but fails in standalone mode as in
> standalone mode we will have given-up our start up attempt BEFORE the request
> to Master for TableDescriptor times out (the read from fs is never attempted).
> The suggested patch here does away w/ reading TableDescriptor and just has
> BoundedRecoveredHFilesOutputSink write generic hfiles.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24603) Zookeeper sync() call is async
[
https://issues.apache.org/jira/browse/HBASE-24603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146163#comment-17146163
]
Hudson commented on HBASE-24603:
Results for branch branch-2.3
[build #157 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/157/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/157/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/157/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/157/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/157/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Zookeeper sync() call is async
> --
>
> Key: HBASE-24603
> URL: https://issues.apache.org/jira/browse/HBASE-24603
> Project: HBase
> Issue Type: Improvement
> Components: master, regionserver
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 1.7.0
>Reporter: Bharath Vissapragada
>Assignee: Bharath Vissapragada
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0
>
>
> Here is the method that does a sync() of lagging followers with leader in the
> quorum. We rely on this to see a consistent snapshot of ZK data from multiple
> clients. However the problem is that the underlying sync() call is actually
> asynchronous since we are passing a 'null' call back. See the ZK API
> [doc|https://zookeeper.apache.org/doc/r3.5.7/apidocs/zookeeper-server/index.html]
> for details. The end-result is that sync() doesn't guarantee that it has
> happened by the time it returns.
> {noformat}
> /**
>* Forces a synchronization of this ZooKeeper client connection.
>*
>* Executing this method before running other methods will ensure that the
>* subsequent operations are up-to-date and consistent as of the time that
>* the sync is complete.
>*
>* This is used for compareAndSwap type operations where we need to read the
>* data of an existing node and delete or transition that node, utilizing
> the
>* previously read version and data. We want to ensure that the version
> read
>* is up-to-date from when we begin the operation.
>*/
> public void sync(String path) throws KeeperException {
> this.recoverableZooKeeper.sync(path, null, null);
> }
> {noformat}
> We rely on this heavily (at least in the older branches that do ZK based
> region assignment). In branch-1 we saw weird "BadVersionException" exceptions
> in RITs because of the inconsistent view of the ZK snapshot. It could
> manifest differently in other branches. Either way, this is something we need
> to fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22504) Optimize the MultiByteBuff#get(ByteBuffer, offset, len)
[
https://issues.apache.org/jira/browse/HBASE-22504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146161#comment-17146161
]
Hudson commented on HBASE-22504:
Results for branch branch-2.3
[build #157 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/157/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/157/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/157/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/157/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/157/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Optimize the MultiByteBuff#get(ByteBuffer, offset, len)
> ---
>
> Key: HBASE-22504
> URL: https://issues.apache.org/jira/browse/HBASE-22504
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
> Attachments: HBASE-22504.HBASE-21879.v01.patch
>
>
> In HBASE-22483, we saw that the BucketCacheWriter thread was quite busy
> [^BucketCacheWriter-is-busy.png], the flame graph also indicated that the
> ByteBufferArray#internalTransfer cost ~6% CPU (see
> [async-prof-pid-25042-cpu-1.svg|https://issues.apache.org/jira/secure/attachment/12970294/async-prof-pid-25042-cpu-1.svg]).
> because we used the hbase.ipc.server.allocator.buffer.size=64KB, each
> HFileBlock will be backend by a MultiByteBuff: one 64KB offheap ByteBuffer
> and one small heap ByteBuffer.
> The path is depending on the MultiByteBuff#get(ByteBuffer, offset, len) now:
> {code:java}
> RAMQueueEntry#writeToCache
> |--> ByteBufferIOEngine#write
> |--> ByteBufferArray#internalTransfer
> |--> ByteBufferArray$WRITER
> |--> MultiByteBuff#get(ByteBuffer, offset, len)
> {code}
> While the MultiByteBuff#get impl is simple and crude now, can optimze this
> implementation:
> {code:java}
> @Override
> public void get(ByteBuffer out, int sourceOffset,
> int length) {
> checkRefCount();
> // Not used from real read path actually. So not going with
> // optimization
> for (int i = 0; i < length; ++i) {
> out.put(this.get(sourceOffset + i));
> }
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase-native-client] phrocker commented on a change in pull request #6: HBASE-23105: Download lib double conversion, fizz, update folly
phrocker commented on a change in pull request #6: URL: https://github.com/apache/hbase-native-client/pull/6#discussion_r446098589 ## File path: cmake/patches/zookeeper.3.4.14.buf ## @@ -0,0 +1,4 @@ +3480c3480 +< static char buf[128]; Review comment: Thanks! I had updated zk but decided not to. You are correct thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-native-client] phrocker commented on a change in pull request #6: HBASE-23105: Download lib double conversion, fizz, update folly
phrocker commented on a change in pull request #6: URL: https://github.com/apache/hbase-native-client/pull/6#discussion_r446100080 ## File path: cmake/DownloadDoubleConversion.cmake ## @@ -0,0 +1,34 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +## Download facebook's folly library. Review comment: yes I didn't go through this PR to look for things like this. sorry for the careless mistakes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24603) Zookeeper sync() call is async
[
https://issues.apache.org/jira/browse/HBASE-24603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146200#comment-17146200
]
Hudson commented on HBASE-24603:
Results for branch branch-2
[build #2720 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2720/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2720/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2720/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2720/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2720/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Zookeeper sync() call is async
> --
>
> Key: HBASE-24603
> URL: https://issues.apache.org/jira/browse/HBASE-24603
> Project: HBase
> Issue Type: Improvement
> Components: master, regionserver
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 1.7.0
>Reporter: Bharath Vissapragada
>Assignee: Bharath Vissapragada
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0
>
>
> Here is the method that does a sync() of lagging followers with leader in the
> quorum. We rely on this to see a consistent snapshot of ZK data from multiple
> clients. However the problem is that the underlying sync() call is actually
> asynchronous since we are passing a 'null' call back. See the ZK API
> [doc|https://zookeeper.apache.org/doc/r3.5.7/apidocs/zookeeper-server/index.html]
> for details. The end-result is that sync() doesn't guarantee that it has
> happened by the time it returns.
> {noformat}
> /**
>* Forces a synchronization of this ZooKeeper client connection.
>*
>* Executing this method before running other methods will ensure that the
>* subsequent operations are up-to-date and consistent as of the time that
>* the sync is complete.
>*
>* This is used for compareAndSwap type operations where we need to read the
>* data of an existing node and delete or transition that node, utilizing
> the
>* previously read version and data. We want to ensure that the version
> read
>* is up-to-date from when we begin the operation.
>*/
> public void sync(String path) throws KeeperException {
> this.recoverableZooKeeper.sync(path, null, null);
> }
> {noformat}
> We rely on this heavily (at least in the older branches that do ZK based
> region assignment). In branch-1 we saw weird "BadVersionException" exceptions
> in RITs because of the inconsistent view of the ZK snapshot. It could
> manifest differently in other branches. Either way, this is something we need
> to fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24603) Zookeeper sync() call is async
[
https://issues.apache.org/jira/browse/HBASE-24603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146199#comment-17146199
]
Hudson commented on HBASE-24603:
Results for branch branch-1
[build #1322 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1322/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1322//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1322//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1322//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Zookeeper sync() call is async
> --
>
> Key: HBASE-24603
> URL: https://issues.apache.org/jira/browse/HBASE-24603
> Project: HBase
> Issue Type: Improvement
> Components: master, regionserver
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 1.7.0
>Reporter: Bharath Vissapragada
>Assignee: Bharath Vissapragada
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0
>
>
> Here is the method that does a sync() of lagging followers with leader in the
> quorum. We rely on this to see a consistent snapshot of ZK data from multiple
> clients. However the problem is that the underlying sync() call is actually
> asynchronous since we are passing a 'null' call back. See the ZK API
> [doc|https://zookeeper.apache.org/doc/r3.5.7/apidocs/zookeeper-server/index.html]
> for details. The end-result is that sync() doesn't guarantee that it has
> happened by the time it returns.
> {noformat}
> /**
>* Forces a synchronization of this ZooKeeper client connection.
>*
>* Executing this method before running other methods will ensure that the
>* subsequent operations are up-to-date and consistent as of the time that
>* the sync is complete.
>*
>* This is used for compareAndSwap type operations where we need to read the
>* data of an existing node and delete or transition that node, utilizing
> the
>* previously read version and data. We want to ensure that the version
> read
>* is up-to-date from when we begin the operation.
>*/
> public void sync(String path) throws KeeperException {
> this.recoverableZooKeeper.sync(path, null, null);
> }
> {noformat}
> We rely on this heavily (at least in the older branches that do ZK based
> region assignment). In branch-1 we saw weird "BadVersionException" exceptions
> in RITs because of the inconsistent view of the ZK snapshot. It could
> manifest differently in other branches. Either way, this is something we need
> to fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase-native-client] phrocker commented on a change in pull request #6: HBASE-23105: Download lib double conversion, fizz, update folly
phrocker commented on a change in pull request #6:
URL: https://github.com/apache/hbase-native-client/pull/6#discussion_r446124852
##
File path: CMakeLists.txt
##
@@ -154,6 +160,9 @@ endif (OPENSSL_FOUND)
if (DOWNLOAD_DEPENDENCIES)
+ download_doubleconversion(${CMAKE_CURRENT_SOURCE_DIR}
${CMAKE_CURRENT_BINARY_DIR})
+ download_fizz(${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_CURRENT_BINARY_DIR})
Review comment:
Oh sorry I have these changes staged. Will roll them in and sanity check
them. Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase-native-client] phrocker commented on a change in pull request #6: HBASE-23105: Download lib double conversion, fizz, update folly
phrocker commented on a change in pull request #6: URL: https://github.com/apache/hbase-native-client/pull/6#discussion_r446125608 ## File path: cmake/DownloadDoubleConversion.cmake ## @@ -0,0 +1,34 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +## Download facebook's folly library. +## SOURCE_DIR is typically the cmake source directory +function(download_doubleconversion SOURCE_DIR BUILD_DIR) Review comment: no! don't apologize. Please let me know. I tried remote dev through ssh (with my IDE ) but it resets my spaces/tabs configuration so it's me that should apologize! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-native-client] phrocker commented on a change in pull request #6: HBASE-23105: Download lib double conversion, fizz, update folly
phrocker commented on a change in pull request #6:
URL: https://github.com/apache/hbase-native-client/pull/6#discussion_r446127013
##
File path: cmake/DownloadDoubleConversion.cmake
##
@@ -0,0 +1,34 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+## Download facebook's folly library.
+## SOURCE_DIR is typically the cmake source directory
+function(download_doubleconversion SOURCE_DIR BUILD_DIR)
+ ExternalProject_Add(
+ doubleconversion-proj
+ GIT_REPOSITORY "https://github.com/google/double-conversion.git";
+ GIT_TAG "master"
+ SOURCE_DIR "${BUILD_DIR}/dependencies/doubleconversion-proj-src"
+ CMAKE_ARGS ${PASSTHROUGH_CMAKE_ARGS}
Review comment:
I'll remove it. I cherry picked from that first big PR and was going to
add pass through args from the root CMAKE like i've done in other projects.
I've removed that feature since I want to isolate the changes a little more
and frankly wanted to take a step back and fix up some docs before I started
worrying about those kinds of improvements. So I will remove. Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1970: HBASE-24625 AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length.
Apache-HBase commented on pull request #1970: URL: https://github.com/apache/hbase/pull/1970#issuecomment-650137191 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 43s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 5s | master passed | | +1 :green_heart: | compile | 1m 39s | master passed | | +1 :green_heart: | shadedjars | 7m 6s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 18s | hbase-asyncfs in master failed. | | -0 :warning: | javadoc | 0m 44s | hbase-server in master failed. | | -0 :warning: | patch | 8m 27s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 57s | the patch passed | | +1 :green_heart: | compile | 1m 57s | the patch passed | | +1 :green_heart: | javac | 1m 57s | the patch passed | | +1 :green_heart: | shadedjars | 7m 0s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 15s | hbase-asyncfs in the patch failed. | | -0 :warning: | javadoc | 0m 48s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 44s | hbase-asyncfs in the patch passed. | | +1 :green_heart: | unit | 190m 8s | hbase-server in the patch passed. | | | | 226m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1970 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 039abb9fd522 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c0461207ee | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-asyncfs.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-asyncfs.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/4/testReport/ | | Max. process+thread count | 3498 (vs. ulimit of 12500) | | modules | C: hbase-asyncfs hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on a change in pull request #1977: HBASE-24221 addendum to restore public interface on LoadIncrementalHFiles
virajjasani commented on a change in pull request #1977:
URL: https://github.com/apache/hbase/pull/1977#discussion_r446138110
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/tool/LoadIncrementalHFiles.java
##
@@ -803,14 +803,35 @@ private boolean checkHFilesCountPerRegionPerFamily(
*
* Protected for testing.
* @return empty list if success, list of items to retry on recoverable
failure
+ * @deprecated as of release 2.3.0. Use {@link BulkLoadHFiles} instead.
Review comment:
Will be removed as part of 3.0.0/4.0.0 ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1970: HBASE-24625 AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length.
Apache-HBase commented on pull request #1970: URL: https://github.com/apache/hbase/pull/1970#issuecomment-650140393 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 9s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 14s | master passed | | +1 :green_heart: | compile | 1m 22s | master passed | | +1 :green_heart: | shadedjars | 6m 23s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 54s | master passed | | -0 :warning: | patch | 7m 35s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 0s | the patch passed | | +1 :green_heart: | compile | 1m 20s | the patch passed | | +1 :green_heart: | javac | 1m 20s | the patch passed | | +1 :green_heart: | shadedjars | 6m 7s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 36s | hbase-server generated 9 new + 28 unchanged - 0 fixed = 37 total (was 28) | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 49s | hbase-asyncfs in the patch passed. | | +1 :green_heart: | unit | 203m 1s | hbase-server in the patch passed. | | | | 234m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1970 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9f9c61cfcf35 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c0461207ee | | Default Java | 1.8.0_232 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/4/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/4/testReport/ | | Max. process+thread count | 3206 (vs. ulimit of 12500) | | modules | C: hbase-asyncfs hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-operator-tools] wchevreuil merged pull request #70: HBASE-24626 [HBCK2] Remove reference to hase I.A. private class Commo…
wchevreuil merged pull request #70: URL: https://github.com/apache/hbase-operator-tools/pull/70 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24626) [HBCK2] Remove reference to hase I.A. private class CommonFsUtils from FsRegionsMetaRecoverer
[ https://issues.apache.org/jira/browse/HBASE-24626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-24626: - Affects Version/s: hbase-operator-tools-1.0.0 > [HBCK2] Remove reference to hase I.A. private class CommonFsUtils from > FsRegionsMetaRecoverer > - > > Key: HBASE-24626 > URL: https://issues.apache.org/jira/browse/HBASE-24626 > Project: HBase > Issue Type: Improvement > Components: hbase-operator-tools, hbck2 >Affects Versions: hbase-operator-tools-1.0.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > > FsRegionsMetaRecoverer used to reference I.A. private targeted interface > FSUtils, which changed on hbase 2.3, causing hbck2 fail to compile. > HBASE-24482 fixed it by pointing to CommonFSUtils interface, where the > methods it was relying upon was actually defined. Since this is also a IA > private interface, there's no compatibility guarantees. This PR removes > reference to CommonFSUtils on FsRegionsMetaRecoverer. Other classes in hbck2 > still require similar work. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24626) [HBCK2] Remove reference to hase I.A. private class CommonFsUtils from FsRegionsMetaRecoverer
[ https://issues.apache.org/jira/browse/HBASE-24626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-24626: - Fix Version/s: hbase-operator-tools-1.1.0 > [HBCK2] Remove reference to hase I.A. private class CommonFsUtils from > FsRegionsMetaRecoverer > - > > Key: HBASE-24626 > URL: https://issues.apache.org/jira/browse/HBASE-24626 > Project: HBase > Issue Type: Improvement > Components: hbase-operator-tools, hbck2 >Affects Versions: hbase-operator-tools-1.0.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > Fix For: hbase-operator-tools-1.1.0 > > > FsRegionsMetaRecoverer used to reference I.A. private targeted interface > FSUtils, which changed on hbase 2.3, causing hbck2 fail to compile. > HBASE-24482 fixed it by pointing to CommonFSUtils interface, where the > methods it was relying upon was actually defined. Since this is also a IA > private interface, there's no compatibility guarantees. This PR removes > reference to CommonFSUtils on FsRegionsMetaRecoverer. Other classes in hbck2 > still require similar work. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24626) [HBCK2] Remove reference to hase I.A. private class CommonFsUtils from FsRegionsMetaRecoverer
[ https://issues.apache.org/jira/browse/HBASE-24626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil resolved HBASE-24626. -- Resolution: Fixed Merged to master. > [HBCK2] Remove reference to hase I.A. private class CommonFsUtils from > FsRegionsMetaRecoverer > - > > Key: HBASE-24626 > URL: https://issues.apache.org/jira/browse/HBASE-24626 > Project: HBase > Issue Type: Improvement > Components: hbase-operator-tools, hbck2 >Affects Versions: hbase-operator-tools-1.0.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > Fix For: hbase-operator-tools-1.1.0 > > > FsRegionsMetaRecoverer used to reference I.A. private targeted interface > FSUtils, which changed on hbase 2.3, causing hbck2 fail to compile. > HBASE-24482 fixed it by pointing to CommonFSUtils interface, where the > methods it was relying upon was actually defined. Since this is also a IA > private interface, there's no compatibility guarantees. This PR removes > reference to CommonFSUtils on FsRegionsMetaRecoverer. Other classes in hbck2 > still require similar work. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1978: HBASE-24638 Edit doc on (offheap) memory management
Apache-HBase commented on pull request #1978: URL: https://github.com/apache/hbase/pull/1978#issuecomment-650150345 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 54s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 55s | master passed | | +1 :green_heart: | compile | 2m 51s | master passed | | +1 :green_heart: | shadedjars | 6m 46s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 46s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 19s | the patch passed | | +1 :green_heart: | compile | 2m 38s | the patch passed | | +1 :green_heart: | javac | 2m 38s | the patch passed | | +1 :green_heart: | shadedjars | 6m 56s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 46s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 323m 24s | root in the patch failed. | | | | 361m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1978/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1978 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux faea2b4af2ba 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c0461207ee | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1978/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1978/1/testReport/ | | Max. process+thread count | 5750 (vs. ulimit of 12500) | | modules | C: hbase-common . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1978/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24382) Flush partial stores of region filtered by seqId when archive wal due to too many wals
[ https://issues.apache.org/jira/browse/HBASE-24382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Wang updated HBASE-24382: --- Release Note: Change the flush level from region to store when there are too many wals, so we can reduce unnessary flush task and small hfiles. > Flush partial stores of region filtered by seqId when archive wal due to too > many wals > -- > > Key: HBASE-24382 > URL: https://issues.apache.org/jira/browse/HBASE-24382 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > Fix For: 3.0.0-alpha-1 > > > When the logRoller archive the oldest wal due to too many wals, if a region > should be flushed, we flush all stores of it, but it is not necessary, maybe > we can use unflushedSeqId of store to filter them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1933: HBASE-24588 : Submit task for NormalizationPlan
Apache-HBase commented on pull request #1933: URL: https://github.com/apache/hbase/pull/1933#issuecomment-650155983 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 43s | master passed | | +1 :green_heart: | checkstyle | 1m 37s | master passed | | +1 :green_heart: | spotbugs | 2m 56s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 22s | the patch passed | | -0 :warning: | checkstyle | 1m 6s | hbase-server: The patch generated 3 new + 101 unchanged - 2 fixed = 104 total (was 103) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 19s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 3m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. | | | | 36m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1933/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1933 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 3d9b3bca292b 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c0461207ee | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1933/6/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1933/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-21782) LoadIncrementalHFiles should not be IA.Public
[ https://issues.apache.org/jira/browse/HBASE-21782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146276#comment-17146276 ] Duo Zhang commented on HBASE-21782: --- [~ndimiduk] Usually we should keep the deprecated classes for a whole major release, as what we describe in the ref guide. But here I define the IA.Public annotation on LoadIncrementalHFiles as a mistake, especially that it exposes several IA.Private classes, so I used a more aggressive policy to drop the class. IIRC I've gotten consensus from the community. We should add a special note on this in our ref guide too. > LoadIncrementalHFiles should not be IA.Public > - > > Key: HBASE-21782 > URL: https://issues.apache.org/jira/browse/HBASE-21782 > Project: HBase > Issue Type: Task > Components: mapreduce >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Labels: bulkload > Fix For: 3.0.0-alpha-1, 2.2.0 > > Attachments: HBASE-21782-v1.patch, HBASE-21782.patch > > > It is an implementation class, so some of the methods which are only supposed > to be used by replication sink are also public to users. And it exposes > methods which take Table and Connection as parameter and inside the > implementation we assume that they are HTable and ConnectionImplementation, > which will be a pain when we want to replace the sync client implementation > with async client. > Here I think we should make the implementation class as > IA.LimitPrivate(TOOL), and introduce an interface for bulking hfiles > programmatically. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24634) Promote TestLogRollingNoCluster to LargeTests
[ https://issues.apache.org/jira/browse/HBASE-24634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146284#comment-17146284 ] Duo Zhang commented on HBASE-24634: --- That's a bit strange then... Let me try again. > Promote TestLogRollingNoCluster to LargeTests > - > > Key: HBASE-24634 > URL: https://issues.apache.org/jira/browse/HBASE-24634 > Project: HBase > Issue Type: Task > Components: test >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Major > Labels: beginner, trivial > Attachments: image-2020-06-26-00-26-51-151.png > > > It spends 7 minutes to finish on my local machine, so I think it should be a > LargeTests instead of MediumTests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24562) Stabilize master startup with meta replicas enabled
[ https://issues.apache.org/jira/browse/HBASE-24562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146287#comment-17146287 ] Duo Zhang commented on HBASE-24562: --- Should this go into other branches as well? [~wchevreuil]. Thanks. > Stabilize master startup with meta replicas enabled > --- > > Key: HBASE-24562 > URL: https://issues.apache.org/jira/browse/HBASE-24562 > Project: HBase > Issue Type: Improvement > Components: meta, read replicas >Affects Versions: 3.0.0-alpha-1 >Reporter: Szabolcs Bukros >Assignee: Szabolcs Bukros >Priority: Major > Fix For: 3.0.0-alpha-1 > > > This is related to HBASE-21624 . > I created a separate ticket because in the original one a "complete solution > for meta replicas" was requested and this is not one. I'm just trying to make > master startup more stable by making assigning meta replicas asynchronous and > preventing a potential assignment failure from crashing master. > The idea is that starting master with less or even no meta replicas assigned > is preferable to not having a running master. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24562) Stabilize master startup with meta replicas enabled
[ https://issues.apache.org/jira/browse/HBASE-24562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146301#comment-17146301 ] Wellington Chevreuil commented on HBASE-24562: -- [~zhangduo], yes, but got some conflicts on the test class when trying to cherry pick into branch-2. Will try resolve those and push to base 2 branches until EOD today. > Stabilize master startup with meta replicas enabled > --- > > Key: HBASE-24562 > URL: https://issues.apache.org/jira/browse/HBASE-24562 > Project: HBase > Issue Type: Improvement > Components: meta, read replicas >Affects Versions: 3.0.0-alpha-1 >Reporter: Szabolcs Bukros >Assignee: Szabolcs Bukros >Priority: Major > Fix For: 3.0.0-alpha-1 > > > This is related to HBASE-21624 . > I created a separate ticket because in the original one a "complete solution > for meta replicas" was requested and this is not one. I'm just trying to make > master startup more stable by making assigning meta replicas asynchronous and > preventing a potential assignment failure from crashing master. > The idea is that starting master with less or even no meta replicas assigned > is preferable to not having a running master. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24639) RequestId Tracing feature for HBase
Pranshu Khandelwal created HBASE-24639: -- Summary: RequestId Tracing feature for HBase Key: HBASE-24639 URL: https://issues.apache.org/jira/browse/HBASE-24639 Project: HBase Issue Type: New Feature Reporter: Pranshu Khandelwal -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24639) RequestId Tracing feature for HBase
[ https://issues.apache.org/jira/browse/HBASE-24639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146310#comment-17146310 ] Lokesh Khurana commented on HBASE-24639: [~vjasani] Can you assign this Jira to Pranshu! > RequestId Tracing feature for HBase > > > Key: HBASE-24639 > URL: https://issues.apache.org/jira/browse/HBASE-24639 > Project: HBase > Issue Type: New Feature >Reporter: Pranshu Khandelwal >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] pranshu1200 opened a new pull request #1979: HBASE-24639 RequestId Tracing Feature for Hbase
pranshu1200 opened a new pull request #1979: URL: https://github.com/apache/hbase/pull/1979 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24639) RequestId Tracing feature for HBase
[ https://issues.apache.org/jira/browse/HBASE-24639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146341#comment-17146341 ] Viraj Jasani commented on HBASE-24639: -- [~pranshu_1999] I have requested for your access over the slack channel, you should receive contributor access soon. In the meanwhile, can you please provide some description of your proposal for everyone to understand? Thank you for your interest. > RequestId Tracing feature for HBase > > > Key: HBASE-24639 > URL: https://issues.apache.org/jira/browse/HBASE-24639 > Project: HBase > Issue Type: New Feature >Reporter: Pranshu Khandelwal >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24639) RequestId Tracing feature for HBase
[ https://issues.apache.org/jira/browse/HBASE-24639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146341#comment-17146341 ] Viraj Jasani edited comment on HBASE-24639 at 6/26/20, 2:18 PM: [~pranshu_1999] I have requested to the project committee for your access over the slack channel, you should receive contributor access in some time. In the meanwhile, can you please provide some description of your proposal for everyone to understand? Thank you for your interest. was (Author: vjasani): [~pranshu_1999] I have requested for your access over the slack channel, you should receive contributor access soon. In the meanwhile, can you please provide some description of your proposal for everyone to understand? Thank you for your interest. > RequestId Tracing feature for HBase > > > Key: HBASE-24639 > URL: https://issues.apache.org/jira/browse/HBASE-24639 > Project: HBase > Issue Type: New Feature >Reporter: Pranshu Khandelwal >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24639) RequestId Tracing feature for HBase
[ https://issues.apache.org/jira/browse/HBASE-24639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146341#comment-17146341 ] Viraj Jasani edited comment on HBASE-24639 at 6/26/20, 2:20 PM: [~pranshu_1999] I have requested to the project committee for your access over the slack channel, you should receive contributor access in some time. In the meanwhile, can you please provide some description of your proposal in the *Description* section of the Jira? Thank you for your interest. was (Author: vjasani): [~pranshu_1999] I have requested to the project committee for your access over the slack channel, you should receive contributor access in some time. In the meanwhile, can you please provide some description of your proposal for everyone to understand? Thank you for your interest. > RequestId Tracing feature for HBase > > > Key: HBASE-24639 > URL: https://issues.apache.org/jira/browse/HBASE-24639 > Project: HBase > Issue Type: New Feature >Reporter: Pranshu Khandelwal >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24639) RequestId Tracing feature for HBase
[ https://issues.apache.org/jira/browse/HBASE-24639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pranshu Khandelwal updated HBASE-24639: --- Description: Associate a TraceId corresponding to Hbase Put Request on the mutation level which will be further propagated to the RPC layer. This will enable cost-effective logging of the Request Flow and help identifying Network bottlenecks. We aim to send the TraceId to the RPC server by modifying the RequestProto. > RequestId Tracing feature for HBase > > > Key: HBASE-24639 > URL: https://issues.apache.org/jira/browse/HBASE-24639 > Project: HBase > Issue Type: New Feature >Reporter: Pranshu Khandelwal >Priority: Major > > Associate a TraceId corresponding to Hbase Put Request on the mutation level > which will be further propagated to the RPC layer. This will enable > cost-effective logging of the Request Flow and help identifying Network > bottlenecks. We aim to send the TraceId to the RPC server by modifying the > RequestProto. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24616) Remove BoundedRecoveredHFilesOutputSink dependency on a TableDescriptor
[
https://issues.apache.org/jira/browse/HBASE-24616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146348#comment-17146348
]
Hudson commented on HBASE-24616:
Results for branch master
[build #1768 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1768/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1768/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1600//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1768/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1768/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove BoundedRecoveredHFilesOutputSink dependency on a TableDescriptor
>
>
> Key: HBASE-24616
> URL: https://issues.apache.org/jira/browse/HBASE-24616
> Project: HBase
> Issue Type: Bug
> Components: HFile, MTTR
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
>
> BoundedRecoveredHFilesOutputSink wants to read TableDescriptor so it writes
> the particular hfile format specified by a table's schema. Getting the table
> schema can be tough at various points of operation especially around startup.
> HBASE-23739 tried to read from the fs if unable to read TableDescriptor from
> Master. This approach works generally but fails in standalone mode as in
> standalone mode we will have given-up our start up attempt BEFORE the request
> to Master for TableDescriptor times out (the read from fs is never attempted).
> The suggested patch here does away w/ reading TableDescriptor and just has
> BoundedRecoveredHFilesOutputSink write generic hfiles.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24631) Loosen Dockerfile pinned package versions of the "debian-revision"
[
https://issues.apache.org/jira/browse/HBASE-24631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146347#comment-17146347
]
Hudson commented on HBASE-24631:
Results for branch master
[build #1768 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1768/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1768/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1600//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1768/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1768/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Loosen Dockerfile pinned package versions of the "debian-revision"
> --
>
> Key: HBASE-24631
> URL: https://issues.apache.org/jira/browse/HBASE-24631
> Project: HBase
> Issue Type: Task
> Components: build
>Affects Versions: 3.0.0-alpha-1
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
>
> Portions of PR jobs have started failing. From the log, our versions of curl
> is no longer available in ubuntu package servers.
> {noformat}
> [2020-06-24T21:41:59.524Z]
>
> [2020-06-24T21:41:59.524Z]
>
> [2020-06-24T21:41:59.524Z]Docker Image Creation
> [2020-06-24T21:41:59.524Z]
>
> [2020-06-24T21:41:59.524Z]
>
> [2020-06-24T21:41:59.524Z]
> [2020-06-24T21:41:59.524Z]
> [2020-06-24T21:41:59.524Z] Sending build context to Docker daemon 18.99kB
> [2020-06-24T21:41:59.524Z] Step 1/60 : FROM ubuntu:18.04 AS BASE_IMAGE
> [2020-06-24T21:42:00.233Z] 18.04: Pulling from library/ubuntu
> [2020-06-24T21:42:00.943Z] Digest:
> sha256:86510528ab9cd7b64209cbbe6946e094a6d10c6db21def64a93ebdd20011de1d
> [2020-06-24T21:42:00.943Z] Status: Downloaded newer image for ubuntu:18.04
> [2020-06-24T21:42:00.943Z] ---> 8e4ce0a6ce69
> [2020-06-24T21:42:00.943Z] Step 2/60 : SHELL ["/bin/bash", "-o", "pipefail",
> "-c"]
> [2020-06-24T21:42:00.943Z] ---> Using cache
> [2020-06-24T21:42:00.943Z] ---> 9170e78be248
> [2020-06-24T21:42:00.943Z] Step 3/60 : RUN DEBIAN_FRONTEND=noninteractive
> apt-get -qq update && DEBIAN_FRONTEND=noninteractive apt-get -qq install
> --no-install-recommends -y ca-certificates=20180409
> curl=7.58.0-2ubuntu3.8 locales=2.27-3ubuntu1 bash=4.4.18-2ubuntu1.2
> build-essential=12.4ubuntu1 diffutils=1:3.6-1
> git=1:2.17.1-1ubuntu0.7 rsync=3.1.2-2.1ubuntu1 tar=1.29b-2ubuntu0.1
> wget=1.19.4-1ubuntu2.2 bats=0.4.0-1.1 libperl-critic-perl=1.130-1
> python3=3.6.7-1~18.04 python3-pip=9.0.1-2.3~ubuntu1.18.04.1
> python3-setuptools=39.0.1-2 ruby=1:2.5.1 ruby-dev=1:2.5.1
> shellcheck=0.4.6-1 && apt-get clean && rm -rf /var/lib/apt/lists/*
> [2020-06-24T21:42:02.413Z] ---> Running in 87d1f25abbc4
> [2020-06-24T21:42:11.760Z] E: Version '7.58.0-2ubuntu3.8' for 'curl' was not
> found
> [2020-06-24T21:42:12.471Z] The command '/bin/bash -o pipefail -c
> DEBIAN_FRONTEND=noninteractive apt-get -qq update &&
> DEBIAN_FRONTEND=noninteractive apt-get -qq install --no-install-recommends -y
> ca-certificates=20180409 curl=7.58.0-2ubuntu3.8
> locales=2.27-3ubuntu1 bash=4.4.18-2ubuntu1.2
> build-essential=12.4ubuntu1 diffutils=1:3.6-1 git=1:2.17.1-1ubuntu0.7
> rsync=3.1.2-2.1ubuntu1 tar=1.29b-2ubuntu0.1
> wget=1.19.4-1ubuntu2.2 bats=0.4.0-1.1 libperl-critic-perl=1.130-1
> python3=3.6.7-1~18.04 python3-pip=9.0.1-2.3~ubuntu1.18.04.1
> python3-setuptools=39.0.1-2 ruby=1:2.5.1 ruby-dev=1:2.5.1
> shellcheck=0.4.6-1 && apt-get clean && rm -rf
> /var/lib/apt/lists/*' returned a non-zero code: 100
> [2020-06-24T21:42:12.472Z] ERROR: Docker fa
[jira] [Commented] (HBASE-24603) Zookeeper sync() call is async
[
https://issues.apache.org/jira/browse/HBASE-24603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146349#comment-17146349
]
Hudson commented on HBASE-24603:
Results for branch master
[build #1768 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1768/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1768/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1600//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1768/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1768/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Zookeeper sync() call is async
> --
>
> Key: HBASE-24603
> URL: https://issues.apache.org/jira/browse/HBASE-24603
> Project: HBase
> Issue Type: Improvement
> Components: master, regionserver
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 1.7.0
>Reporter: Bharath Vissapragada
>Assignee: Bharath Vissapragada
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0
>
>
> Here is the method that does a sync() of lagging followers with leader in the
> quorum. We rely on this to see a consistent snapshot of ZK data from multiple
> clients. However the problem is that the underlying sync() call is actually
> asynchronous since we are passing a 'null' call back. See the ZK API
> [doc|https://zookeeper.apache.org/doc/r3.5.7/apidocs/zookeeper-server/index.html]
> for details. The end-result is that sync() doesn't guarantee that it has
> happened by the time it returns.
> {noformat}
> /**
>* Forces a synchronization of this ZooKeeper client connection.
>*
>* Executing this method before running other methods will ensure that the
>* subsequent operations are up-to-date and consistent as of the time that
>* the sync is complete.
>*
>* This is used for compareAndSwap type operations where we need to read the
>* data of an existing node and delete or transition that node, utilizing
> the
>* previously read version and data. We want to ensure that the version
> read
>* is up-to-date from when we begin the operation.
>*/
> public void sync(String path) throws KeeperException {
> this.recoverableZooKeeper.sync(path, null, null);
> }
> {noformat}
> We rely on this heavily (at least in the older branches that do ZK based
> region assignment). In branch-1 we saw weird "BadVersionException" exceptions
> in RITs because of the inconsistent view of the ZK snapshot. It could
> manifest differently in other branches. Either way, this is something we need
> to fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24382) Flush partial stores of region filtered by seqId when archive wal due to too many wals
[
https://issues.apache.org/jira/browse/HBASE-24382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146350#comment-17146350
]
Hudson commented on HBASE-24382:
Results for branch master
[build #1768 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1768/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1768/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1600//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1768/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1768/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Flush partial stores of region filtered by seqId when archive wal due to too
> many wals
> --
>
> Key: HBASE-24382
> URL: https://issues.apache.org/jira/browse/HBASE-24382
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> When the logRoller archive the oldest wal due to too many wals, if a region
> should be flushed, we flush all stores of it, but it is not necessary, maybe
> we can use unflushedSeqId of store to filter them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-24639) RequestId Tracing feature for HBase
[ https://issues.apache.org/jira/browse/HBASE-24639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil reassigned HBASE-24639: Assignee: Pranshu Khandelwal > RequestId Tracing feature for HBase > > > Key: HBASE-24639 > URL: https://issues.apache.org/jira/browse/HBASE-24639 > Project: HBase > Issue Type: New Feature >Reporter: Pranshu Khandelwal >Assignee: Pranshu Khandelwal >Priority: Major > > Associate a TraceId corresponding to Hbase Put Request on the mutation level > which will be further propagated to the RPC layer. This will enable > cost-effective logging of the Request Flow and help identifying Network > bottlenecks. We aim to send the TraceId to the RPC server by modifying the > RequestProto. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24639) RequestId Tracing feature for HBase
[ https://issues.apache.org/jira/browse/HBASE-24639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146354#comment-17146354 ] Wellington Chevreuil commented on HBASE-24639: -- Hi [~pranshu_1999], thanks for the contribution. I had added yourself to the list of hbase contributors and assigned this jira to yourself. From now on, you should be able to assign hbase jiras to you on your own. > RequestId Tracing feature for HBase > > > Key: HBASE-24639 > URL: https://issues.apache.org/jira/browse/HBASE-24639 > Project: HBase > Issue Type: New Feature >Reporter: Pranshu Khandelwal >Assignee: Pranshu Khandelwal >Priority: Major > > Associate a TraceId corresponding to Hbase Put Request on the mutation level > which will be further propagated to the RPC layer. This will enable > cost-effective logging of the Request Flow and help identifying Network > bottlenecks. We aim to send the TraceId to the RPC server by modifying the > RequestProto. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24382) Flush partial stores of region filtered by seqId when archive wal due to too many wals
[ https://issues.apache.org/jira/browse/HBASE-24382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Wang updated HBASE-24382: --- Release Note: Change the flush level from region to store when there are too many wals, benefit from this we can reduce unnessary flush task and small hfiles. (was: Change the flush level from region to store when there are too many wals, so we can reduce unnessary flush task and small hfiles.) > Flush partial stores of region filtered by seqId when archive wal due to too > many wals > -- > > Key: HBASE-24382 > URL: https://issues.apache.org/jira/browse/HBASE-24382 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > Fix For: 3.0.0-alpha-1 > > > When the logRoller archive the oldest wal due to too many wals, if a region > should be flushed, we flush all stores of it, but it is not necessary, maybe > we can use unflushedSeqId of store to filter them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24382) Flush partial stores of region filtered by seqId when archive wal due to too many wals
[ https://issues.apache.org/jira/browse/HBASE-24382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Wang updated HBASE-24382: --- Release Note: Change the flush level from region to store when there are too many wals, benefit from this we can reduce unnessary flush tasks and small hfiles. (was: Change the flush level from region to store when there are too many wals, benefit from this we can reduce unnessary flush task and small hfiles.) > Flush partial stores of region filtered by seqId when archive wal due to too > many wals > -- > > Key: HBASE-24382 > URL: https://issues.apache.org/jira/browse/HBASE-24382 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > Fix For: 3.0.0-alpha-1 > > > When the logRoller archive the oldest wal due to too many wals, if a region > should be flushed, we flush all stores of it, but it is not necessary, maybe > we can use unflushedSeqId of store to filter them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1933: HBASE-24588 : Submit task for NormalizationPlan
Apache-HBase commented on pull request #1933: URL: https://github.com/apache/hbase/pull/1933#issuecomment-650229669 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 28s | master passed | | +1 :green_heart: | compile | 1m 18s | master passed | | +1 :green_heart: | shadedjars | 5m 32s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 26s | the patch passed | | +1 :green_heart: | compile | 1m 19s | the patch passed | | +1 :green_heart: | javac | 1m 19s | the patch passed | | +1 :green_heart: | shadedjars | 5m 32s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 6s | hbase-client in the patch passed. | | -1 :x: | unit | 158m 48s | hbase-server in the patch failed. | | | | 185m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1933/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1933 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9c25cc56ae4b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c0461207ee | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1933/6/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1933/6/testReport/ | | Max. process+thread count | 4671 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1933/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 opened a new pull request #1980: HBASE-24635 Split TestMetaWithReplicas
Apache9 opened a new pull request #1980: URL: https://github.com/apache/hbase/pull/1980 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24634) Promote TestLogRollingNoCluster to LargeTests
[ https://issues.apache.org/jira/browse/HBASE-24634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146414#comment-17146414 ] niuyulin commented on HBASE-24634: -- Yes, please help confirm it . > Promote TestLogRollingNoCluster to LargeTests > - > > Key: HBASE-24634 > URL: https://issues.apache.org/jira/browse/HBASE-24634 > Project: HBase > Issue Type: Task > Components: test >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Major > Labels: beginner, trivial > Attachments: image-2020-06-26-00-26-51-151.png > > > It spends 7 minutes to finish on my local machine, so I think it should be a > LargeTests instead of MediumTests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-15179) Cell/DBB end-to-end on the write-path
[ https://issues.apache.org/jira/browse/HBASE-15179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-15179: -- Release Note: See http://hbase.apache.org/book.html#regionserver.offheap.writepath > Cell/DBB end-to-end on the write-path > - > > Key: HBASE-15179 > URL: https://issues.apache.org/jira/browse/HBASE-15179 > Project: HBase > Issue Type: Umbrella > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Major > Fix For: 2.0.0 > > > Umbrella jira to make the HBase write path off heap E2E. We have to make sure > we have Cells flowing in entire write path. Starting from request received in > RPC layer, till the Cells get flushed out as HFiles, we have to keep the Cell > data off heap. > https://docs.google.com/document/d/1fj5P8JeutQ-Uadb29ChDscMuMaJqaMNRI86C4k5S1rQ/edit -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on a change in pull request #1977: HBASE-24221 addendum to restore public interface on LoadIncrementalHFiles
ndimiduk commented on a change in pull request #1977:
URL: https://github.com/apache/hbase/pull/1977#discussion_r446271450
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/tool/LoadIncrementalHFiles.java
##
@@ -803,14 +803,35 @@ private boolean checkHFilesCountPerRegionPerFamily(
*
* Protected for testing.
* @return empty list if success, list of items to retry on recoverable
failure
+ * @deprecated as of release 2.3.0. Use {@link BulkLoadHFiles} instead.
Review comment:
Unnecessary, I think, because the whole class already has this comment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1933: HBASE-24588 : Submit task for NormalizationPlan
Apache-HBase commented on pull request #1933: URL: https://github.com/apache/hbase/pull/1933#issuecomment-650259649 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 15s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 41s | master passed | | +1 :green_heart: | compile | 1m 38s | master passed | | +1 :green_heart: | shadedjars | 6m 23s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 27s | hbase-client in master failed. | | -0 :warning: | javadoc | 0m 41s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 34s | the patch passed | | +1 :green_heart: | compile | 1m 39s | the patch passed | | +1 :green_heart: | javac | 1m 39s | the patch passed | | +1 :green_heart: | shadedjars | 6m 24s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 25s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 41s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 23s | hbase-client in the patch passed. | | -1 :x: | unit | 210m 50s | hbase-server in the patch failed. | | | | 243m 33s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1933/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1933 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 7e1af498c8f7 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c0461207ee | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1933/6/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1933/6/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1933/6/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1933/6/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1933/6/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1933/6/testReport/ | | Max. process+thread count | 3589 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1933/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk merged pull request #1977: HBASE-24221 addendum to restore public interface on LoadIncrementalHFiles
ndimiduk merged pull request #1977: URL: https://github.com/apache/hbase/pull/1977 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack merged pull request #1978: HBASE-24638 Edit doc on (offheap) memory management
saintstack merged pull request #1978: URL: https://github.com/apache/hbase/pull/1978 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-24638) Edit doc on (offheap) memory management
[ https://issues.apache.org/jira/browse/HBASE-24638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-24638. --- Fix Version/s: 3.0.0-alpha-1 Assignee: Michael Stack Resolution: Fixed Merged doc change. > Edit doc on (offheap) memory management > --- > > Key: HBASE-24638 > URL: https://issues.apache.org/jira/browse/HBASE-24638 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0-alpha-1 > > > Gave it a read over to try and figure current state of memory management in > hbase-2.3.0. Updated it to reflect more of what the current state is. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk merged pull request #1981: Backport "HBASE-24221 addendum to restore public interface on LoadIncrementalHFiles" to branch-2.3
ndimiduk merged pull request #1981: URL: https://github.com/apache/hbase/pull/1981 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk opened a new pull request #1981: Backport "HBASE-24221 addendum to restore public interface on LoadIncrementalHFiles" to branch-2.3
ndimiduk opened a new pull request #1981: URL: https://github.com/apache/hbase/pull/1981 Signed-off-by: niuyulin Signed-off-by: Wellington Ramos Chevreuil Signed-off-by: Viraj Jasani This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Reopened] (HBASE-24221) Support bulkLoadHFile by family
[ https://issues.apache.org/jira/browse/HBASE-24221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk reopened HBASE-24221: -- Reopening, pardon. need branch-2.2 as well. > Support bulkLoadHFile by family > --- > > Key: HBASE-24221 > URL: https://issues.apache.org/jira/browse/HBASE-24221 > Project: HBase > Issue Type: Improvement > Components: HFile >Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.2.4 >Reporter: niuyulin >Assignee: niuyulin >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0, 2.4.0, 2.2.5 > > > Support bulkLoadHFile by family to avoid long time waiting of bulkLoadHFile > because of compacting at server side -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24221) Support bulkLoadHFile by family
[ https://issues.apache.org/jira/browse/HBASE-24221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-24221. -- Resolution: Fixed Addendum pushed to branch-2 and branch-2.3. > Support bulkLoadHFile by family > --- > > Key: HBASE-24221 > URL: https://issues.apache.org/jira/browse/HBASE-24221 > Project: HBase > Issue Type: Improvement > Components: HFile >Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.2.4 >Reporter: niuyulin >Assignee: niuyulin >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0, 2.4.0, 2.2.5 > > > Support bulkLoadHFile by family to avoid long time waiting of bulkLoadHFile > because of compacting at server side -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1980: HBASE-24635 Split TestMetaWithReplicas
Apache-HBase commented on pull request #1980: URL: https://github.com/apache/hbase/pull/1980#issuecomment-650268212 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 33s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 13s | master passed | | +1 :green_heart: | checkstyle | 1m 30s | master passed | | +1 :green_heart: | spotbugs | 2m 21s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 52s | the patch passed | | +1 :green_heart: | checkstyle | 1m 12s | hbase-server: The patch generated 0 new + 0 unchanged - 3 fixed = 0 total (was 3) | | +1 :green_heart: | checkstyle | 0m 18s | The patch passed checkstyle in hbase-it | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 46s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 3m 6s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. | | | | 40m 37s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1980/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1980 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 368a3d6459e0 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c0461207ee | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-server hbase-it U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1980/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk opened a new pull request #1982: Backport "HBASE-24221 addendum to restore public interface on LoadIncrementalHFiles" to branch-2.2
ndimiduk opened a new pull request #1982: URL: https://github.com/apache/hbase/pull/1982 Signed-off-by: niuyulin Signed-off-by: Wellington Ramos Chevreuil Signed-off-by: Viraj Jasani This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk merged pull request #1982: Backport "HBASE-24221 addendum to restore public interface on LoadIncrementalHFiles" to branch-2.2
ndimiduk merged pull request #1982: URL: https://github.com/apache/hbase/pull/1982 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-24221) Support bulkLoadHFile by family
[ https://issues.apache.org/jira/browse/HBASE-24221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-24221. -- Fix Version/s: (was: 2.4.0) Resolution: Fixed And branch-2.2. Thanks for the reviews. > Support bulkLoadHFile by family > --- > > Key: HBASE-24221 > URL: https://issues.apache.org/jira/browse/HBASE-24221 > Project: HBase > Issue Type: Improvement > Components: HFile >Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.2.4 >Reporter: niuyulin >Assignee: niuyulin >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0, 2.2.5 > > > Support bulkLoadHFile by family to avoid long time waiting of bulkLoadHFile > because of compacting at server side -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24640) Purge use of VisibleForTesting
[ https://issues.apache.org/jira/browse/HBASE-24640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-24640: - Affects Version/s: 2.4.0 3.0.0-alpha-1 > Purge use of VisibleForTesting > -- > > Key: HBASE-24640 > URL: https://issues.apache.org/jira/browse/HBASE-24640 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 3.0.0-alpha-1, 2.4.0 >Reporter: Nick Dimiduk >Priority: Major > > From the dev-list thread ["[DISCUSS] VisibleForTesting annotation as it > pertains to our API compatibility > guidelines"|https://lists.apache.org/thread.html/rc7c7c66f134fe135d0a4454a883215e26ff3d20e5a31ecd6a2d1db77%40%3Cdev.hbase.apache.org%3E], > when used in classes annotated with interface audience other than > IA.Private, the VisibleForTesting annotation is confusing and considered > harmful. The consensus is that we do not want to use this annotation as part > of the definition of our public APIs, and we need to remove the point of > confusion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24640) Purge use of VisibleForTesting
Nick Dimiduk created HBASE-24640: Summary: Purge use of VisibleForTesting Key: HBASE-24640 URL: https://issues.apache.org/jira/browse/HBASE-24640 Project: HBase Issue Type: Task Components: community Reporter: Nick Dimiduk >From the dev-list thread ["[DISCUSS] VisibleForTesting annotation as it >pertains to our API compatibility >guidelines"|https://lists.apache.org/thread.html/rc7c7c66f134fe135d0a4454a883215e26ff3d20e5a31ecd6a2d1db77%40%3Cdev.hbase.apache.org%3E], > when used in classes annotated with interface audience other than IA.Private, >the VisibleForTesting annotation is confusing and considered harmful. The >consensus is that we do not want to use this annotation as part of the >definition of our public APIs, and we need to remove the point of confusion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-20819) Use TableDescriptor to replace HTableDescriptor in hbase-shell module
[
https://issues.apache.org/jira/browse/HBASE-20819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146470#comment-17146470
]
Elliot Miller commented on HBASE-20819:
---
h2. Which Constants Should We Keep?
Below are my opinions on which constants we should keep, based on my last
comment.
h3. L1 Cache Constants
* {color:#de350b}CACHE_DATA_IN_L1{color}
** Fact: Zero usages in HBase repo (via grep)
** Fact: This constant pertains to a feature that is marked for removal in
HBase 3.0.0
*** "From HBase 2.0.0 onwards, the notions of L1 and L2 have been deprecated."
[See HBase Book|http://hbase.apache.org/book.html#offheap.blockcache]
** Conclusion: Delete it!
h3. Comparator Constants
Both of these constants (while public) are marked with
{{@InterfaceAudience.Private}}, and seem to be for internal use only. I'm
inclined to leave them out.
* {color:#de350b}COMPARATOR{color}
* {color:#de350b}COMPARATOR_IGNORE_REPLICATION{color}
h3. MOB (Medium OBjects) Constants
All three of these constants were copied from HTableDescriptor to
ColumnFamilyDescriptorBuilder. In this process, they went from public to
private and three new setters were added to ColumnFamilyDescriptorBuilder:
setMobCompactPartitionPolicy, setMobEnabled, and setMobThreshold. *I worry that
if we include the constants in the shell, it will encourage users to circumvent
our new builder interface setters and directly mess with values.* For that
reason, I think we should not include these constants.
* {color:#ff}IS_MOB_BYTES{color}
* {color:#ff}MOB_COMPACT_PARTITION_POLICY_BYTES{color}
* {color:#ff}MOB_THRESHOLD_BYTES{color}
h3. Replication Scope Constants
* {color:#00875a}REPLICATION_SCOPE_BYTES{color}
** Copied from HColumnDescriptor to ColumnFamilyDescriptor and it's public! We
should *definitely keep this one*.
h3. MISC Constants
* {color:#de350b}ENCODE_ON_DISK{color}
** Zero usages in entire HBase repo
** No longer used... In fact, there was a [comment on the definition
line|https://github.com/apache/hbase/blob/fe5865d9e3ffc7b0c0fdef4980295f6ca07298a2/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java#L74]
added in 2013 stating that it was no longer used
* {color:#00875a}FOREVER{color}
** Moved to HConstants.FOREVER
* {color:#00875a}IS_ROOT{color}
** Used in TableSchemaModel... I'm guessing the constant will be moved to
HConstants in some follow-up work to remove HTableDescriptor from the entire
repo before the 3.0.0 release. For now, I'll keep it.
* {color:#de350b}LENGTH{color}
** As far as I can tell, it hasn't been in use since
[HBASE-1304|https://github.com/apache/hbase/commit/6af4292630daca370de7fce3a2b2b3401cd1abfa]
(2009)
> Use TableDescriptor to replace HTableDescriptor in hbase-shell module
> -
>
> Key: HBASE-20819
> URL: https://issues.apache.org/jira/browse/HBASE-20819
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Affects Versions: 2.0.0
>Reporter: Xiaolin Ha
>Assignee: Elliot Miller
>Priority: Minor
> Fix For: 3.0.0-alpha-1
>
> Attachments: HBASE-20819.branch-2.001.patch,
> HBASE-20819.branch-2.002.patch,
> HBaseConstants-b5563432922268c7a16deacbb51bfba89c0a2aba.txt,
> HBaseConstants-cf2aa593e590133b0c76d3723b4074b28b55dcc9.txt,
> HBaseConstants-diff.txt
>
>
> HTableDescriptor is deprecated as of release 2.0.0, and will be removed in
> 3.0.0. This patch replaces all usages of HTableDescriptor and
> HColumnDescriptor in the hbase-shell module so that HTableDescriptor can be
> removed.
> There a few other consequences of this change:
> * Ruby methods relating to HTableDescriptor and HColumnDescriptor have been
> removed. This is noted in "Release Note" on this issue.
> * We no longer import constants from HTableDescriptor and HColumnDescriptor
> into the ruby HBaseConstants module. Instead, we import them from
> ColumnFamilyDescriptorBuilder and TableDescriptorBuilder.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] virajjasani merged pull request #1933: HBASE-24588 : Submit task for NormalizationPlan
virajjasani merged pull request #1933: URL: https://github.com/apache/hbase/pull/1933 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24588) Normalizer plan execution is not consistent between plan types
[
https://issues.apache.org/jira/browse/HBASE-24588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146475#comment-17146475
]
Viraj Jasani commented on HBASE-24588:
--
Thank you [~ndimiduk] for the review.
I believe we can have this in 2.3, but I am fine with 2.3.1 also, you have
already taken so many such last moment Jiras and I don't want to give more
headache :)
So let me keep this till branch-2 for now?
> Normalizer plan execution is not consistent between plan types
> --
>
> Key: HBASE-24588
> URL: https://issues.apache.org/jira/browse/HBASE-24588
> Project: HBase
> Issue Type: Bug
> Components: master, Normalizer
>Affects Versions: 2.3.0
>Reporter: Nick Dimiduk
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0
>
>
> I left a comment on a merged
> [commit|https://github.com/apache/hbase/commit/5d0e0fc5fd09bddb2d766d1e24e28e472961f454#r39987289]
> where a little discussion has blossomed. Right now the normalizer produces
> two types of plans: "split" or "merge". The master receives the list of plans
> and executes them in a simple loop. The way it does the actual execution is
> delegated off to the plan implementation.
> The bug I noticed is that the two implementations are subtly different. Both
> use async APIs to submit procedures, but "split" blocks on completion while
> "merge" does not. Furthermore, because "split" blocks, it's able to capture
> any exception that's thrown, while "merge" cannot.
> These implementations should be made consistent. My thinking at the moment is
> this {{execute}} method should instead be named {{submit}}, creating and
> returning the {{Future}} that represents whatever work it submitted, and the
> calling context should handle the resolution of those futures in a single
> place.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] bitoffdev commented on a change in pull request #1959: HBASE-20819 Use TableDescriptor to replace HTableDescriptor in hbase-shell module
bitoffdev commented on a change in pull request #1959: URL: https://github.com/apache/hbase/pull/1959#discussion_r446296466 ## File path: hbase-shell/src/main/ruby/hbase_constants.rb ## @@ -109,8 +109,8 @@ def self.promote_constants(constants) end end - promote_constants(org.apache.hadoop.hbase.HColumnDescriptor.constants) - promote_constants(org.apache.hadoop.hbase.HTableDescriptor.constants) + promote_constants(org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder.constants) Review comment: I just pushed a commit that adds back three constants, but leaves the rest out. I left a [comment on Jira](https://issues.apache.org/jira/browse/HBASE-20819?focusedCommentId=17146470#comment-17146470) with my rationale for each of the constants. **Let me know if you disagree with any of my conclusions, and I can easily add more constants back** 😄. Specifically for the [MOB constants](https://github.com/apache/hbase/blob/025ddce868eb06b4072b5152c5ffae5a01e7ae30/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ColumnFamilyDescriptorBuilder.java#L163), we added setters for all the MOB key-value pairs to our builder. I think including these _private_ constants will encourage users to go against our defined interface. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] busbey commented on a change in pull request #1959: HBASE-20819 Use TableDescriptor to replace HTableDescriptor in hbase-shell module
busbey commented on a change in pull request #1959: URL: https://github.com/apache/hbase/pull/1959#discussion_r446299722 ## File path: hbase-shell/src/main/ruby/hbase_constants.rb ## @@ -109,8 +109,8 @@ def self.promote_constants(constants) end end - promote_constants(org.apache.hadoop.hbase.HColumnDescriptor.constants) - promote_constants(org.apache.hadoop.hbase.HTableDescriptor.constants) + promote_constants(org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder.constants) Review comment: That's great work, thanks! Analysis looks reasonable to me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24641) Add linter checks for patches
Bharath Vissapragada created HBASE-24641: Summary: Add linter checks for patches Key: HBASE-24641 URL: https://issues.apache.org/jira/browse/HBASE-24641 Project: HBase Issue Type: Sub-task Components: Client, native-client Reporter: Bharath Vissapragada I've worked with clang-tidy before and I think it works well. There is also a [helper script|https://clang.llvm.org/extra/doxygen/clang-tidy-diff_8py_source.html] that runs clang-tidy on a patch rather than the entire source tree. - pull in clang as a dependency - Define .clang-format - Run clang-tidy-diff on each checkin There are also other tools like cpplint from Google. I don't have any preference, so either works for me. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24641) Add linter checks for patches
[ https://issues.apache.org/jira/browse/HBASE-24641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146488#comment-17146488 ] Bharath Vissapragada commented on HBASE-24641: -- [~phrocker] FYI. Also, do you have a preference for a static analysis tool? > Add linter checks for patches > - > > Key: HBASE-24641 > URL: https://issues.apache.org/jira/browse/HBASE-24641 > Project: HBase > Issue Type: Sub-task > Components: Client, native-client >Reporter: Bharath Vissapragada >Priority: Major > > I've worked with clang-tidy before and I think it works well. There is also a > [helper > script|https://clang.llvm.org/extra/doxygen/clang-tidy-diff_8py_source.html] > that runs clang-tidy on a patch rather than the entire source tree. > - pull in clang as a dependency > - Define .clang-format > - Run clang-tidy-diff on each checkin > There are also other tools like cpplint from Google. I don't have any > preference, so either works for me. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24642) Apache Yetus integration
Bharath Vissapragada created HBASE-24642: Summary: Apache Yetus integration Key: HBASE-24642 URL: https://issues.apache.org/jira/browse/HBASE-24642 Project: HBase Issue Type: Sub-task Reporter: Bharath Vissapragada Assignee: Bharath Vissapragada Now that we have a clean test run with all the tests passing on trunk (with HBase trunk) , lets get the Yetus integration with test-patch utility working. That makes it much easier to implement a precommit with github. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase-native-client] bharathv commented on a change in pull request #6: HBASE-23105: Download lib double conversion, fizz, update folly
bharathv commented on a change in pull request #6: URL: https://github.com/apache/hbase-native-client/pull/6#discussion_r446305722 ## File path: cmake/DownloadDoubleConversion.cmake ## @@ -0,0 +1,34 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +## Download facebook's folly library. +## SOURCE_DIR is typically the cmake source directory +function(download_doubleconversion SOURCE_DIR BUILD_DIR) Review comment: Here you go. https://issues.apache.org/jira/browse/HBASE-24641. I think this one makes everyone's life easier :-) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #1933: HBASE-24588 : Submit task for NormalizationPlan
ndimiduk commented on pull request #1933: URL: https://github.com/apache/hbase/pull/1933#issuecomment-650292872 Looks like there were consistent unit test failures in TestNormalizerOnCluster. let's keep an eye on it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24588) Normalizer plan execution is not consistent between plan types
[
https://issues.apache.org/jira/browse/HBASE-24588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146494#comment-17146494
]
Nick Dimiduk commented on HBASE-24588:
--
This is a good fix as the existing behavior is broken. +1 for branch-2.3 and
2.3.0rc1.
> Normalizer plan execution is not consistent between plan types
> --
>
> Key: HBASE-24588
> URL: https://issues.apache.org/jira/browse/HBASE-24588
> Project: HBase
> Issue Type: Bug
> Components: master, Normalizer
>Affects Versions: 2.3.0
>Reporter: Nick Dimiduk
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0
>
>
> I left a comment on a merged
> [commit|https://github.com/apache/hbase/commit/5d0e0fc5fd09bddb2d766d1e24e28e472961f454#r39987289]
> where a little discussion has blossomed. Right now the normalizer produces
> two types of plans: "split" or "merge". The master receives the list of plans
> and executes them in a simple loop. The way it does the actual execution is
> delegated off to the plan implementation.
> The bug I noticed is that the two implementations are subtly different. Both
> use async APIs to submit procedures, but "split" blocks on completion while
> "merge" does not. Furthermore, because "split" blocks, it's able to capture
> any exception that's thrown, while "merge" cannot.
> These implementations should be made consistent. My thinking at the moment is
> this {{execute}} method should instead be named {{submit}}, creating and
> returning the {{Future}} that represents whatever work it submitted, and the
> calling context should handle the resolution of those futures in a single
> place.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24588) Normalizer plan execution is not consistent between plan types
[
https://issues.apache.org/jira/browse/HBASE-24588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146500#comment-17146500
]
Viraj Jasani commented on HBASE-24588:
--
True, let me create two backport PRs for branch-2 and branch-2.3. After QA
build results, we can merge them.
> Normalizer plan execution is not consistent between plan types
> --
>
> Key: HBASE-24588
> URL: https://issues.apache.org/jira/browse/HBASE-24588
> Project: HBase
> Issue Type: Bug
> Components: master, Normalizer
>Affects Versions: 2.3.0
>Reporter: Nick Dimiduk
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0
>
>
> I left a comment on a merged
> [commit|https://github.com/apache/hbase/commit/5d0e0fc5fd09bddb2d766d1e24e28e472961f454#r39987289]
> where a little discussion has blossomed. Right now the normalizer produces
> two types of plans: "split" or "merge". The master receives the list of plans
> and executes them in a simple loop. The way it does the actual execution is
> delegated off to the plan implementation.
> The bug I noticed is that the two implementations are subtly different. Both
> use async APIs to submit procedures, but "split" blocks on completion while
> "merge" does not. Furthermore, because "split" blocks, it's able to capture
> any exception that's thrown, while "merge" cannot.
> These implementations should be made consistent. My thinking at the moment is
> this {{execute}} method should instead be named {{submit}}, creating and
> returning the {{Future}} that represents whatever work it submitted, and the
> calling context should handle the resolution of those futures in a single
> place.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1959: HBASE-20819 Use TableDescriptor to replace HTableDescriptor in hbase-shell module
Apache-HBase commented on pull request #1959: URL: https://github.com/apache/hbase/pull/1959#issuecomment-650296731 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 26s | master passed | | +1 :green_heart: | compile | 0m 48s | master passed | | +1 :green_heart: | shadedjars | 5m 35s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 27s | the patch passed | | +1 :green_heart: | compile | 0m 46s | the patch passed | | +1 :green_heart: | javac | 0m 46s | the patch passed | | +1 :green_heart: | shadedjars | 5m 34s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 2s | hbase-client in the patch passed. | | -1 :x: | unit | 2m 11s | hbase-shell in the patch failed. | | | | 26m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1959 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux dc1f385806fd 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 71af97cd05 | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-shell.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/testReport/ | | Max. process+thread count | 2106 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-shell U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1959: HBASE-20819 Use TableDescriptor to replace HTableDescriptor in hbase-shell module
Apache-HBase commented on pull request #1959: URL: https://github.com/apache/hbase/pull/1959#issuecomment-650297656 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 15s | master passed | | +1 :green_heart: | compile | 0m 50s | master passed | | +1 :green_heart: | shadedjars | 5m 45s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 28s | hbase-client in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | compile | 0m 50s | the patch passed | | +1 :green_heart: | javac | 0m 50s | the patch passed | | +1 :green_heart: | shadedjars | 5m 46s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 26s | hbase-client in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 9s | hbase-client in the patch passed. | | -1 :x: | unit | 1m 41s | hbase-shell in the patch failed. | | | | 28m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1959 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 14d0db47cea6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 71af97cd05 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-shell.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/testReport/ | | Max. process+thread count | 2129 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-shell U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24643) Replace Cluster#primariesOfRegionsPerServer from int array to treemap
Huaxiang Sun created HBASE-24643: Summary: Replace Cluster#primariesOfRegionsPerServer from int array to treemap Key: HBASE-24643 URL: https://issues.apache.org/jira/browse/HBASE-24643 Project: HBase Issue Type: Improvement Components: Balancer Affects Versions: 2.3.0 Reporter: Huaxiang Sun Assignee: Huaxiang Sun Currently, primariesOfRegionsPerServer is an int array, moveRegion does heavy work by searching the array (linearly) and insert/remove an element requires allocating/copying the whole array. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] wchevreuil commented on pull request #1883: HBASE-24530 Introduce a split policy similar with SteppingSplitPolicy…
wchevreuil commented on pull request #1883: URL: https://github.com/apache/hbase/pull/1883#issuecomment-650299753 @bsglz , sorry for the delay. So majority of people on the DISCUSS thread opined towards #3. I think @busbey suggestion to fix IncreasingToUpperBoundRegionSplitPolicy for minor updates only is reasonable, so maybe we can merge this current one to branch-2.2 and branch-2.3, then for master and branch-2 we fix IncreasingToUpperBoundRegionSplitPolicy. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani opened a new pull request #1983: HBASE-24588 : Submit task for NormalizationPlan (#1933)
virajjasani opened a new pull request #1983: URL: https://github.com/apache/hbase/pull/1983 Signed-off-by: Nick Dimiduk This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24644) Add a clause to the book noting that sometimes we short-circuit the deprecation cycle
[
https://issues.apache.org/jira/browse/HBASE-24644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146525#comment-17146525
]
Nick Dimiduk commented on HBASE-24644:
--
{quote}
Usually we should keep the deprecated classes for a whole major release, as
what we describe in the ref guide.
But here I define the IA.Public annotation on LoadIncrementalHFiles as a
mistake, especially that it exposes several IA.Private classes, so I used a
more aggressive policy to drop the class.
IIRC I've gotten consensus from the community. We should add a special note on
this in our ref guide too.
{quote}
> Add a clause to the book noting that sometimes we short-circuit the
> deprecation cycle
> -
>
> Key: HBASE-24644
> URL: https://issues.apache.org/jira/browse/HBASE-24644
> Project: HBase
> Issue Type: Task
> Components: community, documentation
>Reporter: Nick Dimiduk
>Priority: Minor
>
> Let's add a note to the book that describes the circumstances around
> HBASE-21782 and how that can result in code not following our stated
> deprecation cycle guidelines.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24644) Add a clause to the book noting that sometimes we short-circuit the deprecation cycle
Nick Dimiduk created HBASE-24644: Summary: Add a clause to the book noting that sometimes we short-circuit the deprecation cycle Key: HBASE-24644 URL: https://issues.apache.org/jira/browse/HBASE-24644 Project: HBase Issue Type: Task Components: community, documentation Reporter: Nick Dimiduk Let's add a note to the book that describes the circumstances around HBASE-21782 and how that can result in code not following our stated deprecation cycle guidelines. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24562) Stabilize master startup with meta replicas enabled
[ https://issues.apache.org/jira/browse/HBASE-24562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146526#comment-17146526 ] Wellington Chevreuil commented on HBASE-24562: -- Had fixed conflicts and pushed into branch-2 and branch-2.3. Tried on branch-2.2 as well, but got TestMetaWithReplicas.testFailedReplicaAssigment test failing. [~bszabolcs], could you make a branch-2.2 compatible PR? > Stabilize master startup with meta replicas enabled > --- > > Key: HBASE-24562 > URL: https://issues.apache.org/jira/browse/HBASE-24562 > Project: HBase > Issue Type: Improvement > Components: meta, read replicas >Affects Versions: 3.0.0-alpha-1 >Reporter: Szabolcs Bukros >Assignee: Szabolcs Bukros >Priority: Major > Fix For: 3.0.0-alpha-1 > > > This is related to HBASE-21624 . > I created a separate ticket because in the original one a "complete solution > for meta replicas" was requested and this is not one. I'm just trying to make > master startup more stable by making assigning meta replicas asynchronous and > preventing a potential assignment failure from crashing master. > The idea is that starting master with less or even no meta replicas assigned > is preferable to not having a running master. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-24644) Add a clause to the book noting that sometimes we short-circuit the deprecation cycle
[ https://issues.apache.org/jira/browse/HBASE-24644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk reassigned HBASE-24644: Assignee: Nick Dimiduk > Add a clause to the book noting that sometimes we short-circuit the > deprecation cycle > - > > Key: HBASE-24644 > URL: https://issues.apache.org/jira/browse/HBASE-24644 > Project: HBase > Issue Type: Task > Components: community, documentation >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > > Let's add a note to the book that describes the circumstances around > HBASE-21782 and how that can result in code not following our stated > deprecation cycle guidelines. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24562) Stabilize master startup with meta replicas enabled
[ https://issues.apache.org/jira/browse/HBASE-24562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-24562: - Fix Version/s: 2.4.0 2.3.1 > Stabilize master startup with meta replicas enabled > --- > > Key: HBASE-24562 > URL: https://issues.apache.org/jira/browse/HBASE-24562 > Project: HBase > Issue Type: Improvement > Components: meta, read replicas >Affects Versions: 3.0.0-alpha-1 >Reporter: Szabolcs Bukros >Assignee: Szabolcs Bukros >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0 > > > This is related to HBASE-21624 . > I created a separate ticket because in the original one a "complete solution > for meta replicas" was requested and this is not one. I'm just trying to make > master startup more stable by making assigning meta replicas asynchronous and > preventing a potential assignment failure from crashing master. > The idea is that starting master with less or even no meta replicas assigned > is preferable to not having a running master. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24562) Stabilize master startup with meta replicas enabled
[ https://issues.apache.org/jira/browse/HBASE-24562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-24562: - Affects Version/s: 2.4.0 2.3.0 2.2.5 > Stabilize master startup with meta replicas enabled > --- > > Key: HBASE-24562 > URL: https://issues.apache.org/jira/browse/HBASE-24562 > Project: HBase > Issue Type: Improvement > Components: meta, read replicas >Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.4.0, 2.2.5 >Reporter: Szabolcs Bukros >Assignee: Szabolcs Bukros >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0 > > > This is related to HBASE-21624 . > I created a separate ticket because in the original one a "complete solution > for meta replicas" was requested and this is not one. I'm just trying to make > master startup more stable by making assigning meta replicas asynchronous and > preventing a potential assignment failure from crashing master. > The idea is that starting master with less or even no meta replicas assigned > is preferable to not having a running master. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk opened a new pull request #1984: HBASE-24644 Add a clause to the book noting that sometimes we short-circuit the deprecation cycle
ndimiduk opened a new pull request #1984: URL: https://github.com/apache/hbase/pull/1984 Here's an exception where we willingly don't follow the guidelines as stated. Let's make that clear. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani opened a new pull request #1985: HBASE-24588 : Submit task for NormalizationPlan (#1933)
virajjasani opened a new pull request #1985: URL: https://github.com/apache/hbase/pull/1985 Signed-off-by: Nick Dimiduk This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Work started] (HBASE-24644) Add a clause to the book noting that sometimes we short-circuit the deprecation cycle
[ https://issues.apache.org/jira/browse/HBASE-24644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-24644 started by Nick Dimiduk. > Add a clause to the book noting that sometimes we short-circuit the > deprecation cycle > - > > Key: HBASE-24644 > URL: https://issues.apache.org/jira/browse/HBASE-24644 > Project: HBase > Issue Type: Task > Components: community, documentation >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > > Let's add a note to the book that describes the circumstances around > HBASE-21782 and how that can result in code not following our stated > deprecation cycle guidelines. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-21782) LoadIncrementalHFiles should not be IA.Public
[ https://issues.apache.org/jira/browse/HBASE-21782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146529#comment-17146529 ] Nick Dimiduk commented on HBASE-21782: -- And there's a release note as well. Yes, I think this issue is will documented. Filed HBASE-24644 and posted a patch. > LoadIncrementalHFiles should not be IA.Public > - > > Key: HBASE-21782 > URL: https://issues.apache.org/jira/browse/HBASE-21782 > Project: HBase > Issue Type: Task > Components: mapreduce >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Labels: bulkload > Fix For: 3.0.0-alpha-1, 2.2.0 > > Attachments: HBASE-21782-v1.patch, HBASE-21782.patch > > > It is an implementation class, so some of the methods which are only supposed > to be used by replication sink are also public to users. And it exposes > methods which take Table and Connection as parameter and inside the > implementation we assume that they are HTable and ConnectionImplementation, > which will be a pain when we want to replace the sync client implementation > with async client. > Here I think we should make the implementation class as > IA.LimitPrivate(TOOL), and introduce an interface for bulking hfiles > programmatically. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1959: HBASE-20819 Use TableDescriptor to replace HTableDescriptor in hbase-shell module
Apache-HBase commented on pull request #1959: URL: https://github.com/apache/hbase/pull/1959#issuecomment-650304489 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 47s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 3s | master passed | | +1 :green_heart: | checkstyle | 0m 44s | master passed | | +1 :green_heart: | spotbugs | 1m 16s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 38s | the patch passed | | -0 :warning: | checkstyle | 0m 35s | hbase-client: The patch generated 1 new + 27 unchanged - 0 fixed = 28 total (was 27) | | -0 :warning: | rubocop | 0m 23s | The patch generated 116 new + 478 unchanged - 95 fixed = 594 total (was 573) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 14m 44s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 1m 30s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. | | | | 41m 23s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1959 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle rubocop | | uname | Linux 691d482fbab7 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 71af97cd05 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-client.txt | | rubocop | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-shell U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 rubocop=0.80.0 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #1922: HBASE-24583 Normalizer can't actually merge empty regions when neighbor is larger than average size
ndimiduk commented on a change in pull request #1922:
URL: https://github.com/apache/hbase/pull/1922#discussion_r446319764
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/MergeNormalizationPlan.java
##
@@ -78,4 +82,30 @@ public void execute(Admin admin) {
LOG.error("Error during region merge: ", ex);
}
}
+
+ @Override
+ public boolean equals(Object o) {
+if (this == o) {
+ return true;
+}
+
+if (o == null || getClass() != o.getClass()) {
+ return false;
+}
+
+MergeNormalizationPlan that = (MergeNormalizationPlan) o;
+
+return new EqualsBuilder()
+ .append(firstRegion, that.firstRegion)
+ .append(secondRegion, that.secondRegion)
+ .isEquals();
+ }
+
+ @Override
+ public int hashCode() {
Review comment:
Actually, I think i'd like to keep them. I'm finding they're handy for
some of the experimental changes I've been trying. I'd prefer keep them for now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1984: HBASE-24644 Add a clause to the book noting that sometimes we short-circuit the deprecation cycle
Apache-HBase commented on pull request #1984: URL: https://github.com/apache/hbase/pull/1984#issuecomment-650309424 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #1922: HBASE-24583 Normalizer can't actually merge empty regions when neighbor is larger than average size
ndimiduk commented on a change in pull request #1922:
URL: https://github.com/apache/hbase/pull/1922#discussion_r446323076
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/MergeNormalizationPlan.java
##
@@ -78,4 +82,30 @@ public void execute(Admin admin) {
LOG.error("Error during region merge: ", ex);
}
}
+
+ @Override
+ public boolean equals(Object o) {
+if (this == o) {
+ return true;
+}
+
+if (o == null || getClass() != o.getClass()) {
+ return false;
+}
+
+MergeNormalizationPlan that = (MergeNormalizationPlan) o;
+
+return new EqualsBuilder()
+ .append(firstRegion, that.firstRegion)
+ .append(secondRegion, that.secondRegion)
+ .isEquals();
+ }
+
+ @Override
+ public int hashCode() {
+return new HashCodeBuilder(17, 37)
Review comment:
From
[javadoc](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Object.html#equals(java.lang.Object))
on `Object#equals(Object)`
> Note that it is generally necessary to override the `hashCode` method
whenever this method is overridden, so as to maintain the general contract for
the `hashCode` method, which states that equal objects must have equal hash
codes.
And besides since I want to treat these objects like POJOs, it is useful to
be able to use them in maps and sets.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #1922: HBASE-24583 Normalizer can't actually merge empty regions when neighbor is larger than average size
ndimiduk commented on a change in pull request #1922:
URL: https://github.com/apache/hbase/pull/1922#discussion_r446324754
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -369,7 +369,8 @@ private boolean skipForMerge(final RegionStates
regionStates, final RegionInfo r
}
final long currentSizeMb = getRegionSizeMB(current);
final long nextSizeMb = getRegionSizeMB(next);
- if (currentSizeMb + nextSizeMb < avgRegionSizeMb) {
+ // always merge away empty regions when they present themselves.
+ if (currentSizeMb == 0 || nextSizeMb == 0 || currentSizeMb + nextSizeMb
< avgRegionSizeMb) {
Review comment:
@huaxiangsun I've been thinking about this and haven't come to a formula
that I like. So let me propose this: rather than basing this decision on a
scaling factor, how about we decide on some arbitrary value to use as meaning
"effectively 0" in size. Say, any region <= 10mb will be considered as
"effectively 0-sized" and we'll merge it into it's neighbor when possible. I
supposed we can make this configurable as well...
What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1979: HBASE-24639 RequestId Tracing Feature for Hbase
Apache-HBase commented on pull request #1979: URL: https://github.com/apache/hbase/pull/1979#issuecomment-650311420 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 20s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ branch-1.4 Compile Tests _ | | +0 :ok: | mvndep | 2m 18s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 7m 43s | branch-1.4 passed | | +1 :green_heart: | compile | 1m 28s | branch-1.4 passed with JDK v1.8.0_252 | | +1 :green_heart: | compile | 1m 39s | branch-1.4 passed with JDK v1.7.0_262 | | +1 :green_heart: | checkstyle | 5m 31s | branch-1.4 passed | | +1 :green_heart: | shadedjars | 3m 11s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 7s | branch-1.4 passed with JDK v1.8.0_252 | | +1 :green_heart: | javadoc | 1m 21s | branch-1.4 passed with JDK v1.7.0_262 | | +0 :ok: | spotbugs | 2m 48s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 6m 44s | branch-1.4 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 4s | the patch passed | | +1 :green_heart: | compile | 1m 27s | the patch passed with JDK v1.8.0_252 | | +1 :green_heart: | cc | 1m 27s | the patch passed | | +1 :green_heart: | javac | 1m 27s | the patch passed | | +1 :green_heart: | compile | 1m 39s | the patch passed with JDK v1.7.0_262 | | +1 :green_heart: | cc | 1m 39s | the patch passed | | +1 :green_heart: | javac | 1m 39s | the patch passed | | -1 :x: | checkstyle | 0m 44s | hbase-client: The patch generated 26 new + 351 unchanged - 5 fixed = 377 total (was 356) | | -1 :x: | checkstyle | 1m 44s | hbase-server: The patch generated 20 new + 90 unchanged - 2 fixed = 110 total (was 92) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 3m 0s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 2m 21s | Patch does not cause any errors with Hadoop 2.7.7. | | +1 :green_heart: | hbaseprotoc | 1m 54s | the patch passed | | +1 :green_heart: | javadoc | 1m 0s | the patch passed with JDK v1.8.0_252 | | +1 :green_heart: | javadoc | 1m 18s | the patch passed with JDK v1.7.0_262 | | +1 :green_heart: | findbugs | 7m 10s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 28s | hbase-protocol in the patch passed. | | -1 :x: | unit | 2m 24s | hbase-client in the patch failed. | | -1 :x: | unit | 148m 11s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 1m 1s | The patch does not generate ASF License warnings. | | | | 216m 21s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.client.TestAsyncProcess | | | hadoop.hbase.regionserver.TestRegionServerMetrics | | | hadoop.hbase.client.TestReplicasClient | | | hadoop.hbase.client.TestRpcControllerFactory | | | hadoop.hbase.client.TestMalformedCellFromClient | | | hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFiles | | | hadoop.hbase.client.TestSnapshotCloneIndependence | | | hadoop.hbase.coprocessor.TestHTableWrapper | | | hadoop.hbase.coprocessor.TestBatchCoprocessorEndpoint | | | hadoop.hbase.client.TestFromClientSide3 | | | hadoop.hbase.client.TestMultiParallel | | | hadoop.hbase.client.TestMultiRespectsLimits | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1979/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1979 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile cc hbaseprotoc prototool | | uname | Linux 838b63b04572 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Person
[GitHub] [hbase] bsglz commented on pull request #1883: HBASE-24530 Introduce a split policy similar with SteppingSplitPolicy…
bsglz commented on pull request #1883: URL: https://github.com/apache/hbase/pull/1883#issuecomment-650316101 > @bsglz , sorry for the delay. So majority of people on the DISCUSS thread opined towards #3. I think @busbey suggestion to fix IncreasingToUpperBoundRegionSplitPolicy for minor updates only is reasonable, so maybe we can merge this current one to branch-2.2 and branch-2.3, then for master and branch-2 we fix IncreasingToUpperBoundRegionSplitPolicy. Ok, will do it in a follow-on issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bsglz edited a comment on pull request #1883: HBASE-24530 Introduce a split policy similar with SteppingSplitPolicy…
bsglz edited a comment on pull request #1883: URL: https://github.com/apache/hbase/pull/1883#issuecomment-650316101 > @bsglz , sorry for the delay. So majority of people on the DISCUSS thread opined towards #3. I think @busbey suggestion to fix IncreasingToUpperBoundRegionSplitPolicy for minor updates only is reasonable, so maybe we can merge this current one to branch-2.2 and branch-2.3, then for master and branch-2 we fix IncreasingToUpperBoundRegionSplitPolicy. Ok, will do it in a follow-on issue. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1984: HBASE-24644 Add a clause to the book noting that sometimes we short-circuit the deprecation cycle
Apache-HBase commented on pull request #1984: URL: https://github.com/apache/hbase/pull/1984#issuecomment-650318684 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 39s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 21s | master passed | | +0 :ok: | refguide | 5m 33s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 47s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +0 :ok: | refguide | 5m 51s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 22m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1984/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1984 | | Optional Tests | dupname asflicense refguide | | uname | Linux 02c0823a279e 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 51ad588565 | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1984/1/artifact/yetus-general-check/output/branch-site/book.html | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1984/1/artifact/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 66 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1984/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
