[GitHub] [hbase-thirdparty] Apache-HBase commented on pull request #57: HBASE-25609 [hbase-thirdparty] Bump version to 4.0.0-SNAPSHOT on mast…
Apache-HBase commented on pull request #57: URL: https://github.com/apache/hbase-thirdparty/pull/57#issuecomment-981371971 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 1s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 5s | Maven dependency ordering for branch | | -1 :x: | mvninstall | 0m 12s | root in master failed. | | -1 :x: | compile | 0m 5s | hbase-shaded-jetty in master failed. | | +1 :green_heart: | javadoc | 0m 41s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 25s | master passed | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 33s | the patch passed | | +1 :green_heart: | compile | 0m 48s | the patch passed | | +1 :green_heart: | javac | 0m 48s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 6s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 10s | hbase-shaded-protobuf in the patch passed. | | +1 :green_heart: | unit | 0m 4s | hbase-shaded-netty in the patch passed. | | +1 :green_heart: | unit | 0m 4s | hbase-shaded-gson in the patch passed. | | +1 :green_heart: | unit | 0m 5s | hbase-shaded-miscellaneous in the patch passed. | | +1 :green_heart: | unit | 0m 4s | hbase-shaded-jetty in the patch passed. | | +1 :green_heart: | unit | 0m 5s | hbase-shaded-jersey in the patch passed. | | +1 :green_heart: | unit | 0m 25s | hbase-noop-htrace in the patch passed. | | +1 :green_heart: | unit | 0m 32s | root in the patch passed. | | +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. | | | | 8m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-57/4/artifact/yetus-precommit-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase-thirdparty/pull/57 | | Optional Tests | dupname asflicense javac javadoc unit xml compile | | uname | Linux 0baba3b14ef8 5.4.0-1054-aws #57~18.04.1-Ubuntu SMP Thu Jul 15 03:21:36 UTC 2021 x86_64 GNU/Linux | | Build tool | maven | | git revision | master / 247c926 | | Default Java | Oracle Corporation-1.8.0_282-b08 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-57/4/artifact/yetus-precommit-check/output/branch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-57/4/artifact/yetus-precommit-check/output/branch-compile-hbase-shaded-jetty.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-57/4/testReport/ | | Max. process+thread count | 413 (vs. ulimit of 1000) | | modules | C: hbase-shaded-protobuf hbase-shaded-netty hbase-shaded-gson hbase-shaded-miscellaneous hbase-shaded-jetty hbase-shaded-jersey hbase-noop-htrace . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-57/4/console | | versions | git=2.20.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #3883: HBASE-26490 Add builder for class ReplicationLoadSink
Apache9 commented on a change in pull request #3883:
URL: https://github.com/apache/hbase/pull/3883#discussion_r758089951
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationLoadSink.java
##
@@ -47,4 +46,42 @@ public long getTimestampStarted() {
public long getTotalOpsProcessed() {
return totalOpsProcessed;
}
+
+ public static ReplicationLoadSinkBuilder newBuilder() {
Review comment:
I think the above IA.Priavte annotation on the constructor means we do
not want end users to create this? So I suggest that we move
ReplicationLoadSinkBuilder to a separated file, marked it as IA.Private, and
provide a create method for creating the ReplicationLoadSinkBuilder. And the
constructor of ReplicationLoadSink could be package private.
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationLoadSink.java
##
@@ -22,9 +22,8 @@
private final long timestampStarted;
private final long totalOpsProcessed;
- // TODO: add the builder for this class
@InterfaceAudience.Private
Review comment:
Since this is private now, we do not need IA.Private annotation then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-26460) Close netty channel causes regionserver crash in handleTooBigRequest
[
https://issues.apache.org/jira/browse/HBASE-26460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450229#comment-17450229
]
Xiaolin Ha commented on HBASE-26460:
[~binlijin] Do you think a corrupt request from the client made the wrong data,
so the regionserver crashed when processing the data that the request carries?
But both the netty server and the simple server only process the header of the
too big request, right? Or the request you described here is not a too big
request and the crash is not relevant to it too?
I think the too big request here is not the direct request that crashes the
regionserver. That is a request A is a normal request, and another request B is
a too big request, then the channel closed after processing the header of
request B, and released some byte buffers that contains the data of request A,
then when processing the data content of request A, the regionserver crashed.
> Close netty channel causes regionserver crash in handleTooBigRequest
>
>
> Key: HBASE-26460
> URL: https://issues.apache.org/jira/browse/HBASE-26460
> Project: HBase
> Issue Type: Bug
> Components: rpc
>Affects Versions: 3.0.0-alpha-1, 2.0.0
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Critical
>
> In HBASE-26170, I proposed the coredump problem after calling
> handleTooBigRequest, but that issue did not resolve the regionserver crash
> problem, which occurs before the WAL corruption in HBASE-24984.
> After looking through the codes, I think the problem is in CLOSE channel.
> The direct byte buffer used by RPC call request is allocated by Netty, though
> we add a reference count to record when to release the direct byte buffer,
> the byte buffer is managed by Netty actually. It is allocated from Netty
> PoolArena, and is released there.
> When the HBase ipc handler is processing a request, the Netty channel handler
> can process the channel events and message coming back in succession. When
> there is a too big request by NettyRpcFrameDecoder, the channel will be
> closed, and all the resources of the channel will be released, though there
> is HBase ipc handlers using the direct byte buffer to process previous
> requests.
> Netty provides two methods to request the pooled byte buffer, one is through
> the PoolThreadCache, each handler thread owns a private one. Another is
> through PoolArena#allocateNormal. Each ChannelHandler has a local
> PoolThreadCache.
> When a new Netty channel is created, a new ChannelHandler instance is
> created.
> And when a channel is closed, the relevant channel handler will be removed
> from the pipeline. I found this annotation in the Channel class of Netty,
> {code:java}
> It is important to call close() or close(ChannelPromise) to release all
> resources once you are done with the Channel. This ensures all resources are
> released in a proper way, i.e. filehandles. {code}
> And when channel handler is removed in ByteToMessageDecoder#handlerRemoved,
> it will release the byte buffer,
> {code:java}
> @Override
> public final void handlerRemoved(ChannelHandlerContext ctx) throws Exception {
> if (decodeState == STATE_CALLING_CHILD_DECODE) {
> decodeState = STATE_HANDLER_REMOVED_PENDING;
> return;
> }
> ByteBuf buf = cumulation;
> if (buf != null) {
> // Directly set this to null so we are sure we not access it in any
> other method here anymore.
> cumulation = null;
> int readable = buf.readableBytes();
> if (readable > 0) {
> ByteBuf bytes = buf.readBytes(readable);
> buf.release();
> ctx.fireChannelRead(bytes);
> } else {
> buf.release();
> }
> ... {code}
> We should not close the channel when encountering too big request, I think it
> should just skip the bytes like that in LengthFieldBasedFrameDecoder.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (HBASE-26323) Introduce a SnapshotProcedure
[ https://issues.apache.org/jira/browse/HBASE-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450204#comment-17450204 ] ruanhui edited comment on HBASE-26323 at 11/29/21, 5:47 AM: hi [~zhangduo] here is a simple design doc in google docs. [https://docs.google.com/document/d/1Il_PB1SenXGr1-mmCIWEogxEMeGZe2fpuN3bMbjqiGI/edit] would you mind taking a look ? I added some new details: a. some implementations about the snapshot procedure checks split/merge procedures b. the reason why we can't let the procedure register itself in the procId map which maintained by the snapshot manager. Because the procedure executor starts before the snapshot manager, so the snapshot manager has not been initialized when we replay procedures. but we may can let the snapshot procedure unregister itself. Thanks. was (Author: frostruan): hi [~zhangduo] here is a simple design doc in google docs. [https://docs.google.com/document/d/1Il_PB1SenXGr1-mmCIWEogxEMeGZe2fpuN3bMbjqiGI/edit] would you mind taking a look ? I added some new details: a. some implementation about the snapshot procedure checks split/merge procedures b. the reason why we can't let the procedure register itself in the procId map which maintained by the snapshot manager. Because the procedure executor starts before the snapshot manager, so the snapshot manager has not been initialized when we replay procedures. but we may can let the snapshot procedure unregister itself. Thanks. > Introduce a SnapshotProcedure > - > > Key: HBASE-26323 > URL: https://issues.apache.org/jira/browse/HBASE-26323 > Project: HBase > Issue Type: New Feature > Components: proc-v2, snapshots >Reporter: ruanhui >Assignee: ruanhui >Priority: Major > > Currently，snapshot in hbase uses zk as coordinator. It has some limitations, > a. Snapshot maybe fails when there are region server crashes. > b. Snapshot maybe failed when master restarts. > c. Only one snapshot per table can be taken in a time. > d. Snapshot verify will be handled by master, which may take long time when > our table has a large number of regions, for example 1. > > Since we have procedure v2 framework now, it is possible to solve the above > problems. So here is a procedure2-based snapshot implementation. It has some > goals, > a. Snapshot can continue when there are region server crashes. > b. Snapshot can continue when master restarts. > c. More than one snapshot per table can be taken in a time. > d. We can use region servers to verify snapshot to accelerate procedure. > > Here are some details about implementation. > *SnapshotProcedure* > SnapshotProcedure is used to take snapshot on a table. It acquires shared > table lock on the snapshot table and hold the shared lock during suspend and > yield. > *SnapshotRegionProcedure* > SnapshotRegionProcedure is used to take snapshot on a specific region of the > snapshot table. It acquires exclusive region lock and releases lock during > suspend and yield. Before dispatch remote snapshot operations to region > server, it will check target region in RIT or not. If target region is in > RIT, it will sleep some time and retry. > *SnapshotVerifyProcedure* > SnapshotVerifyProcedure is used to send snapshot verify request to region > server. If snapshot is corrupted, it will notify parent snapshot to retry. > When remote region server is crashed, it will choose another online server > and retry. > > I would be very grateful for any advice and guidance. Is anyone interested in > taking a look? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26323) Introduce a SnapshotProcedure
[ https://issues.apache.org/jira/browse/HBASE-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450204#comment-17450204 ] ruanhui commented on HBASE-26323: - hi [~zhangduo] here is a simple design doc in google docs. [https://docs.google.com/document/d/1Il_PB1SenXGr1-mmCIWEogxEMeGZe2fpuN3bMbjqiGI/edit] would you mind taking a look ? I added some new details: a. some implementation about the snapshot procedure checks split/merge procedures b. the reason why we can't let the procedure register itself in the procId map which maintained by the snapshot manager. Because the procedure executor starts before the snapshot manager, so the snapshot manager has not been initialized when we replay procedures. but we may can let the snapshot procedure unregister itself. Thanks. > Introduce a SnapshotProcedure > - > > Key: HBASE-26323 > URL: https://issues.apache.org/jira/browse/HBASE-26323 > Project: HBase > Issue Type: New Feature > Components: proc-v2, snapshots >Reporter: ruanhui >Assignee: ruanhui >Priority: Major > > Currently，snapshot in hbase uses zk as coordinator. It has some limitations, > a. Snapshot maybe fails when there are region server crashes. > b. Snapshot maybe failed when master restarts. > c. Only one snapshot per table can be taken in a time. > d. Snapshot verify will be handled by master, which may take long time when > our table has a large number of regions, for example 1. > > Since we have procedure v2 framework now, it is possible to solve the above > problems. So here is a procedure2-based snapshot implementation. It has some > goals, > a. Snapshot can continue when there are region server crashes. > b. Snapshot can continue when master restarts. > c. More than one snapshot per table can be taken in a time. > d. We can use region servers to verify snapshot to accelerate procedure. > > Here are some details about implementation. > *SnapshotProcedure* > SnapshotProcedure is used to take snapshot on a table. It acquires shared > table lock on the snapshot table and hold the shared lock during suspend and > yield. > *SnapshotRegionProcedure* > SnapshotRegionProcedure is used to take snapshot on a specific region of the > snapshot table. It acquires exclusive region lock and releases lock during > suspend and yield. Before dispatch remote snapshot operations to region > server, it will check target region in RIT or not. If target region is in > RIT, it will sleep some time and retry. > *SnapshotVerifyProcedure* > SnapshotVerifyProcedure is used to send snapshot verify request to region > server. If snapshot is corrupted, it will notify parent snapshot to retry. > When remote region server is crashed, it will choose another online server > and retry. > > I would be very grateful for any advice and guidance. Is anyone interested in > taking a look? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] YutSean commented on a change in pull request #3875: HBASE-26459 HMaster should move non-meta region only if meta is ONLINE
YutSean commented on a change in pull request #3875: URL: https://github.com/apache/hbase/pull/3875#discussion_r758045287 ## File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java ## @@ -136,6 +136,12 @@ /** Default value for the max percent of regions in transition */ public static final double DEFAULT_HBASE_MASTER_BALANCER_MAX_RIT_PERCENT = 1.0; + /** Time in milliseconds to wait meta region assignment, when moving non-meta regions. */ + public static final String HBASE_MASTER_WAITING_META_ASSIGNMENT_TIMEOUT = +"hbase.master.waiting.meta.assignment.timeout"; + + public static final long HBASE_MASTER_WAITING_META_ASSIGNMENT_TIMEOUT_DEFAULT = 1; Review comment: Clear. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] YutSean commented on a change in pull request #3875: HBASE-26459 HMaster should move non-meta region only if meta is ONLINE
YutSean commented on a change in pull request #3875:
URL: https://github.com/apache/hbase/pull/3875#discussion_r758045158
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
##
@@ -1845,12 +1851,21 @@ public void move(final byte[] encodedRegionName,
// closed
serverManager.sendRegionWarmup(rp.getDestination(), hri);
+ // Here wait until all the meta regions are not in transition.
+ if (!hri.isMetaRegion() &&
assignmentManager.getRegionStates().isMetaRegionInTransition()) {
+Thread.sleep(timeoutWaitMetaRegionAssignment);
+if (assignmentManager.getRegionStates().isMetaRegionInTransition()) {
+ LOG.error("This is fail-fast of the region move because "
++ "hbase:meta region is in transition. Failed region move info: "
+ rp);
+ return;
Review comment:
Right, exception could be detected by the client. Have changed to throw
exception in the latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] sunhelly commented on pull request #3855: HBASE-26422 Support priority select reference files in StripeCompactionPolicy
sunhelly commented on pull request #3855: URL: https://github.com/apache/hbase/pull/3855#issuecomment-981293686 Hi, @Apache9 , can you help review this PR at your convenience? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-connectors] Apache-HBase commented on pull request #86: HBASE-26354 [hbase-connectors] Added python client for HBase thrift service
Apache-HBase commented on pull request #86: URL: https://github.com/apache/hbase-connectors/pull/86#issuecomment-981266206 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 13s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +0 :ok: | pylint | 0m 0s | Pylint was not available. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | -0 :warning: | patch | 4m 1s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | shellcheck | 0m 0s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | | | 5m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-86/7/artifact/yetus-precommit-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase-connectors/pull/86 | | Optional Tests | dupname markdownlint pylint shellcheck shelldocs | | uname | Linux 53308e010a60 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 GNU/Linux | | Build tool | hb_maven | | Personality | dev-support/jenkins/hbase-personality.sh | | git revision | master / 9a839b2 | | Max. process+thread count | 42 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-86/7/console | | versions | git=2.20.1 shellcheck=0.5.0 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-thirdparty] Apache-HBase commented on pull request #57: HBASE-25609 [hbase-thirdparty] Bump version to 4.0.0-SNAPSHOT on mast…
Apache-HBase commented on pull request #57: URL: https://github.com/apache/hbase-thirdparty/pull/57#issuecomment-981253231 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 4s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 0m 22s | master passed | | +1 :green_heart: | compile | 0m 46s | master passed | | +1 :green_heart: | javadoc | 0m 35s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 29s | master passed | | +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 20s | the patch passed | | +1 :green_heart: | compile | 0m 47s | the patch passed | | +1 :green_heart: | javac | 0m 47s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 5s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 9s | hbase-shaded-protobuf in the patch passed. | | +1 :green_heart: | unit | 0m 4s | hbase-shaded-netty in the patch passed. | | +1 :green_heart: | unit | 0m 4s | hbase-shaded-gson in the patch passed. | | +1 :green_heart: | unit | 0m 5s | hbase-shaded-miscellaneous in the patch passed. | | +1 :green_heart: | unit | 0m 4s | hbase-shaded-jetty in the patch passed. | | +1 :green_heart: | unit | 0m 5s | hbase-shaded-jersey in the patch passed. | | +1 :green_heart: | unit | 0m 28s | hbase-noop-htrace in the patch passed. | | +1 :green_heart: | unit | 0m 32s | root in the patch passed. | | +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. | | | | 8m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-57/3/artifact/yetus-precommit-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase-thirdparty/pull/57 | | Optional Tests | dupname asflicense javac javadoc unit xml compile | | uname | Linux 193bd53a52a2 5.4.0-1054-aws #57~18.04.1-Ubuntu SMP Thu Jul 15 03:21:36 UTC 2021 x86_64 GNU/Linux | | Build tool | maven | | git revision | master / 247c926 | | Default Java | Oracle Corporation-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-57/3/testReport/ | | Max. process+thread count | 411 (vs. ulimit of 1000) | | modules | C: hbase-shaded-protobuf hbase-shaded-netty hbase-shaded-gson hbase-shaded-miscellaneous hbase-shaded-jetty hbase-shaded-jersey hbase-noop-htrace . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-57/3/console | | versions | git=2.20.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-thirdparty] Apache-HBase commented on pull request #57: HBASE-25609 [hbase-thirdparty] Bump version to 4.0.0-SNAPSHOT on mast…
Apache-HBase commented on pull request #57: URL: https://github.com/apache/hbase-thirdparty/pull/57#issuecomment-981249345 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 1m 18s | master passed | | +1 :green_heart: | compile | 0m 47s | master passed | | +1 :green_heart: | javadoc | 0m 35s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 21s | master passed | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 18s | the patch passed | | +1 :green_heart: | compile | 0m 47s | the patch passed | | +1 :green_heart: | javac | 0m 47s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 5s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 34s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 8s | hbase-shaded-protobuf in the patch passed. | | +1 :green_heart: | unit | 0m 5s | hbase-shaded-netty in the patch passed. | | +1 :green_heart: | unit | 0m 4s | hbase-shaded-gson in the patch passed. | | +1 :green_heart: | unit | 0m 4s | hbase-shaded-miscellaneous in the patch passed. | | +1 :green_heart: | unit | 0m 5s | hbase-shaded-jetty in the patch passed. | | +1 :green_heart: | unit | 0m 4s | hbase-shaded-jersey in the patch passed. | | +1 :green_heart: | unit | 0m 26s | hbase-noop-htrace in the patch passed. | | +1 :green_heart: | unit | 0m 29s | root in the patch passed. | | +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. | | | | 9m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-57/2/artifact/yetus-precommit-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase-thirdparty/pull/57 | | Optional Tests | dupname asflicense javac javadoc unit xml compile | | uname | Linux 7450045cc58b 5.4.0-1054-aws #57~18.04.1-Ubuntu SMP Thu Jul 15 03:21:36 UTC 2021 x86_64 GNU/Linux | | Build tool | maven | | git revision | master / 247c926 | | Default Java | Oracle Corporation-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-57/2/testReport/ | | Max. process+thread count | 414 (vs. ulimit of 1000) | | modules | C: hbase-shaded-protobuf hbase-shaded-netty hbase-shaded-gson hbase-shaded-miscellaneous hbase-shaded-jetty hbase-shaded-jersey hbase-noop-htrace . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-57/2/console | | versions | git=2.20.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-thirdparty] Apache-HBase commented on pull request #58: HBASE-26500 [hbase-thirdparty] Bump protobuf version to 3.19.1
Apache-HBase commented on pull request #58: URL: https://github.com/apache/hbase-thirdparty/pull/58#issuecomment-981246006 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 4s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 0m 34s | master passed | | +1 :green_heart: | compile | 0m 18s | master passed | | +1 :green_heart: | javadoc | 0m 11s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 4s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 0m 34s | the patch passed | | +1 :green_heart: | compile | 0m 18s | the patch passed | | +1 :green_heart: | javac | 0m 18s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 0s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 10s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 9s | hbase-shaded-protobuf in the patch passed. | | +1 :green_heart: | unit | 0m 30s | root in the patch passed. | | +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. | | | | 4m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-58/1/artifact/yetus-precommit-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase-thirdparty/pull/58 | | Optional Tests | dupname asflicense javac javadoc unit xml compile | | uname | Linux 642eb762453a 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 GNU/Linux | | Build tool | maven | | git revision | master / 247c926 | | Default Java | Oracle Corporation-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-58/1/testReport/ | | Max. process+thread count | 388 (vs. ulimit of 1000) | | modules | C: hbase-shaded-protobuf . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-58/1/console | | versions | git=2.20.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-26500) [hbase-thirdparty] Bump protobuf version to 3.19.1
[ https://issues.apache.org/jira/browse/HBASE-26500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-26500: - Assignee: Duo Zhang > [hbase-thirdparty] Bump protobuf version to 3.19.1 > -- > > Key: HBASE-26500 > URL: https://issues.apache.org/jira/browse/HBASE-26500 > Project: HBase > Issue Type: Sub-task > Components: hbase-thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work started] (HBASE-26500) [hbase-thirdparty] Bump protobuf version to 3.19.1
[ https://issues.apache.org/jira/browse/HBASE-26500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-26500 started by Duo Zhang. - > [hbase-thirdparty] Bump protobuf version to 3.19.1 > -- > > Key: HBASE-26500 > URL: https://issues.apache.org/jira/browse/HBASE-26500 > Project: HBase > Issue Type: Sub-task > Components: hbase-thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26500) [hbase-thirdparty] Bump protobuf version to 3.19.1
[ https://issues.apache.org/jira/browse/HBASE-26500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450165#comment-17450165 ] Duo Zhang commented on HBASE-26500: --- Ah, it is added by us, I need to apply the patch in order... > [hbase-thirdparty] Bump protobuf version to 3.19.1 > -- > > Key: HBASE-26500 > URL: https://issues.apache.org/jira/browse/HBASE-26500 > Project: HBase > Issue Type: Sub-task > Components: hbase-thirdparty >Reporter: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26500) [hbase-thirdparty] Bump protobuf version to 3.19.1
[ https://issues.apache.org/jira/browse/HBASE-26500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450164#comment-17450164 ] Duo Zhang commented on HBASE-26500: --- So seems ByteInput has been removed from protobuf... Let's see how to deal with this change. > [hbase-thirdparty] Bump protobuf version to 3.19.1 > -- > > Key: HBASE-26500 > URL: https://issues.apache.org/jira/browse/HBASE-26500 > Project: HBase > Issue Type: Sub-task > Components: hbase-thirdparty >Reporter: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase-thirdparty] Apache-HBase commented on pull request #57: HBASE-25609 [hbase-thirdparty] Bump version to 4.0.0-SNAPSHOT on mast…
Apache-HBase commented on pull request #57: URL: https://github.com/apache/hbase-thirdparty/pull/57#issuecomment-981217963 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 0m 33s | master passed | | +1 :green_heart: | compile | 0m 47s | master passed | | +1 :green_heart: | javadoc | 0m 36s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 22s | master passed | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 40s | the patch passed | | +1 :green_heart: | compile | 0m 56s | the patch passed | | +1 :green_heart: | javac | 0m 56s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 6s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 38s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 12s | hbase-shaded-protobuf in the patch passed. | | +1 :green_heart: | unit | 0m 4s | hbase-shaded-netty in the patch passed. | | +1 :green_heart: | unit | 0m 6s | hbase-shaded-gson in the patch passed. | | +1 :green_heart: | unit | 0m 4s | hbase-shaded-miscellaneous in the patch passed. | | +1 :green_heart: | unit | 0m 5s | hbase-shaded-jetty in the patch passed. | | +1 :green_heart: | unit | 0m 6s | hbase-shaded-jersey in the patch passed. | | -1 :x: | unit | 3m 45s | hbase-noop-htrace in the patch failed. | | +1 :green_heart: | unit | 0m 29s | root in the patch passed. | | +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. | | | | 13m 1s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-57/1/artifact/yetus-precommit-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase-thirdparty/pull/57 | | Optional Tests | dupname asflicense javac javadoc unit xml compile | | uname | Linux a60e78146de3 5.4.0-1054-aws #57~18.04.1-Ubuntu SMP Thu Jul 15 03:21:36 UTC 2021 x86_64 GNU/Linux | | Build tool | maven | | git revision | master / 247c926 | | Default Java | Oracle Corporation-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-57/1/artifact/yetus-precommit-check/output/patch-unit-hbase-noop-htrace.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-57/1/testReport/ | | Max. process+thread count | 395 (vs. ulimit of 1000) | | modules | C: hbase-shaded-protobuf hbase-shaded-netty hbase-shaded-gson hbase-shaded-miscellaneous hbase-shaded-jetty hbase-shaded-jersey hbase-noop-htrace . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-57/1/console | | versions | git=2.20.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-thirdparty] Apache9 opened a new pull request #57: HBASE-25609 [hbase-thirdparty] Bump version to 4.0.0-SNAPSHOT on mast…
Apache9 opened a new pull request #57: URL: https://github.com/apache/hbase-thirdparty/pull/57 …er branch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-26509) [hbase-thirdparty] Bump version to 4.0.0-SNAPSHOT on master branch
Duo Zhang created HBASE-26509: - Summary: [hbase-thirdparty] Bump version to 4.0.0-SNAPSHOT on master branch Key: HBASE-26509 URL: https://issues.apache.org/jira/browse/HBASE-26509 Project: HBase Issue Type: Sub-task Components: build, hbase-thirdparty, pom Reporter: Duo Zhang Assignee: Duo Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26496) [hbase-thirdparty] Exclude the original protobuf-java jar when shading
[
https://issues.apache.org/jira/browse/HBASE-26496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-26496:
--
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
Merged to master.
Thanks [~apurtell] for reviewing.
> [hbase-thirdparty] Exclude the original protobuf-java jar when shading
> --
>
> Key: HBASE-26496
> URL: https://issues.apache.org/jira/browse/HBASE-26496
> Project: HBase
> Issue Type: Bug
> Components: hbase-thirdparty
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: thirdparty-4.0.0, thirdparty-3.5.2
>
>
> We have a patched version of protobuf and we will compile it, and then in
> shade we also include the original protobuf-java, where we will get lot of
> warnings
> {noformat}
> WARNING] hbase-shaded-protobuf-3.5.1.jar, protobuf-java-3.17.1.jar define 670
> overlapping class and resource:
> [WARNING] - META-INF/MANIFEST.MF
> [WARNING] - com.google.protobuf.AbstractMessage
> [WARNING] - com.google.protobuf.AbstractMessage$Builder
> [WARNING] - com.google.protobuf.AbstractMessage$BuilderParent
> [WARNING] - com.google.protobuf.AbstractMessageLite
> [WARNING] - com.google.protobuf.AbstractMessageLite$Builder
> [WARNING] -
> com.google.protobuf.AbstractMessageLite$Builder$LimitedInputStream
> [WARNING] - com.google.protobuf.AbstractMessageLite$InternalOneOfEnum
> [WARNING] - com.google.protobuf.AbstractParser
> [WARNING] - com.google.protobuf.AbstractProtobufList
> [WARNING] - 660 more...
> [WARNING] maven-shade-plugin has detected that some class files are
> [WARNING] present in two or more JARs. When this happens, only one
> [WARNING] single version of the class is copied to the uber jar.
> [WARNING] Usually this is not harmful and you can skip these warnings,
> [WARNING] otherwise try to manually exclude artifacts based on
> [WARNING] mvn dependency:tree -Ddetail=true and the above output.
> [WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
> [WARNING] hbase-shaded-protobuf-3.5.1-sources.jar,
> protobuf-java-3.17.1-sources.jar define 202 overlapping resources:
> [WARNING] - META-INF/MANIFEST.MF
> [WARNING] - com/google/protobuf/AbstractMessage.java
> [WARNING] - com/google/protobuf/AbstractMessageLite.java
> [WARNING] - com/google/protobuf/AbstractParser.java
> [WARNING] - com/google/protobuf/AbstractProtobufList.java
> [WARNING] - com/google/protobuf/AllocatedBuffer.java
> [WARNING] - com/google/protobuf/Android.java
> [WARNING] - com/google/protobuf/Any.java
> [WARNING] - com/google/protobuf/AnyOrBuilder.java
> [WARNING] - com/google/protobuf/AnyProto.java
> [WARNING] - 192 more...
> [WARNING] maven-shade-plugin has detected that some class files are
> [WARNING] present in two or more JARs. When this happens, only one
> [WARNING] single version of the class is copied to the uber jar.
> [WARNING] Usually this is not harmful and you can skip these warnings,
> [WARNING] otherwise try to manually exclude artifacts based on
> [WARNING] mvn dependency:tree -Ddetail=true and the above output.
> [WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
> {noformat}
> Maybe this is the case why we bundle the incorrect protobuf classes and cause
> the failure when running with JDK8.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-26400) EOL 1.4.x
[ https://issues.apache.org/jira/browse/HBASE-26400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26400. --- Resolution: Fixed All sub tasks have been done. Resolve. > EOL 1.4.x > - > > Key: HBASE-26400 > URL: https://issues.apache.org/jira/browse/HBASE-26400 > Project: HBase > Issue Type: Task > Components: community >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase-thirdparty] Apache9 merged pull request #56: HBASE-26496 [hbase-thirdparty] Exclude the original protobuf-java jar…
Apache9 merged pull request #56: URL: https://github.com/apache/hbase-thirdparty/pull/56 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450151#comment-17450151
]
Hudson commented on HBASE-26067:
Results for branch HBASE-26067
[build #6 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/6/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/6/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/6/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/6/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26508) Update HEADER.html to mention the EOL of 1.4.x
[
https://issues.apache.org/jira/browse/HBASE-26508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450068#comment-17450068
]
Duo Zhang commented on HBASE-26508:
---
svn diff
{noformat}
Index: HEADER.html
===
--- HEADER.html (版本 51114)
+++ HEADER.html (工作副本)
@@ -8,8 +8,8 @@
The 2.4.z series is the current stable release line, it supercedes earlier
release lines
If you are a user of HBase 1.y and can not yet upgrade to the stable
release line, we suggest downloading the latest 1.y minor release.
Note that:
-0.96 was EOM'd September 2014; 0.94 and 0.98 were EOM'd April 2017
-1.0 was EOM'd January 2016; 1.1 was EOM'd December 2017; 1.2 was EOM'd
June 2019; 1.3 was EOM'd in August 2020
-2.0 was EOM'd September 2019; 2.1 was EOM'd May 2020; 2.2 was EOM'd April
2021; 2.3 was EOM'd Oct 2021
+0.96 was EOM'd September 2014; 0.94 and 0.98 were EOM'd April 2017i
+1.0 was EOM'd January 2016; 1.1 was EOM'd December 2017; 1.2 was EOM'd
June 2019; 1.3 was EOM'd in August 2020; 1.4 was EOM'd in October 2021
+2.0 was EOM'd September 2019; 2.1 was EOM'd May 2020; 2.2 was EOM'd April
2021; 2.3 was EOM'd Oct 2021
For older versions, check the https://archive.apache.org/dist/hbase/";>foundation archive of release
artifacts.
{noformat}
> Update HEADER.html to mention the EOL of 1.4.x
> --
>
> Key: HBASE-26508
> URL: https://issues.apache.org/jira/browse/HBASE-26508
> Project: HBase
> Issue Type: Sub-task
> Components: community
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-26508) Update HEADER.html to mention the EOL of 1.4.x
[ https://issues.apache.org/jira/browse/HBASE-26508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26508. --- Resolution: Fixed Done. > Update HEADER.html to mention the EOL of 1.4.x > -- > > Key: HBASE-26508 > URL: https://issues.apache.org/jira/browse/HBASE-26508 > Project: HBase > Issue Type: Sub-task > Components: community >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26508) Update HEADER.html to mention the EOL of 1.4.x
Duo Zhang created HBASE-26508: - Summary: Update HEADER.html to mention the EOL of 1.4.x Key: HBASE-26508 URL: https://issues.apache.org/jira/browse/HBASE-26508 Project: HBase Issue Type: Sub-task Components: community Reporter: Duo Zhang Assignee: Duo Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26507) Remove 1.4.14 from dist.a.o
[ https://issues.apache.org/jira/browse/HBASE-26507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26507. --- Resolution: Fixed Done. > Remove 1.4.14 from dist.a.o > --- > > Key: HBASE-26507 > URL: https://issues.apache.org/jira/browse/HBASE-26507 > Project: HBase > Issue Type: Sub-task > Components: community >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26507) Remove 1.4.14 from dist.a.o
Duo Zhang created HBASE-26507: - Summary: Remove 1.4.14 from dist.a.o Key: HBASE-26507 URL: https://issues.apache.org/jira/browse/HBASE-26507 Project: HBase Issue Type: Sub-task Components: community Reporter: Duo Zhang Assignee: Duo Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26323) Introduce a SnapshotProcedure
[ https://issues.apache.org/jira/browse/HBASE-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450061#comment-17450061 ] ruanhui commented on HBASE-26323: - ok. I will continue to try. Thank you. > Introduce a SnapshotProcedure > - > > Key: HBASE-26323 > URL: https://issues.apache.org/jira/browse/HBASE-26323 > Project: HBase > Issue Type: New Feature > Components: proc-v2, snapshots >Reporter: ruanhui >Assignee: ruanhui >Priority: Major > > Currently，snapshot in hbase uses zk as coordinator. It has some limitations, > a. Snapshot maybe fails when there are region server crashes. > b. Snapshot maybe failed when master restarts. > c. Only one snapshot per table can be taken in a time. > d. Snapshot verify will be handled by master, which may take long time when > our table has a large number of regions, for example 1. > > Since we have procedure v2 framework now, it is possible to solve the above > problems. So here is a procedure2-based snapshot implementation. It has some > goals, > a. Snapshot can continue when there are region server crashes. > b. Snapshot can continue when master restarts. > c. More than one snapshot per table can be taken in a time. > d. We can use region servers to verify snapshot to accelerate procedure. > > Here are some details about implementation. > *SnapshotProcedure* > SnapshotProcedure is used to take snapshot on a table. It acquires shared > table lock on the snapshot table and hold the shared lock during suspend and > yield. > *SnapshotRegionProcedure* > SnapshotRegionProcedure is used to take snapshot on a specific region of the > snapshot table. It acquires exclusive region lock and releases lock during > suspend and yield. Before dispatch remote snapshot operations to region > server, it will check target region in RIT or not. If target region is in > RIT, it will sleep some time and retry. > *SnapshotVerifyProcedure* > SnapshotVerifyProcedure is used to send snapshot verify request to region > server. If snapshot is corrupted, it will notify parent snapshot to retry. > When remote region server is crashed, it will choose another online server > and retry. > > I would be very grateful for any advice and guidance. Is anyone interested in > taking a look? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26323) Introduce a SnapshotProcedure
[ https://issues.apache.org/jira/browse/HBASE-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450060#comment-17450060 ] Duo Zhang commented on HBASE-26323: --- I could give it a first review, but still suggest that later we move to google doc so more people in the community can review and comment on it. People outside China are not likely to have an account for qq doc... Thanks. > Introduce a SnapshotProcedure > - > > Key: HBASE-26323 > URL: https://issues.apache.org/jira/browse/HBASE-26323 > Project: HBase > Issue Type: New Feature > Components: proc-v2, snapshots >Reporter: ruanhui >Assignee: ruanhui >Priority: Major > > Currently，snapshot in hbase uses zk as coordinator. It has some limitations, > a. Snapshot maybe fails when there are region server crashes. > b. Snapshot maybe failed when master restarts. > c. Only one snapshot per table can be taken in a time. > d. Snapshot verify will be handled by master, which may take long time when > our table has a large number of regions, for example 1. > > Since we have procedure v2 framework now, it is possible to solve the above > problems. So here is a procedure2-based snapshot implementation. It has some > goals, > a. Snapshot can continue when there are region server crashes. > b. Snapshot can continue when master restarts. > c. More than one snapshot per table can be taken in a time. > d. We can use region servers to verify snapshot to accelerate procedure. > > Here are some details about implementation. > *SnapshotProcedure* > SnapshotProcedure is used to take snapshot on a table. It acquires shared > table lock on the snapshot table and hold the shared lock during suspend and > yield. > *SnapshotRegionProcedure* > SnapshotRegionProcedure is used to take snapshot on a specific region of the > snapshot table. It acquires exclusive region lock and releases lock during > suspend and yield. Before dispatch remote snapshot operations to region > server, it will check target region in RIT or not. If target region is in > RIT, it will sleep some time and retry. > *SnapshotVerifyProcedure* > SnapshotVerifyProcedure is used to send snapshot verify request to region > server. If snapshot is corrupted, it will notify parent snapshot to retry. > When remote region server is crashed, it will choose another online server > and retry. > > I would be very grateful for any advice and guidance. Is anyone interested in > taking a look? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26403) Remove 1.4.x from our download page
[ https://issues.apache.org/jira/browse/HBASE-26403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26403. --- Fix Version/s: 3.0.0-alpha-2 Hadoop Flags: Reviewed Resolution: Fixed Merged to master. Thanks [~GeorryHuang] for reviewing. > Remove 1.4.x from our download page > --- > > Key: HBASE-26403 > URL: https://issues.apache.org/jira/browse/HBASE-26403 > Project: HBase > Issue Type: Sub-task > Components: website >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-2 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] Apache9 merged pull request #3881: HBASE-26403 Remove 1.4.x from our download page
Apache9 merged pull request #3881: URL: https://github.com/apache/hbase/pull/3881 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26496) [hbase-thirdparty] Exclude the original protobuf-java jar when shading
[
https://issues.apache.org/jira/browse/HBASE-26496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-26496:
--
Status: Patch Available (was: In Progress)
> [hbase-thirdparty] Exclude the original protobuf-java jar when shading
> --
>
> Key: HBASE-26496
> URL: https://issues.apache.org/jira/browse/HBASE-26496
> Project: HBase
> Issue Type: Bug
> Components: hbase-thirdparty
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: thirdparty-4.0.0, thirdparty-3.5.2
>
>
> We have a patched version of protobuf and we will compile it, and then in
> shade we also include the original protobuf-java, where we will get lot of
> warnings
> {noformat}
> WARNING] hbase-shaded-protobuf-3.5.1.jar, protobuf-java-3.17.1.jar define 670
> overlapping class and resource:
> [WARNING] - META-INF/MANIFEST.MF
> [WARNING] - com.google.protobuf.AbstractMessage
> [WARNING] - com.google.protobuf.AbstractMessage$Builder
> [WARNING] - com.google.protobuf.AbstractMessage$BuilderParent
> [WARNING] - com.google.protobuf.AbstractMessageLite
> [WARNING] - com.google.protobuf.AbstractMessageLite$Builder
> [WARNING] -
> com.google.protobuf.AbstractMessageLite$Builder$LimitedInputStream
> [WARNING] - com.google.protobuf.AbstractMessageLite$InternalOneOfEnum
> [WARNING] - com.google.protobuf.AbstractParser
> [WARNING] - com.google.protobuf.AbstractProtobufList
> [WARNING] - 660 more...
> [WARNING] maven-shade-plugin has detected that some class files are
> [WARNING] present in two or more JARs. When this happens, only one
> [WARNING] single version of the class is copied to the uber jar.
> [WARNING] Usually this is not harmful and you can skip these warnings,
> [WARNING] otherwise try to manually exclude artifacts based on
> [WARNING] mvn dependency:tree -Ddetail=true and the above output.
> [WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
> [WARNING] hbase-shaded-protobuf-3.5.1-sources.jar,
> protobuf-java-3.17.1-sources.jar define 202 overlapping resources:
> [WARNING] - META-INF/MANIFEST.MF
> [WARNING] - com/google/protobuf/AbstractMessage.java
> [WARNING] - com/google/protobuf/AbstractMessageLite.java
> [WARNING] - com/google/protobuf/AbstractParser.java
> [WARNING] - com/google/protobuf/AbstractProtobufList.java
> [WARNING] - com/google/protobuf/AllocatedBuffer.java
> [WARNING] - com/google/protobuf/Android.java
> [WARNING] - com/google/protobuf/Any.java
> [WARNING] - com/google/protobuf/AnyOrBuilder.java
> [WARNING] - com/google/protobuf/AnyProto.java
> [WARNING] - 192 more...
> [WARNING] maven-shade-plugin has detected that some class files are
> [WARNING] present in two or more JARs. When this happens, only one
> [WARNING] single version of the class is copied to the uber jar.
> [WARNING] Usually this is not harmful and you can skip these warnings,
> [WARNING] otherwise try to manually exclude artifacts based on
> [WARNING] mvn dependency:tree -Ddetail=true and the above output.
> [WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
> {noformat}
> Maybe this is the case why we bundle the incorrect protobuf classes and cause
> the failure when running with JDK8.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450058#comment-17450058
]
Hudson commented on HBASE-26067:
Results for branch HBASE-26067
[build #5 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/5/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/5/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/5/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/5/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26496) [hbase-thirdparty] Exclude the original protobuf-java jar when shading
[
https://issues.apache.org/jira/browse/HBASE-26496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450048#comment-17450048
]
Andrew Kyle Purtell commented on HBASE-26496:
-
Yes, maybe this is the reason.
> [hbase-thirdparty] Exclude the original protobuf-java jar when shading
> --
>
> Key: HBASE-26496
> URL: https://issues.apache.org/jira/browse/HBASE-26496
> Project: HBase
> Issue Type: Bug
> Components: hbase-thirdparty
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: thirdparty-4.0.0, thirdparty-3.5.2
>
>
> We have a patched version of protobuf and we will compile it, and then in
> shade we also include the original protobuf-java, where we will get lot of
> warnings
> {noformat}
> WARNING] hbase-shaded-protobuf-3.5.1.jar, protobuf-java-3.17.1.jar define 670
> overlapping class and resource:
> [WARNING] - META-INF/MANIFEST.MF
> [WARNING] - com.google.protobuf.AbstractMessage
> [WARNING] - com.google.protobuf.AbstractMessage$Builder
> [WARNING] - com.google.protobuf.AbstractMessage$BuilderParent
> [WARNING] - com.google.protobuf.AbstractMessageLite
> [WARNING] - com.google.protobuf.AbstractMessageLite$Builder
> [WARNING] -
> com.google.protobuf.AbstractMessageLite$Builder$LimitedInputStream
> [WARNING] - com.google.protobuf.AbstractMessageLite$InternalOneOfEnum
> [WARNING] - com.google.protobuf.AbstractParser
> [WARNING] - com.google.protobuf.AbstractProtobufList
> [WARNING] - 660 more...
> [WARNING] maven-shade-plugin has detected that some class files are
> [WARNING] present in two or more JARs. When this happens, only one
> [WARNING] single version of the class is copied to the uber jar.
> [WARNING] Usually this is not harmful and you can skip these warnings,
> [WARNING] otherwise try to manually exclude artifacts based on
> [WARNING] mvn dependency:tree -Ddetail=true and the above output.
> [WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
> [WARNING] hbase-shaded-protobuf-3.5.1-sources.jar,
> protobuf-java-3.17.1-sources.jar define 202 overlapping resources:
> [WARNING] - META-INF/MANIFEST.MF
> [WARNING] - com/google/protobuf/AbstractMessage.java
> [WARNING] - com/google/protobuf/AbstractMessageLite.java
> [WARNING] - com/google/protobuf/AbstractParser.java
> [WARNING] - com/google/protobuf/AbstractProtobufList.java
> [WARNING] - com/google/protobuf/AllocatedBuffer.java
> [WARNING] - com/google/protobuf/Android.java
> [WARNING] - com/google/protobuf/Any.java
> [WARNING] - com/google/protobuf/AnyOrBuilder.java
> [WARNING] - com/google/protobuf/AnyProto.java
> [WARNING] - 192 more...
> [WARNING] maven-shade-plugin has detected that some class files are
> [WARNING] present in two or more JARs. When this happens, only one
> [WARNING] single version of the class is copied to the uber jar.
> [WARNING] Usually this is not harmful and you can skip these warnings,
> [WARNING] otherwise try to manually exclude artifacts based on
> [WARNING] mvn dependency:tree -Ddetail=true and the above output.
> [WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
> {noformat}
> Maybe this is the case why we bundle the incorrect protobuf classes and cause
> the failure when running with JDK8.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-26455) TestStochasticLoadBalancerRegionReplicaWithRacks fails consistently
[ https://issues.apache.org/jira/browse/HBASE-26455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26455. --- Fix Version/s: 2.5.0 3.0.0-alpha-2 Hadoop Flags: Reviewed Resolution: Fixed Pushed to master and branch-2. Thanks [~Xiaolin Ha] and [~GeorryHuang] for reviewing. > TestStochasticLoadBalancerRegionReplicaWithRacks fails consistently > --- > > Key: HBASE-26455 > URL: https://issues.apache.org/jira/browse/HBASE-26455 > Project: HBase > Issue Type: Bug > Components: Balancer >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-2 > > > https://nightlies.apache.org/hbase/HBase/HBase-Find-Flaky-Tests/master/442/output/dashboard.html > [~claraxiong] Mind taking a look? I guess it is because some recent changes... > Thanks. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26455) TestStochasticLoadBalancerRegionReplicaWithRacks fails consistently
[ https://issues.apache.org/jira/browse/HBASE-26455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450046#comment-17450046 ] Duo Zhang commented on HBASE-26455: --- Seems fine. Let me cherry-pick to other branches and resolve. > TestStochasticLoadBalancerRegionReplicaWithRacks fails consistently > --- > > Key: HBASE-26455 > URL: https://issues.apache.org/jira/browse/HBASE-26455 > Project: HBase > Issue Type: Bug > Components: Balancer >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > https://nightlies.apache.org/hbase/HBase/HBase-Find-Flaky-Tests/master/442/output/dashboard.html > [~claraxiong] Mind taking a look? I guess it is because some recent changes... > Thanks. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase-thirdparty] Apache-HBase commented on pull request #56: HBASE-26496 [hbase-thirdparty] Exclude the original protobuf-java jar…
Apache-HBase commented on pull request #56: URL: https://github.com/apache/hbase-thirdparty/pull/56#issuecomment-981092017 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 41s | master passed | | +1 :green_heart: | compile | 0m 9s | master passed | | +1 :green_heart: | javadoc | 0m 7s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 11s | the patch passed | | +1 :green_heart: | compile | 0m 8s | the patch passed | | +1 :green_heart: | javac | 0m 8s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 0s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 5s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 9s | hbase-shaded-protobuf in the patch passed. | | +1 :green_heart: | asflicense | 0m 7s | The patch does not generate ASF License warnings. | | | | 3m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-56/1/artifact/yetus-precommit-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase-thirdparty/pull/56 | | Optional Tests | dupname asflicense javac javadoc unit xml compile | | uname | Linux 8827a3778286 5.4.0-1047-aws #49~18.04.1-Ubuntu SMP Wed Apr 28 23:08:58 UTC 2021 x86_64 GNU/Linux | | Build tool | maven | | git revision | master / c28a235 | | Default Java | Oracle Corporation-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-56/1/testReport/ | | Max. process+thread count | 39 (vs. ulimit of 1000) | | modules | C: hbase-shaded-protobuf U: hbase-shaded-protobuf | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-56/1/console | | versions | git=2.20.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-thirdparty] Apache9 commented on pull request #51: HBASE-25863 Shade javax.ws.rs package for use with shaded Jersey
Apache9 commented on pull request #51: URL: https://github.com/apache/hbase-thirdparty/pull/51#issuecomment-981091794 So any updates here? I plan to do a 4.0.0 release for hbase-thirdparty, so we can include this now? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-26498) [hbase-thirdparty] Make a 4.0.0 release for hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-26498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-26498: - Assignee: Duo Zhang > [hbase-thirdparty] Make a 4.0.0 release for hbase-thirdparty > > > Key: HBASE-26498 > URL: https://issues.apache.org/jira/browse/HBASE-26498 > Project: HBase > Issue Type: Umbrella > Components: community, hbase-thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase-thirdparty] Apache9 opened a new pull request #56: HBASE-26496 [hbase-thirdparty] Exclude the original protobuf-java jar…
Apache9 opened a new pull request #56: URL: https://github.com/apache/hbase-thirdparty/pull/56 … when shading -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26496) [hbase-thirdparty] Exclude the original protobuf-java jar when shading
[
https://issues.apache.org/jira/browse/HBASE-26496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-26496:
--
Fix Version/s: thirdparty-4.0.0
thirdparty-3.5.2
> [hbase-thirdparty] Exclude the original protobuf-java jar when shading
> --
>
> Key: HBASE-26496
> URL: https://issues.apache.org/jira/browse/HBASE-26496
> Project: HBase
> Issue Type: Bug
> Components: hbase-thirdparty
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: thirdparty-4.0.0, thirdparty-3.5.2
>
>
> We have a patched version of protobuf and we will compile it, and then in
> shade we also include the original protobuf-java, where we will get lot of
> warnings
> {noformat}
> WARNING] hbase-shaded-protobuf-3.5.1.jar, protobuf-java-3.17.1.jar define 670
> overlapping class and resource:
> [WARNING] - META-INF/MANIFEST.MF
> [WARNING] - com.google.protobuf.AbstractMessage
> [WARNING] - com.google.protobuf.AbstractMessage$Builder
> [WARNING] - com.google.protobuf.AbstractMessage$BuilderParent
> [WARNING] - com.google.protobuf.AbstractMessageLite
> [WARNING] - com.google.protobuf.AbstractMessageLite$Builder
> [WARNING] -
> com.google.protobuf.AbstractMessageLite$Builder$LimitedInputStream
> [WARNING] - com.google.protobuf.AbstractMessageLite$InternalOneOfEnum
> [WARNING] - com.google.protobuf.AbstractParser
> [WARNING] - com.google.protobuf.AbstractProtobufList
> [WARNING] - 660 more...
> [WARNING] maven-shade-plugin has detected that some class files are
> [WARNING] present in two or more JARs. When this happens, only one
> [WARNING] single version of the class is copied to the uber jar.
> [WARNING] Usually this is not harmful and you can skip these warnings,
> [WARNING] otherwise try to manually exclude artifacts based on
> [WARNING] mvn dependency:tree -Ddetail=true and the above output.
> [WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
> [WARNING] hbase-shaded-protobuf-3.5.1-sources.jar,
> protobuf-java-3.17.1-sources.jar define 202 overlapping resources:
> [WARNING] - META-INF/MANIFEST.MF
> [WARNING] - com/google/protobuf/AbstractMessage.java
> [WARNING] - com/google/protobuf/AbstractMessageLite.java
> [WARNING] - com/google/protobuf/AbstractParser.java
> [WARNING] - com/google/protobuf/AbstractProtobufList.java
> [WARNING] - com/google/protobuf/AllocatedBuffer.java
> [WARNING] - com/google/protobuf/Android.java
> [WARNING] - com/google/protobuf/Any.java
> [WARNING] - com/google/protobuf/AnyOrBuilder.java
> [WARNING] - com/google/protobuf/AnyProto.java
> [WARNING] - 192 more...
> [WARNING] maven-shade-plugin has detected that some class files are
> [WARNING] present in two or more JARs. When this happens, only one
> [WARNING] single version of the class is copied to the uber jar.
> [WARNING] Usually this is not harmful and you can skip these warnings,
> [WARNING] otherwise try to manually exclude artifacts based on
> [WARNING] mvn dependency:tree -Ddetail=true and the above output.
> [WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
> {noformat}
> Maybe this is the case why we bundle the incorrect protobuf classes and cause
> the failure when running with JDK8.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-25960) Build includes unshaded netty .so; clashes w/ downstreamers who would use a different version of netty
[
https://issues.apache.org/jira/browse/HBASE-25960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-25960:
--
Fix Version/s: (was: thirdparty-4.0.0)
(was: thirdparty-3.5.2)
> Build includes unshaded netty .so; clashes w/ downstreamers who would use a
> different version of netty
> --
>
> Key: HBASE-25960
> URL: https://issues.apache.org/jira/browse/HBASE-25960
> Project: HBase
> Issue Type: Bug
> Components: build
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
>
> A coworker was trying to use hbase client in a fat application that uses a
> different netty version to what hbase uses internally. Their app would fail
> to launch because it kept bumping into an incompatible netty .so lib. Here
> are the unshaded netty .so's we bundle looking at hbase-2.4.1...:
> ./lib/hbase-shaded-netty-3.4.1.jar has:
> {code}
> META-INF/native/libnetty_transport_native_epoll_aarch_64.so
> META-INF/native/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_64.so
> META-INF/native/libnetty_transport_native_epoll_x86_64.so
> {code}
> (HBASE-25959 should fix the non-relocation of
> libnetty_transport_native_epoll_aarch_64).
> ./lib/shaded-clients/hbase-shaded-client-byo-hadoop-2.4.1.1-apple.jar has the
> same three .sos as does
> ./lib/shaded-clients/hbase-shaded-mapreduce-2.4.1.1-apple.jar
> and ./lib/shaded-clients/hbase-shaded-client-2.4.1.1-apple.jar
> We even bundle ./lib/netty-all-4.1.17.Final.jar which unsurprisingly has the
> netty .sos in it.
> Looking at published builds of hbase-thirdparty, I see that these too include
> the above trio of .sos... The hbase-shaded-netty includes them in 3.4.1
> https://repo1.maven.org/maven2/org/apache/hbase/thirdparty/hbase-shaded-netty/3.4.1/
> as does 3.5.0.
> I just tried running a build of hbase-thirdparty and it does NOT include the
> extras
> META-INF/native/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_aarch_64.so
> META-INF/native/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_64.so
> (it has the fix for aarch included... when I built)
> Here is link to the snapshot I made:
> https://repository.apache.org/content/repositories/orgapachehbase-1451/org/apache/hbase/thirdparty/hbase-shaded-netty/3.5.1-stack4/
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26505) [hbase-thirdparty] Bump commons-cli version to 1.5.0
[ https://issues.apache.org/jira/browse/HBASE-26505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26505: -- Summary: [hbase-thirdparty] Bump commons-cli version to 1.5.0 (was: [hbase-thirdparty] Bump commons-cli to 1.5.0) > [hbase-thirdparty] Bump commons-cli version to 1.5.0 > > > Key: HBASE-26505 > URL: https://issues.apache.org/jira/browse/HBASE-26505 > Project: HBase > Issue Type: Sub-task > Components: hbase-thirdparty >Reporter: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26506) [hbase-thirdparty] Bump jersey version to 2.35
[ https://issues.apache.org/jira/browse/HBASE-26506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26506: -- Summary: [hbase-thirdparty] Bump jersey version to 2.35 (was: [hbase-thirdparty] Bump jersey to 2.35) > [hbase-thirdparty] Bump jersey version to 2.35 > -- > > Key: HBASE-26506 > URL: https://issues.apache.org/jira/browse/HBASE-26506 > Project: HBase > Issue Type: Sub-task > Components: hbase-thirdparty >Reporter: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26506) [hbase-thirdparty] Bump jersey to 2.35
Duo Zhang created HBASE-26506: - Summary: [hbase-thirdparty] Bump jersey to 2.35 Key: HBASE-26506 URL: https://issues.apache.org/jira/browse/HBASE-26506 Project: HBase Issue Type: Sub-task Components: hbase-thirdparty Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26505) [hbase-thirdparty] Bump commons-cli to 1.5.0
[ https://issues.apache.org/jira/browse/HBASE-26505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26505: -- Component/s: hbase-thirdparty > [hbase-thirdparty] Bump commons-cli to 1.5.0 > > > Key: HBASE-26505 > URL: https://issues.apache.org/jira/browse/HBASE-26505 > Project: HBase > Issue Type: Sub-task > Components: hbase-thirdparty >Reporter: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26505) [hbase-thirdparty] Bump commons-cli to 1.5.0
Duo Zhang created HBASE-26505: - Summary: [hbase-thirdparty] Bump commons-cli to 1.5.0 Key: HBASE-26505 URL: https://issues.apache.org/jira/browse/HBASE-26505 Project: HBase Issue Type: Sub-task Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26504) [hbase-thirdparty] Bump maven plugin versions
Duo Zhang created HBASE-26504: - Summary: [hbase-thirdparty] Bump maven plugin versions Key: HBASE-26504 URL: https://issues.apache.org/jira/browse/HBASE-26504 Project: HBase Issue Type: Sub-task Components: hbase-thirdparty Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26503) [hbase-thirdparty] Bump guava version to 31.0.1-jre
Duo Zhang created HBASE-26503: - Summary: [hbase-thirdparty] Bump guava version to 31.0.1-jre Key: HBASE-26503 URL: https://issues.apache.org/jira/browse/HBASE-26503 Project: HBase Issue Type: Sub-task Components: hbase-thirdparty Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26502) [hbase-thirdparty] Bump gson version to 2.8.9
Duo Zhang created HBASE-26502: - Summary: [hbase-thirdparty] Bump gson version to 2.8.9 Key: HBASE-26502 URL: https://issues.apache.org/jira/browse/HBASE-26502 Project: HBase Issue Type: Sub-task Components: hbase-thirdparty Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26501) [hbase-thirdparty] Bump jetty version to 9.4.44.v20210927
Duo Zhang created HBASE-26501: - Summary: [hbase-thirdparty] Bump jetty version to 9.4.44.v20210927 Key: HBASE-26501 URL: https://issues.apache.org/jira/browse/HBASE-26501 Project: HBase Issue Type: Sub-task Components: hbase-thirdparty Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26500) [hbase-thirdparty] Bump protobuf version to 3.19.1
Duo Zhang created HBASE-26500: - Summary: [hbase-thirdparty] Bump protobuf version to 3.19.1 Key: HBASE-26500 URL: https://issues.apache.org/jira/browse/HBASE-26500 Project: HBase Issue Type: Sub-task Components: hbase-thirdparty Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26498) [hbase-thirdparty] Make a 4.0.0 release for hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-26498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26498: -- Component/s: hbase-thirdparty > [hbase-thirdparty] Make a 4.0.0 release for hbase-thirdparty > > > Key: HBASE-26498 > URL: https://issues.apache.org/jira/browse/HBASE-26498 > Project: HBase > Issue Type: Umbrella > Components: community, hbase-thirdparty >Reporter: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26499) [hbase-thirdparty] Bump netty version to 4.1.70.Final
Duo Zhang created HBASE-26499: - Summary: [hbase-thirdparty] Bump netty version to 4.1.70.Final Key: HBASE-26499 URL: https://issues.apache.org/jira/browse/HBASE-26499 Project: HBase Issue Type: Sub-task Components: hbase-thirdparty Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26498) [hbase-thirdparty] Make a 4.0.0 release for hbase-thirdparty
Duo Zhang created HBASE-26498: - Summary: [hbase-thirdparty] Make a 4.0.0 release for hbase-thirdparty Key: HBASE-26498 URL: https://issues.apache.org/jira/browse/HBASE-26498 Project: HBase Issue Type: Umbrella Components: community Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HBASE-26496) [hbase-thirdparty] Exclude the original protobuf-java jar when shading
[
https://issues.apache.org/jira/browse/HBASE-26496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang reassigned HBASE-26496:
-
Assignee: Duo Zhang
> [hbase-thirdparty] Exclude the original protobuf-java jar when shading
> --
>
> Key: HBASE-26496
> URL: https://issues.apache.org/jira/browse/HBASE-26496
> Project: HBase
> Issue Type: Bug
> Components: hbase-thirdparty
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> We have a patched version of protobuf and we will compile it, and then in
> shade we also include the original protobuf-java, where we will get lot of
> warnings
> {noformat}
> WARNING] hbase-shaded-protobuf-3.5.1.jar, protobuf-java-3.17.1.jar define 670
> overlapping class and resource:
> [WARNING] - META-INF/MANIFEST.MF
> [WARNING] - com.google.protobuf.AbstractMessage
> [WARNING] - com.google.protobuf.AbstractMessage$Builder
> [WARNING] - com.google.protobuf.AbstractMessage$BuilderParent
> [WARNING] - com.google.protobuf.AbstractMessageLite
> [WARNING] - com.google.protobuf.AbstractMessageLite$Builder
> [WARNING] -
> com.google.protobuf.AbstractMessageLite$Builder$LimitedInputStream
> [WARNING] - com.google.protobuf.AbstractMessageLite$InternalOneOfEnum
> [WARNING] - com.google.protobuf.AbstractParser
> [WARNING] - com.google.protobuf.AbstractProtobufList
> [WARNING] - 660 more...
> [WARNING] maven-shade-plugin has detected that some class files are
> [WARNING] present in two or more JARs. When this happens, only one
> [WARNING] single version of the class is copied to the uber jar.
> [WARNING] Usually this is not harmful and you can skip these warnings,
> [WARNING] otherwise try to manually exclude artifacts based on
> [WARNING] mvn dependency:tree -Ddetail=true and the above output.
> [WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
> [WARNING] hbase-shaded-protobuf-3.5.1-sources.jar,
> protobuf-java-3.17.1-sources.jar define 202 overlapping resources:
> [WARNING] - META-INF/MANIFEST.MF
> [WARNING] - com/google/protobuf/AbstractMessage.java
> [WARNING] - com/google/protobuf/AbstractMessageLite.java
> [WARNING] - com/google/protobuf/AbstractParser.java
> [WARNING] - com/google/protobuf/AbstractProtobufList.java
> [WARNING] - com/google/protobuf/AllocatedBuffer.java
> [WARNING] - com/google/protobuf/Android.java
> [WARNING] - com/google/protobuf/Any.java
> [WARNING] - com/google/protobuf/AnyOrBuilder.java
> [WARNING] - com/google/protobuf/AnyProto.java
> [WARNING] - 192 more...
> [WARNING] maven-shade-plugin has detected that some class files are
> [WARNING] present in two or more JARs. When this happens, only one
> [WARNING] single version of the class is copied to the uber jar.
> [WARNING] Usually this is not harmful and you can skip these warnings,
> [WARNING] otherwise try to manually exclude artifacts based on
> [WARNING] mvn dependency:tree -Ddetail=true and the above output.
> [WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
> {noformat}
> Maybe this is the case why we bundle the incorrect protobuf classes and cause
> the failure when running with JDK8.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work started] (HBASE-26496) [hbase-thirdparty] Exclude the original protobuf-java jar when shading
[
https://issues.apache.org/jira/browse/HBASE-26496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-26496 started by Duo Zhang.
-
> [hbase-thirdparty] Exclude the original protobuf-java jar when shading
> --
>
> Key: HBASE-26496
> URL: https://issues.apache.org/jira/browse/HBASE-26496
> Project: HBase
> Issue Type: Bug
> Components: hbase-thirdparty
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> We have a patched version of protobuf and we will compile it, and then in
> shade we also include the original protobuf-java, where we will get lot of
> warnings
> {noformat}
> WARNING] hbase-shaded-protobuf-3.5.1.jar, protobuf-java-3.17.1.jar define 670
> overlapping class and resource:
> [WARNING] - META-INF/MANIFEST.MF
> [WARNING] - com.google.protobuf.AbstractMessage
> [WARNING] - com.google.protobuf.AbstractMessage$Builder
> [WARNING] - com.google.protobuf.AbstractMessage$BuilderParent
> [WARNING] - com.google.protobuf.AbstractMessageLite
> [WARNING] - com.google.protobuf.AbstractMessageLite$Builder
> [WARNING] -
> com.google.protobuf.AbstractMessageLite$Builder$LimitedInputStream
> [WARNING] - com.google.protobuf.AbstractMessageLite$InternalOneOfEnum
> [WARNING] - com.google.protobuf.AbstractParser
> [WARNING] - com.google.protobuf.AbstractProtobufList
> [WARNING] - 660 more...
> [WARNING] maven-shade-plugin has detected that some class files are
> [WARNING] present in two or more JARs. When this happens, only one
> [WARNING] single version of the class is copied to the uber jar.
> [WARNING] Usually this is not harmful and you can skip these warnings,
> [WARNING] otherwise try to manually exclude artifacts based on
> [WARNING] mvn dependency:tree -Ddetail=true and the above output.
> [WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
> [WARNING] hbase-shaded-protobuf-3.5.1-sources.jar,
> protobuf-java-3.17.1-sources.jar define 202 overlapping resources:
> [WARNING] - META-INF/MANIFEST.MF
> [WARNING] - com/google/protobuf/AbstractMessage.java
> [WARNING] - com/google/protobuf/AbstractMessageLite.java
> [WARNING] - com/google/protobuf/AbstractParser.java
> [WARNING] - com/google/protobuf/AbstractProtobufList.java
> [WARNING] - com/google/protobuf/AllocatedBuffer.java
> [WARNING] - com/google/protobuf/Android.java
> [WARNING] - com/google/protobuf/Any.java
> [WARNING] - com/google/protobuf/AnyOrBuilder.java
> [WARNING] - com/google/protobuf/AnyProto.java
> [WARNING] - 192 more...
> [WARNING] maven-shade-plugin has detected that some class files are
> [WARNING] present in two or more JARs. When this happens, only one
> [WARNING] single version of the class is copied to the uber jar.
> [WARNING] Usually this is not harmful and you can skip these warnings,
> [WARNING] otherwise try to manually exclude artifacts based on
> [WARNING] mvn dependency:tree -Ddetail=true and the above output.
> [WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
> {noformat}
> Maybe this is the case why we bundle the incorrect protobuf classes and cause
> the failure when running with JDK8.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26496) [hbase-thirdparty] Exclude the original protobuf-java jar when shading
[
https://issues.apache.org/jira/browse/HBASE-26496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450042#comment-17450042
]
Duo Zhang commented on HBASE-26496:
---
Not sure if this is the root cause so just link with related to, not broken by
> [hbase-thirdparty] Exclude the original protobuf-java jar when shading
> --
>
> Key: HBASE-26496
> URL: https://issues.apache.org/jira/browse/HBASE-26496
> Project: HBase
> Issue Type: Bug
> Components: hbase-thirdparty
>Reporter: Duo Zhang
>Priority: Major
>
> We have a patched version of protobuf and we will compile it, and then in
> shade we also include the original protobuf-java, where we will get lot of
> warnings
> {noformat}
> WARNING] hbase-shaded-protobuf-3.5.1.jar, protobuf-java-3.17.1.jar define 670
> overlapping class and resource:
> [WARNING] - META-INF/MANIFEST.MF
> [WARNING] - com.google.protobuf.AbstractMessage
> [WARNING] - com.google.protobuf.AbstractMessage$Builder
> [WARNING] - com.google.protobuf.AbstractMessage$BuilderParent
> [WARNING] - com.google.protobuf.AbstractMessageLite
> [WARNING] - com.google.protobuf.AbstractMessageLite$Builder
> [WARNING] -
> com.google.protobuf.AbstractMessageLite$Builder$LimitedInputStream
> [WARNING] - com.google.protobuf.AbstractMessageLite$InternalOneOfEnum
> [WARNING] - com.google.protobuf.AbstractParser
> [WARNING] - com.google.protobuf.AbstractProtobufList
> [WARNING] - 660 more...
> [WARNING] maven-shade-plugin has detected that some class files are
> [WARNING] present in two or more JARs. When this happens, only one
> [WARNING] single version of the class is copied to the uber jar.
> [WARNING] Usually this is not harmful and you can skip these warnings,
> [WARNING] otherwise try to manually exclude artifacts based on
> [WARNING] mvn dependency:tree -Ddetail=true and the above output.
> [WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
> [WARNING] hbase-shaded-protobuf-3.5.1-sources.jar,
> protobuf-java-3.17.1-sources.jar define 202 overlapping resources:
> [WARNING] - META-INF/MANIFEST.MF
> [WARNING] - com/google/protobuf/AbstractMessage.java
> [WARNING] - com/google/protobuf/AbstractMessageLite.java
> [WARNING] - com/google/protobuf/AbstractParser.java
> [WARNING] - com/google/protobuf/AbstractProtobufList.java
> [WARNING] - com/google/protobuf/AllocatedBuffer.java
> [WARNING] - com/google/protobuf/Android.java
> [WARNING] - com/google/protobuf/Any.java
> [WARNING] - com/google/protobuf/AnyOrBuilder.java
> [WARNING] - com/google/protobuf/AnyProto.java
> [WARNING] - 192 more...
> [WARNING] maven-shade-plugin has detected that some class files are
> [WARNING] present in two or more JARs. When this happens, only one
> [WARNING] single version of the class is copied to the uber jar.
> [WARNING] Usually this is not harmful and you can skip these warnings,
> [WARNING] otherwise try to manually exclude artifacts based on
> [WARNING] mvn dependency:tree -Ddetail=true and the above output.
> [WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
> {noformat}
> Maybe this is the case why we bundle the incorrect protobuf classes and cause
> the failure when running with JDK8.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26497) [hbase-thirdparty] Create a branch-3 in case we still need to support JDK7 for hbase 1.x
Duo Zhang created HBASE-26497: - Summary: [hbase-thirdparty] Create a branch-3 in case we still need to support JDK7 for hbase 1.x Key: HBASE-26497 URL: https://issues.apache.org/jira/browse/HBASE-26497 Project: HBase Issue Type: Bug Reporter: Duo Zhang And then bump the version for master to 4.x. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HBASE-26497) [hbase-thirdparty] Create a branch-3 in case we still need to support JDK7 for hbase 1.x
[ https://issues.apache.org/jira/browse/HBASE-26497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-26497: - Assignee: Duo Zhang > [hbase-thirdparty] Create a branch-3 in case we still need to support JDK7 > for hbase 1.x > > > Key: HBASE-26497 > URL: https://issues.apache.org/jira/browse/HBASE-26497 > Project: HBase > Issue Type: Bug > Components: hbase-thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > And then bump the version for master to 4.x. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26497) [hbase-thirdparty] Create a branch-3 in case we still need to support JDK7 for hbase 1.x
[ https://issues.apache.org/jira/browse/HBASE-26497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26497: -- Component/s: hbase-thirdparty > [hbase-thirdparty] Create a branch-3 in case we still need to support JDK7 > for hbase 1.x > > > Key: HBASE-26497 > URL: https://issues.apache.org/jira/browse/HBASE-26497 > Project: HBase > Issue Type: Bug > Components: hbase-thirdparty >Reporter: Duo Zhang >Priority: Major > > And then bump the version for master to 4.x. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26496) [hbase-thirdparty] Exclude the original protobuf-java jar when shading
Duo Zhang created HBASE-26496:
-
Summary: [hbase-thirdparty] Exclude the original protobuf-java jar
when shading
Key: HBASE-26496
URL: https://issues.apache.org/jira/browse/HBASE-26496
Project: HBase
Issue Type: Bug
Components: hbase-thirdparty
Reporter: Duo Zhang
We have a patched version of protobuf and we will compile it, and then in shade
we also include the original protobuf-java, where we will get lot of warnings
{noformat}
WARNING] hbase-shaded-protobuf-3.5.1.jar, protobuf-java-3.17.1.jar define 670
overlapping class and resource:
[WARNING] - META-INF/MANIFEST.MF
[WARNING] - com.google.protobuf.AbstractMessage
[WARNING] - com.google.protobuf.AbstractMessage$Builder
[WARNING] - com.google.protobuf.AbstractMessage$BuilderParent
[WARNING] - com.google.protobuf.AbstractMessageLite
[WARNING] - com.google.protobuf.AbstractMessageLite$Builder
[WARNING] - com.google.protobuf.AbstractMessageLite$Builder$LimitedInputStream
[WARNING] - com.google.protobuf.AbstractMessageLite$InternalOneOfEnum
[WARNING] - com.google.protobuf.AbstractParser
[WARNING] - com.google.protobuf.AbstractProtobufList
[WARNING] - 660 more...
[WARNING] maven-shade-plugin has detected that some class files are
[WARNING] present in two or more JARs. When this happens, only one
[WARNING] single version of the class is copied to the uber jar.
[WARNING] Usually this is not harmful and you can skip these warnings,
[WARNING] otherwise try to manually exclude artifacts based on
[WARNING] mvn dependency:tree -Ddetail=true and the above output.
[WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
[WARNING] hbase-shaded-protobuf-3.5.1-sources.jar,
protobuf-java-3.17.1-sources.jar define 202 overlapping resources:
[WARNING] - META-INF/MANIFEST.MF
[WARNING] - com/google/protobuf/AbstractMessage.java
[WARNING] - com/google/protobuf/AbstractMessageLite.java
[WARNING] - com/google/protobuf/AbstractParser.java
[WARNING] - com/google/protobuf/AbstractProtobufList.java
[WARNING] - com/google/protobuf/AllocatedBuffer.java
[WARNING] - com/google/protobuf/Android.java
[WARNING] - com/google/protobuf/Any.java
[WARNING] - com/google/protobuf/AnyOrBuilder.java
[WARNING] - com/google/protobuf/AnyProto.java
[WARNING] - 192 more...
[WARNING] maven-shade-plugin has detected that some class files are
[WARNING] present in two or more JARs. When this happens, only one
[WARNING] single version of the class is copied to the uber jar.
[WARNING] Usually this is not harmful and you can skip these warnings,
[WARNING] otherwise try to manually exclude artifacts based on
[WARNING] mvn dependency:tree -Ddetail=true and the above output.
[WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
{noformat}
Maybe this is the case why we bundle the incorrect protobuf classes and cause
the failure when running with JDK8.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26496) [hbase-thirdparty] Exclude the original protobuf-java jar when shading
[
https://issues.apache.org/jira/browse/HBASE-26496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450038#comment-17450038
]
Duo Zhang commented on HBASE-26496:
---
[~apurtell] FYI.
> [hbase-thirdparty] Exclude the original protobuf-java jar when shading
> --
>
> Key: HBASE-26496
> URL: https://issues.apache.org/jira/browse/HBASE-26496
> Project: HBase
> Issue Type: Bug
> Components: hbase-thirdparty
>Reporter: Duo Zhang
>Priority: Major
>
> We have a patched version of protobuf and we will compile it, and then in
> shade we also include the original protobuf-java, where we will get lot of
> warnings
> {noformat}
> WARNING] hbase-shaded-protobuf-3.5.1.jar, protobuf-java-3.17.1.jar define 670
> overlapping class and resource:
> [WARNING] - META-INF/MANIFEST.MF
> [WARNING] - com.google.protobuf.AbstractMessage
> [WARNING] - com.google.protobuf.AbstractMessage$Builder
> [WARNING] - com.google.protobuf.AbstractMessage$BuilderParent
> [WARNING] - com.google.protobuf.AbstractMessageLite
> [WARNING] - com.google.protobuf.AbstractMessageLite$Builder
> [WARNING] -
> com.google.protobuf.AbstractMessageLite$Builder$LimitedInputStream
> [WARNING] - com.google.protobuf.AbstractMessageLite$InternalOneOfEnum
> [WARNING] - com.google.protobuf.AbstractParser
> [WARNING] - com.google.protobuf.AbstractProtobufList
> [WARNING] - 660 more...
> [WARNING] maven-shade-plugin has detected that some class files are
> [WARNING] present in two or more JARs. When this happens, only one
> [WARNING] single version of the class is copied to the uber jar.
> [WARNING] Usually this is not harmful and you can skip these warnings,
> [WARNING] otherwise try to manually exclude artifacts based on
> [WARNING] mvn dependency:tree -Ddetail=true and the above output.
> [WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
> [WARNING] hbase-shaded-protobuf-3.5.1-sources.jar,
> protobuf-java-3.17.1-sources.jar define 202 overlapping resources:
> [WARNING] - META-INF/MANIFEST.MF
> [WARNING] - com/google/protobuf/AbstractMessage.java
> [WARNING] - com/google/protobuf/AbstractMessageLite.java
> [WARNING] - com/google/protobuf/AbstractParser.java
> [WARNING] - com/google/protobuf/AbstractProtobufList.java
> [WARNING] - com/google/protobuf/AllocatedBuffer.java
> [WARNING] - com/google/protobuf/Android.java
> [WARNING] - com/google/protobuf/Any.java
> [WARNING] - com/google/protobuf/AnyOrBuilder.java
> [WARNING] - com/google/protobuf/AnyProto.java
> [WARNING] - 192 more...
> [WARNING] maven-shade-plugin has detected that some class files are
> [WARNING] present in two or more JARs. When this happens, only one
> [WARNING] single version of the class is copied to the uber jar.
> [WARNING] Usually this is not harmful and you can skip these warnings,
> [WARNING] otherwise try to manually exclude artifacts based on
> [WARNING] mvn dependency:tree -Ddetail=true and the above output.
> [WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
> {noformat}
> Maybe this is the case why we bundle the incorrect protobuf classes and cause
> the failure when running with JDK8.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26492) [branch-2, hbase-thirdparty, Java 8] TestUnloadAccessController and other unit tests fail to start due to ByteBuffer link error
[
https://issues.apache.org/jira/browse/HBASE-26492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450031#comment-17450031
]
Duo Zhang commented on HBASE-26492:
---
Anyway, let's publish a new hbase thirdparty. I think this time we coud bump
the version to 4.0.0 and drop the support for JDK7 which is for branch-1.
> [branch-2, hbase-thirdparty, Java 8] TestUnloadAccessController and other
> unit tests fail to start due to ByteBuffer link error
> ---
>
> Key: HBASE-26492
> URL: https://issues.apache.org/jira/browse/HBASE-26492
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.4.8, 2.4.9
> Environment: Java 1.8.0_312, vendor: Azul Systems, Inc., runtime:
> /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre
> OS name: "mac os x", version: "12.0.1", arch: "aarch64", family: "mac"
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 2.5.0, 2.4.9
>
>
> org.apache.hadoop.hbase.security.access.TestUnloadAccessController
> Hang in setUpBeforeClass. Master will not initialize. Root cause is a
> NoSuchMethodError.
> {noformat}
> 2021-11-26 19:09:56,465 WARN
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=62950]
> ipc.RpcExecutor$Handler(370):
> Handler errors java.lang.NoSuchMethodError:
> java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
> at
> org.apache.hbase.thirdparty.com.google.protobuf.CodedOutputStream$HeapNioEncoder.flush(CodedOutputStream.java:1546)
> at
> org.apache.hadoop.hbase.ipc.ServerCall.writeToCOS(ServerCall.java:378)
> at
> org.apache.hadoop.hbase.ipc.ServerCall.createHeaderAndMessageBytes(ServerCall.java:385)
> at
> org.apache.hadoop.hbase.ipc.ServerCall.createHeaderAndMessageBytes(ServerCall.java:363)
> at
> org.apache.hadoop.hbase.ipc.ServerCall.setResponse(ServerCall.java:267)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:168)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:354)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:334)
> {noformat}
> This is a known issue with ByteBuffer in JDK 8 vs ByteBuffer in later
> versions. When code is compiled with Java 9 or later using a specific subset
> of ByteBuffer APIs, the resulting bytecode will not link with Java 8's
> runtime. It works fine the other way. When compiled with Java 8, the bytecode
> will link with later Java runtimes just fine.
> protobuf included into hbase-thirdparty was likely compiled with Java 9 or
> later. We shade that bytecode as is into hbase-thirdparty. Tests were
> attempted with Java 8, so this failure case manifested.
> Apache Maven 3.8.3 (ff8e977a158738155dc465c6a97ffaf31982d739)
> Java version: 1.8.0_312, vendor: Azul Systems, Inc., runtime:
> /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre
> OS name: "mac os x", version: "12.0.1", arch: "aarch64", family: "mac"
> We should be able to fix this problem by compiling protobuf with Java 8 and
> then shading the result when building hbase-thirdparty.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26482) HMaster may clean wals that is replicating in rare cases
[
https://issues.apache.org/jira/browse/HBASE-26482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450030#comment-17450030
]
Duo Zhang commented on HBASE-26482:
---
[~anoop.hbase] It is broken by HBASE-12770 again.
In the past, we will claim all the queues for a dead region server at once, and
we will delete the znode for the rs at last, so the cversion of the
replicationZNode will be changed.
After HBASE-12770, we will not always delete the rs znode because we will not
claim all the queues at once, so the cversion of the replicationZNode may not
be changed after a claim queue operation.
The patch here adds a dummy znode under the replicationZNode and then delete
it, which will force the cversion of replicationZNode to be changed, which can
fix the problem.
> HMaster may clean wals that is replicating in rare cases
>
>
> Key: HBASE-26482
> URL: https://issues.apache.org/jira/browse/HBASE-26482
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: zhuobin zheng
>Assignee: zhuobin zheng
>Priority: Critical
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.9
>
>
> In our cluster, i can found some FileNotFoundException when
> ReplicationSourceWALReader running for replication recovery queue.
> I guss the wal most likely removed by hmaste. And i found something to
> support it.
> The method getAllWALs:
> [https://github.com/apache/hbase/blob/master/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationQueueStorage.java#L509
>
> |https://github.com/apache/hbase/blob/master/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationQueueStorage.java#L509]Use
> zk cversion of /hbase/replication/rs as an optimistic lock to control
> concurrent ops.
> But, zk cversion *only can only reflect the changes of child nodes, but not
> the changes of grandchildren.*
> So, HMaster may loss some wal from this method in follow situation.
> # HMaster do log clean , and invoke getAllWALs to filter log which should
> not be deleted.
> # HMaster cache current cversion of /hbase/replication/rs as *v0*
> # HMaster cache all RS server name, and traverse them, get the WAL in each
> Queue
> # *RS2* dead after HMaster traverse {*}RS1{*}, and before traverse *RS2*
> # *RS1* claim one queue of *RS2,* which named *peerid-RS2* now
> # By the way , the cversion of /hbase/replication/rs not changed before all
> of *RS2* queue is removed, because the children of /hbase/replication/rs not
> change.
> # So, Hmaster will lost the wals in *peerid-RS2,* because we have already
> traversed *RS1 ,* and ** this queue not exists in *RS2*
> The above expression is currently only speculation, not confirmed
> Flie Not Found Log.
>
> {code:java}
> // code placeholder
> 2021-11-22 15:18:39,593 ERROR
> [ReplicationExecutor-0.replicationSource,peer_id-hostname,60020,1636802867348.replicationSource.wal-reader.hostname%2C60020%2C1636802867348,peer_id-hostname,60020,1636802867348]
> regionserver.WALEntryStream: Couldn't locate log:
> hdfs://namenode/hbase/oldWALs/hostname%2C60020%2C1636802867348.1636944748704
> 2021-11-22 15:18:39,593 ERROR
> [ReplicationExecutor-0.replicationSource,peer_id-hostname,60020,1636802867348.replicationSource.wal-reader.hostname%2C60020%2C1636802867348,peer_id-hostname,60020,1636802867348]
> regionserver.ReplicationSourceWALReader: Failed to read stream of
> replication entries
> java.io.FileNotFoundException: File does not exist:
> hdfs://namenode/hbase/oldWALs/hostname%2C60020%2C1636802867348.1636944748704
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1612)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1605)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1620)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:64)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:168)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:321)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:303)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:291)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:427)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.openReader(WALEntryStream.java:355)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.openNextLog(WALEntryStream.java:303)
>
[jira] [Assigned] (HBASE-26485) Introduce a method to clean restore directory after Snapshot Scan
[ https://issues.apache.org/jira/browse/HBASE-26485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liu zhuang reassigned HBASE-26485: -- Assignee: liu zhuang (was: ruanhui) > Introduce a method to clean restore directory after Snapshot Scan > - > > Key: HBASE-26485 > URL: https://issues.apache.org/jira/browse/HBASE-26485 > Project: HBase > Issue Type: Improvement > Components: snapshots >Reporter: ruanhui >Assignee: liu zhuang >Priority: Minor > > SnapshotScan is widely used in our company. However, after the snapshot scan > job, the restore directory is not cleaned, and this maybe puts a lot of > pressure on HDFS after a long time. So maybe we can introduce a method for > users to clean the snapshot restore directory after job. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26482) HMaster may clean wals that is replicating in rare cases
[
https://issues.apache.org/jira/browse/HBASE-26482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450014#comment-17450014
]
Anoop Sam John commented on HBASE-26482:
bq. JIRA: HBASE-12865 Has fixed this problem.
[~zhengzhuobinzzb] You mean to say that old jira fix was incomplete?
> HMaster may clean wals that is replicating in rare cases
>
>
> Key: HBASE-26482
> URL: https://issues.apache.org/jira/browse/HBASE-26482
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: zhuobin zheng
>Assignee: zhuobin zheng
>Priority: Critical
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.9
>
>
> In our cluster, i can found some FileNotFoundException when
> ReplicationSourceWALReader running for replication recovery queue.
> I guss the wal most likely removed by hmaste. And i found something to
> support it.
> The method getAllWALs:
> [https://github.com/apache/hbase/blob/master/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationQueueStorage.java#L509
>
> |https://github.com/apache/hbase/blob/master/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationQueueStorage.java#L509]Use
> zk cversion of /hbase/replication/rs as an optimistic lock to control
> concurrent ops.
> But, zk cversion *only can only reflect the changes of child nodes, but not
> the changes of grandchildren.*
> So, HMaster may loss some wal from this method in follow situation.
> # HMaster do log clean , and invoke getAllWALs to filter log which should
> not be deleted.
> # HMaster cache current cversion of /hbase/replication/rs as *v0*
> # HMaster cache all RS server name, and traverse them, get the WAL in each
> Queue
> # *RS2* dead after HMaster traverse {*}RS1{*}, and before traverse *RS2*
> # *RS1* claim one queue of *RS2,* which named *peerid-RS2* now
> # By the way , the cversion of /hbase/replication/rs not changed before all
> of *RS2* queue is removed, because the children of /hbase/replication/rs not
> change.
> # So, Hmaster will lost the wals in *peerid-RS2,* because we have already
> traversed *RS1 ,* and ** this queue not exists in *RS2*
> The above expression is currently only speculation, not confirmed
> Flie Not Found Log.
>
> {code:java}
> // code placeholder
> 2021-11-22 15:18:39,593 ERROR
> [ReplicationExecutor-0.replicationSource,peer_id-hostname,60020,1636802867348.replicationSource.wal-reader.hostname%2C60020%2C1636802867348,peer_id-hostname,60020,1636802867348]
> regionserver.WALEntryStream: Couldn't locate log:
> hdfs://namenode/hbase/oldWALs/hostname%2C60020%2C1636802867348.1636944748704
> 2021-11-22 15:18:39,593 ERROR
> [ReplicationExecutor-0.replicationSource,peer_id-hostname,60020,1636802867348.replicationSource.wal-reader.hostname%2C60020%2C1636802867348,peer_id-hostname,60020,1636802867348]
> regionserver.ReplicationSourceWALReader: Failed to read stream of
> replication entries
> java.io.FileNotFoundException: File does not exist:
> hdfs://namenode/hbase/oldWALs/hostname%2C60020%2C1636802867348.1636944748704
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1612)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1605)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1620)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:64)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:168)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:321)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:303)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:291)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:427)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.openReader(WALEntryStream.java:355)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.openNextLog(WALEntryStream.java:303)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.checkReader(WALEntryStream.java:294)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:175)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:101)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.ja
[GitHub] [hbase] frostruan commented on pull request #3716: HBASE-26323 introduce a SnapshotProcedure
frostruan commented on pull request #3716: URL: https://github.com/apache/hbase/pull/3716#issuecomment-981037400 hi @Apache9 here is a simple doc, would you mind taking a look ? https://docs.qq.com/doc/DS2NKVEpPY1dGR014 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26323) Introduce a SnapshotProcedure
[ https://issues.apache.org/jira/browse/HBASE-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17449969#comment-17449969 ] ruanhui commented on HBASE-26323: - hi [~zhangduo] I post a simple doc here. [https://docs.qq.com/doc/DS2NKVEpPY1dGR014] I have some problems with google docs, so I used a qq doc. sorry my english is not good, If my statement makes you confused, I am very happy to explain it in chinese. thanks. :D > Introduce a SnapshotProcedure > - > > Key: HBASE-26323 > URL: https://issues.apache.org/jira/browse/HBASE-26323 > Project: HBase > Issue Type: New Feature > Components: proc-v2, snapshots >Reporter: ruanhui >Assignee: ruanhui >Priority: Major > > Currently，snapshot in hbase uses zk as coordinator. It has some limitations, > a. Snapshot maybe fails when there are region server crashes. > b. Snapshot maybe failed when master restarts. > c. Only one snapshot per table can be taken in a time. > d. Snapshot verify will be handled by master, which may take long time when > our table has a large number of regions, for example 1. > > Since we have procedure v2 framework now, it is possible to solve the above > problems. So here is a procedure2-based snapshot implementation. It has some > goals, > a. Snapshot can continue when there are region server crashes. > b. Snapshot can continue when master restarts. > c. More than one snapshot per table can be taken in a time. > d. We can use region servers to verify snapshot to accelerate procedure. > > Here are some details about implementation. > *SnapshotProcedure* > SnapshotProcedure is used to take snapshot on a table. It acquires shared > table lock on the snapshot table and hold the shared lock during suspend and > yield. > *SnapshotRegionProcedure* > SnapshotRegionProcedure is used to take snapshot on a specific region of the > snapshot table. It acquires exclusive region lock and releases lock during > suspend and yield. Before dispatch remote snapshot operations to region > server, it will check target region in RIT or not. If target region is in > RIT, it will sleep some time and retry. > *SnapshotVerifyProcedure* > SnapshotVerifyProcedure is used to send snapshot verify request to region > server. If snapshot is corrupted, it will notify parent snapshot to retry. > When remote region server is crashed, it will choose another online server > and retry. > > I would be very grateful for any advice and guidance. Is anyone interested in > taking a look? -- This message was sent by Atlassian Jira (v8.20.1#820001)
