ivandika3 commented on code in PR #3229:
URL: https://github.com/apache/ozone/pull/3229#discussion_r1900648593
##########
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java:
##########
@@ -406,12 +410,26 @@ private void executePutBlock(boolean close,
byteBufferList = null;
}
waitFuturesComplete();
+ final BlockData blockData = containerBlockData.build();
if (close) {
- dataStreamCloseReply = out.closeAsync();
+ final ContainerCommandRequestProto putBlockRequest
+ = ContainerProtocolCalls.getPutBlockRequest(
+ xceiverClient.getPipeline(), blockData, true, token);
+ dataStreamCloseReply = executePutBlockClose(putBlockRequest,
+ PUT_BLOCK_REQUEST_LENGTH_MAX, out);
+ dataStreamCloseReply.whenComplete((reply, e) -> {
+ if (e != null || reply == null || !reply.isSuccess()) {
+ LOG.warn("Failed executePutBlockClose, reply=" + reply, e);
+ try {
+ executePutBlock(true, false);
+ } catch (IOException ex) {
+ throw new CompletionException(ex);
+ }
+ }
+ });
Review Comment:
@szetszwo May I clarify why during the `BlockDataStreamOutput` close, there
seems to be two PutBlock requests sent?
1. `executePutBlockClose` using the `DataStreamOutput#writeAsync` with
`StandardWriteOption.CLOSE` which will trigger `sendForward` Async API to the
leader and will be processed by the `StateMachine#link` as a `PutBlock` request.
2. `putBlockAsync` which will send another `PutBlock` request using the
Async API. This will update the `BlockDataStreamOutput` `BlockID` which will be
used when committing the key to OM.
I'm wondering whether there might be some ordering issues that if the
`putBlockAsync` is executed first and the `executePutBlockClose` executed
second (which does not seem to update the BlockID), the committed bcsId in OM
will be different from the ones in DN? If that's so, maybe we can only run
`executePutBlockClose` during close?
##########
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java:
##########
@@ -406,12 +410,26 @@ private void executePutBlock(boolean close,
byteBufferList = null;
}
waitFuturesComplete();
+ final BlockData blockData = containerBlockData.build();
if (close) {
- dataStreamCloseReply = out.closeAsync();
+ final ContainerCommandRequestProto putBlockRequest
+ = ContainerProtocolCalls.getPutBlockRequest(
+ xceiverClient.getPipeline(), blockData, true, token);
+ dataStreamCloseReply = executePutBlockClose(putBlockRequest,
+ PUT_BLOCK_REQUEST_LENGTH_MAX, out);
+ dataStreamCloseReply.whenComplete((reply, e) -> {
+ if (e != null || reply == null || !reply.isSuccess()) {
+ LOG.warn("Failed executePutBlockClose, reply=" + reply, e);
+ try {
+ executePutBlock(true, false);
+ } catch (IOException ex) {
+ throw new CompletionException(ex);
+ }
+ }
+ });
Review Comment:
@szetszwo May I clarify why during the `BlockDataStreamOutput` close, there
seems to be two PutBlock requests sent?
1. `executePutBlockClose` using the `DataStreamOutput#writeAsync` with
`StandardWriteOption.CLOSE` which will trigger `sendForward` Async API to the
leader and will be processed by the `StateMachine#link` as a `PutBlock` request.
2. `putBlockAsync` which will send another `PutBlock` request using the
Async API. This will update the `BlockDataStreamOutput` `BlockID` which will be
used when committing the key to OM.
I'm wondering whether there might be some ordering issues that if the
`putBlockAsync` is executed first and the `executePutBlockClose` executed
second (which does not seem to update the BlockID), the committed bcsId in OM
will be different from the ones in DN? If that's so, maybe we can only run
`executePutBlockClose` during close?
Thanks in advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
