amaliujia commented on code in PR #10823:
URL: https://github.com/apache/ozone/pull/10823#discussion_r3670545042
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/BlockManagerImpl.java:
##########
@@ -209,6 +209,20 @@ public long persistPutBlock(KeyValueContainer container,
// container to determine whether the blockCount is already incremented
// for this block in the DB or not.
long localID = data.getLocalID();
+ // For the PutBlock that is endOfBlock and meanwhile bscId = 0, it means
+ // this PutBlock comes from data stream close without going through the
+ // Ratis, thus there is no log index. In this case, we should not let
+ // 0 to overwrite previous possible PutBlocks from Ratis log that were
+ // generated during immediate flushes from an active data stream.
Instead,
+ // we should load the latest bscid and reuse that id.
+ if (endOfBlock && bcsId == 0) {
+ BlockData existing = db.getStore().getBlockDataTable()
+ .get(containerData.getBlockKey(localID));
+ if (existing != null) {
+ bcsId = existing.getBlockCommitSequenceId();
Review Comment:
Actually it pretty much won't be 0.
If there is any existing bscId, it should come form flushes from a
BlockDataStreamOutput:
https://github.com/apache/ozone/blob/7d9223cd5a94a7073f81ac54445c5cb4053df8ec/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java#L314.
When there is a middle flush, there will be a PutBlock triggered. That
PutBlock is not migrated out of the Raft yet. So that PutBlock carries a log
index which is non zero and persisted into DN.
this test case is the relevant one:
`TestBlockDataStreamOutput.testPutBlockAtBoundary`. If there is a boundary
reached, there is a flush and PutBlock.
The fix here is short term, the longer term fix needs Ratis data stream
supporting a control plane API (created
https://issues.apache.org/jira/browse/RATIS-2625) so no long the caller can
send raw bytes, but also metadata commits through API. So the metadata commit
can also happen before steam close, for example flushes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
