ChenSammi commented on code in PR #8845:
URL: https://github.com/apache/ozone/pull/8845#discussion_r2238131637
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/DeletedBlockLogStateManagerImpl.java:
##########
@@ -154,21 +187,95 @@ public void
addTransactionsToDB(ArrayList<DeletedBlocksTransaction> txs)
containerIdToTxnIdMap.compute(ContainerID.valueOf(tx.getContainerID()),
(k, v) -> v != null && v > tid ? v : tid);
transactionBuffer.addToBuffer(deletedTable, tx.getTxID(), tx);
+ if
(VersionedDatanodeFeatures.isFinalized(HDDSLayoutFeature.DATA_DISTRIBUTION) &&
+ tx.hasTotalBlockReplicatedSize()) {
+ if (!isFirstTxIdForDataDistributionSet) {
+ // set the first transaction ID for data distribution
+ isFirstTxIdForDataDistributionSet = true;
+ firstTxIdForDataDistribution = tx.getTxID();
+ }
+ transactionBuffer.addToBuffer(statefulConfigTable, SERVICE_NAME,
+ incrDeletedBlocksSummary(tx).toByteString());
+ }
}
+
containerManager.updateDeleteTransactionId(containerIdToTxnIdMap);
}
+ private DeletedBlocksTransactionSummary
incrDeletedBlocksSummary(DeletedBlocksTransaction tx) {
+ totalTxCount.addAndGet(1);
+ totalBlockCount.addAndGet(tx.getLocalIDCount());
+ totalBlocksSize.addAndGet(tx.getTotalBlockSize());
+ totalReplicatedBlocksSize.addAndGet(tx.getTotalBlockReplicatedSize());
+ return DeletedBlocksTransactionSummary.newBuilder()
+ .setFirstTxID(firstTxIdForDataDistribution)
+ .setTotalTransactionCount(totalTxCount.get())
+ .setTotalBlockCount(totalBlockCount.get())
+ .setTotalBlockSize(totalBlocksSize.get())
+ .setTotalBlockReplicatedSize(totalReplicatedBlocksSize.get())
+ .build();
+ }
+
@Override
public void removeTransactionsFromDB(ArrayList<Long> txIDs)
throws IOException {
if (deletingTxIDs != null) {
deletingTxIDs.addAll(txIDs);
}
+
for (Long txID : txIDs) {
transactionBuffer.removeFromBuffer(deletedTable, txID);
+ if
(VersionedDatanodeFeatures.isFinalized(HDDSLayoutFeature.DATA_DISTRIBUTION) &&
+ txID >= firstTxIdForDataDistribution) {
+ DeletedBlockLogImpl.TxBlockInfo txBlockInfo =
txBlockInfoMap.remove(txID);
+ if (txBlockInfo != null) {
+ transactionBuffer.addToBuffer(statefulConfigTable, SERVICE_NAME,
+ descDeletedBlocksSummary(txBlockInfo).toByteString());
+ metrics.incrBlockDeletionTransactionSizeFromCache();
+ } else {
+ // Fetch the transaction from DB to get the size. This happens during
+ // 1. SCM leader transfer, deletion command send by one SCM,
+ // while the deletion ack received by a different SCM
+ // 2. SCM restarts, txBlockInfoMap is empty, while receiving the
deletion ack from DN
+ DeletedBlocksTransaction tx = deletedTable.get(txID);
+ if (tx != null && tx.hasTotalBlockReplicatedSize()) {
+ transactionBuffer.addToBuffer(statefulConfigTable, SERVICE_NAME,
+ descDeletedBlocksSummary(tx).toByteString());
+ }
+ metrics.incrBlockDeletionTransactionSizeFromDB();
Review Comment:
@sumitagrawl , thanks for the review.
a. this metrics doesn't persist to DB, metrics is local to each SCM.
b. summary is written to DB through RATIS only by leader SCM.
c. when there is leader switch over, the in memory map content of
txBlockInfoMap will be lost, but the DB of new leader remains the same as old
leader. You mentioned that this response will be just ignored. I'm not sure how
these response will be ignored, is it because the response carrying the old SCM
ratis term, so that the response is ignored? Anyway, if the response is
ignored, it's not ignored here in
DeletedBlockLogStateManagerImpl#removeTransactionsFromDB, must be somewhere
before DeletedBlockLogStateManagerImpl#removeTransactionsFromDB. So I believe
the metrics and summary here is accurate in either cases. Please let me know
if I understand the response ignore you mentioned incorrectly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]