ChenSammi commented on code in PR #8845:
URL: https://github.com/apache/ozone/pull/8845#discussion_r2238131637


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/DeletedBlockLogStateManagerImpl.java:
##########
@@ -154,21 +187,95 @@ public void 
addTransactionsToDB(ArrayList<DeletedBlocksTransaction> txs)
       containerIdToTxnIdMap.compute(ContainerID.valueOf(tx.getContainerID()),
           (k, v) -> v != null && v > tid ? v : tid);
       transactionBuffer.addToBuffer(deletedTable, tx.getTxID(), tx);
+      if 
(VersionedDatanodeFeatures.isFinalized(HDDSLayoutFeature.DATA_DISTRIBUTION) &&
+          tx.hasTotalBlockReplicatedSize()) {
+        if (!isFirstTxIdForDataDistributionSet) {
+          // set the first transaction ID for data distribution
+          isFirstTxIdForDataDistributionSet = true;
+          firstTxIdForDataDistribution = tx.getTxID();
+        }
+        transactionBuffer.addToBuffer(statefulConfigTable, SERVICE_NAME,
+            incrDeletedBlocksSummary(tx).toByteString());
+      }
     }
+
     containerManager.updateDeleteTransactionId(containerIdToTxnIdMap);
   }
 
+  private DeletedBlocksTransactionSummary 
incrDeletedBlocksSummary(DeletedBlocksTransaction tx) {
+    totalTxCount.addAndGet(1);
+    totalBlockCount.addAndGet(tx.getLocalIDCount());
+    totalBlocksSize.addAndGet(tx.getTotalBlockSize());
+    totalReplicatedBlocksSize.addAndGet(tx.getTotalBlockReplicatedSize());
+    return DeletedBlocksTransactionSummary.newBuilder()
+        .setFirstTxID(firstTxIdForDataDistribution)
+        .setTotalTransactionCount(totalTxCount.get())
+        .setTotalBlockCount(totalBlockCount.get())
+        .setTotalBlockSize(totalBlocksSize.get())
+        .setTotalBlockReplicatedSize(totalReplicatedBlocksSize.get())
+        .build();
+  }
+
   @Override
   public void removeTransactionsFromDB(ArrayList<Long> txIDs)
       throws IOException {
     if (deletingTxIDs != null) {
       deletingTxIDs.addAll(txIDs);
     }
+
     for (Long txID : txIDs) {
       transactionBuffer.removeFromBuffer(deletedTable, txID);
+      if 
(VersionedDatanodeFeatures.isFinalized(HDDSLayoutFeature.DATA_DISTRIBUTION) &&
+          txID >= firstTxIdForDataDistribution) {
+        DeletedBlockLogImpl.TxBlockInfo txBlockInfo = 
txBlockInfoMap.remove(txID);
+        if (txBlockInfo != null) {
+          transactionBuffer.addToBuffer(statefulConfigTable, SERVICE_NAME,
+              descDeletedBlocksSummary(txBlockInfo).toByteString());
+          metrics.incrBlockDeletionTransactionSizeFromCache();
+        } else {
+          // Fetch the transaction from DB to get the size. This happens during
+          // 1. SCM leader transfer, deletion command send by one SCM,
+          //    while the deletion ack received by a different SCM
+          // 2. SCM restarts, txBlockInfoMap is empty, while receiving the 
deletion ack from DN
+          DeletedBlocksTransaction tx = deletedTable.get(txID);
+          if (tx != null && tx.hasTotalBlockReplicatedSize()) {
+            transactionBuffer.addToBuffer(statefulConfigTable, SERVICE_NAME,
+                descDeletedBlocksSummary(tx).toByteString());
+          }
+          metrics.incrBlockDeletionTransactionSizeFromDB();

Review Comment:
   @sumitagrawl , thanks for the review.  
   
   a.  this metrics doesn't persist to DB, metrics is local to each SCM. 
   b.  summary is written to DB through RATIS only by leader SCM.
   c.  when there is leader switch over, the in memory map content of 
txBlockInfoMap will be lost, but the DB of new leader remains the same as old 
leader. You mentioned that this response will be just ignored. I'm not sure how 
these response will be ignored, is it because the response carrying the old SCM 
ratis term, so that the response is ignored?  Anyway, if the response is 
ignored, it's not ignored here in 
DeletedBlockLogStateManagerImpl#removeTransactionsFromDB, must be somewhere 
before DeletedBlockLogStateManagerImpl#removeTransactionsFromDB. So I believe 
the metrics and summary here is accurate in either cases.  Please let me know 
if I understand the response ignore you mentioned incorrectly. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to