ashishkumar50 commented on code in PR #8845:
URL: https://github.com/apache/ozone/pull/8845#discussion_r2224308715
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/DeletedBlockLogImpl.java:
##########
@@ -229,13 +246,23 @@ private int resetRetryCount(List<Long> txIDs) throws
IOException {
}
private DeletedBlocksTransaction constructNewTransaction(
- long txID, long containerID, List<Long> blocks) {
- return DeletedBlocksTransaction.newBuilder()
+ long txID, long containerID, List<DeletedBlock> blocks) {
+ List<Long> localIdList = blocks.stream().map(b ->
b.getBlockID().getLocalID()).collect(Collectors.toList());
+ DeletedBlocksTransaction.Builder builder =
DeletedBlocksTransaction.newBuilder()
.setTxID(txID)
.setContainerID(containerID)
- .addAllLocalID(blocks)
- .setCount(0)
- .build();
+ .addAllLocalID(localIdList)
+ .setCount(0);
+
+ if
(VersionedDatanodeFeatures.isFinalized(HDDSLayoutFeature.DATA_DISTRIBUTION)) {
+ long replicatedSize =
blocks.stream().mapToLong(DeletedBlock::getReplicatedSize).sum();
+ // even when HDDSLayoutFeature.DATA_DISTRIBUTION is finalized, old OM
can still call the old API
+ if (replicatedSize >= 0) {
Review Comment:
`replicatedSize` can be either 0 or greater than 0, this check is not
useful.
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/ScmBlockDeletingServiceMetrics.java:
##########
@@ -100,15 +102,38 @@ public final class ScmBlockDeletingServiceMetrics
implements MetricsSource {
private final Map<DatanodeID, DatanodeCommandDetails> numCommandsDatanode =
new ConcurrentHashMap<>();
- private ScmBlockDeletingServiceMetrics() {
+ @Metric(about = "The number of transactions whose totalBlockSize is fetched
from in memory cache")
+ private MutableGaugeLong numBlockDeletionTransactionSizeFromCache;
+
+ @Metric(about = "The number of transactions whose totalBlockSize is fetched
from DB")
+ private MutableGaugeLong numBlockDeletionTransactionSizeFromDB;
+
+ private static final MetricsInfo NUM_BLOCK_DELETION_TRANSACTION =
Interns.info(
+ "numBlockDeletionTransaction",
+ "The number of transactions in DB.");
+
+ private static final MetricsInfo NUM_BLOCK_IN_DELETION_TRANSACTION =
Interns.info(
+ "numBlockOfDeletionTransaction",
+ "The number of blocks in all transactions in DB.");
Review Comment:
Better name as `NUM_BLOCK_IN_ALL_DELETION_TRANSACTION`
`numBlockInAllDeletionTransaction`
##########
hadoop-hdds/interface-client/src/main/proto/hdds.proto:
##########
@@ -513,6 +513,14 @@ message DeletedBlocksTransactionInfo {
optional int32 count = 4;
}
+message DeletedBlocksTransactionSummary {
+ optional int64 firstTxID = 1;
Review Comment:
What is the use of firstTxID in the summary result?
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/DeletedBlockLogStateManagerImpl.java:
##########
@@ -247,21 +354,72 @@ public void reinitialize(
// before reinitialization. Just update deletedTable here.
Preconditions.checkArgument(deletingTxIDs.isEmpty());
this.deletedTable = deletedBlocksTXTable;
+ this.statefulConfigTable = configTable;
}
public static Builder newBuilder() {
return new Builder();
}
+ @Override
+ public DeletedBlocksTransactionSummary getTransactionSummary() {
+ return DeletedBlocksTransactionSummary.newBuilder()
+ .setFirstTxID(firstTxIdForDataDistribution)
+ .setTotalTransactionCount(totalTxCount.get())
+ .setTotalBlockCount(totalBlockCount.get())
+ .setTotalBlockSize(totalBlocksSize.get())
+ .setTotalBlockReplicatedSize(totalReplicatedBlocksSize.get())
+ .build();
+ }
+
+ private void initDataDistributionData() throws IOException {
Review Comment:
How much time `initDataDistributionData` is taking in case of million
transactions?
##########
hadoop-hdds/interface-server/src/main/proto/ScmServerDatanodeHeartbeatProtocol.proto:
##########
@@ -368,6 +368,8 @@ message DeletedBlocksTransaction {
repeated int64 localID = 3;
// the retry time of sending deleting command to datanode.
required int32 count = 4;
+ optional uint64 totalBlockSize = 5;
+ optional uint64 totalBlockReplicatedSize = 6;
Review Comment:
Do we need to store `totalBlockReplicatedSize` here as well along with
`totalBlockSize?` as this can be derived using `totalBlockSize` and replication
config from container info.
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/DeletedBlockLogImpl.java:
##########
@@ -229,13 +246,23 @@ private int resetRetryCount(List<Long> txIDs) throws
IOException {
}
private DeletedBlocksTransaction constructNewTransaction(
- long txID, long containerID, List<Long> blocks) {
- return DeletedBlocksTransaction.newBuilder()
+ long txID, long containerID, List<DeletedBlock> blocks) {
+ List<Long> localIdList = blocks.stream().map(b ->
b.getBlockID().getLocalID()).collect(Collectors.toList());
+ DeletedBlocksTransaction.Builder builder =
DeletedBlocksTransaction.newBuilder()
.setTxID(txID)
.setContainerID(containerID)
- .addAllLocalID(blocks)
- .setCount(0)
- .build();
+ .addAllLocalID(localIdList)
+ .setCount(0);
+
+ if
(VersionedDatanodeFeatures.isFinalized(HDDSLayoutFeature.DATA_DISTRIBUTION)) {
Review Comment:
Also ` && txn.hasTotalBlockReplicatedSize()` can be added here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]