errose28 commented on code in PR #10084:
URL: https://github.com/apache/ozone/pull/10084#discussion_r3119223448
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/upgrade/SCMUpgradeFinalizer.java:
##########
@@ -177,4 +146,76 @@ private void
waitForDatanodesToFinalize(SCMUpgradeFinalizationContext context)
}
}
}
+
+ public static DatanodeFinalizationCounts getNumFinalizedDatanodes(
Review Comment:
This should be an instance method in `NodeManager`, not a static method that
takes an instance as its first/only argument. `DatanodeFinalizationCounts` can
be moved there as well, or moved to a standalone class.
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java:
##########
@@ -1153,13 +1155,16 @@ public HddsProtos.UpgradeStatus queryUpgradeStatus()
throws IOException {
try {
getScm().checkAdminAccess(getRemoteUser(), true);
- // Returning a placeholder for now.
- HddsProtos.UpgradeStatus result = HddsProtos.UpgradeStatus.newBuilder()
- .setScmFinalized(true)
- .setNumDatanodesFinalized(10)
- .setNumDatanodesTotal(10)
- .setShouldFinalize(true)
- .build();
+ UpgradeFinalization.Status scmUpgradeStatus =
scm.getLayoutVersionManager().getUpgradeState();
Review Comment:
This upgrade status enum is going to be removed in the new version manager.
We will only use the proto facing version for compatibility with the older
APIs. This call can be changed to
`scm.getLayoutVersionManager().needsFinalization()` which easily maps to the
same call in the [new version
manager](https://github.com/apache/ozone/blob/50f407e2f61ad0d9ca959fe4bd9d8911ea61f330/hadoop-hdds/framework/src/main/java/org/apache/hadoop/ozone/upgrade/ComponentVersionManager.java#L87)
when we switch. From here we can:
- Remove the `isScmFinalized` helper method
- Add a `DatanodeFinalizationCounts#allNodesFinalized` method as suggested
[here](https://github.com/apache/ozone/pull/10084/changes#r3119214490)
- Inline the `shouldFinalize` method to just a variable: `shouldFinalize =
scmFinalized && datanodeFinalizationCounts.allNodesFinalized()`
- Inline `buildUpgradeStatus` to invoke the builder directly in this method
since the labelled builder setters are clearer than the unlabeled method
parameters and it is now just a passthrough.
This minimizes the amount of upgrade specific logic in the server side
translator.
##########
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/server/TestSCMClientProtocolServer.java:
##########
@@ -138,6 +140,80 @@ public void testScmListContainer() throws Exception {
HddsProtos.ReplicationFactor.THREE).getContainerInfoList().size());
}
+ @Test
Review Comment:
With the simplifications mentioned above I don't think we need all these
tests, it looks like the AI went overboard here. These aren't actually testing
the `queryUpgradeStatus` method either. Looking at the other methods in this
class it looks like we usually only cover them with integration tests. So in
this PR I would switch all test usage of `queryUpgradeFinalizationProgress` to
`queryUpgradeStatus` for test coverage.
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/upgrade/SCMUpgradeFinalizer.java:
##########
@@ -177,4 +146,76 @@ private void
waitForDatanodesToFinalize(SCMUpgradeFinalizationContext context)
}
}
}
+
+ public static DatanodeFinalizationCounts getNumFinalizedDatanodes(
+ NodeManager nodeManager) {
+ int finalizedNodes = 0;
+ int totalHealthyNodes = 0;
+ int unfinalizedNodes = 0;
+
+ for (DatanodeDetails dn : nodeManager.getAllNodes()) {
+ try {
+ // Only check HEALTHY nodes. STALE/DEAD nodes will be told to
+ // finalize when they recover.
+ if (!nodeManager.getNodeStatus(dn).isHealthy()) {
Review Comment:
This doesn't address the [previous
comment](https://github.com/apache/ozone/pull/10084#discussion_r3110465695)
because it is only checking the health state returned by `getNodeStatus`, not
the operational state. We can probably just loop over
`nodeManager.getNodes(NodeStatus.inServiceHealthy())` then not need to skip any
entries in the loop.
##########
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodeManager.java:
##########
@@ -737,6 +738,71 @@ public void testProcessLayoutVersion() throws IOException {
testProcessLayoutVersionReportHigherMlv();
}
+ @Test
+ public void testDatanodeFinalizedCounterTracksLayoutVersionReports()
+ throws IOException, AuthenticationException {
+ try (SCMNodeManager nodeManager = createNodeManager(getConf())) {
+ DatanodeDetails node =
+ HddsTestUtils.createRandomDatanodeAndRegister(nodeManager);
+ assertEquals(1, SCMUpgradeFinalizer.getNumFinalizedDatanodes(nodeManager)
+ .getNumFinalizedDatanodes(),
+ "Initial datanode should be counted as finalized");
+
+ int softwareVersion =
+ nodeManager.getLayoutVersionManager().getSoftwareLayoutVersion();
+ int metadataVersion =
+ nodeManager.getLayoutVersionManager().getMetadataLayoutVersion();
+ nodeManager.processLayoutVersionReport(node,
+ LayoutVersionProto.newBuilder()
+ .setMetadataLayoutVersion(metadataVersion - 1)
+ .setSoftwareLayoutVersion(softwareVersion)
+ .build());
+ assertEquals(0, SCMUpgradeFinalizer.getNumFinalizedDatanodes(nodeManager)
+ .getNumFinalizedDatanodes(),
+ "Lower metadata layout version should decrement finalized count");
+
+ nodeManager.processLayoutVersionReport(node,
+ LayoutVersionProto.newBuilder()
+ .setMetadataLayoutVersion(metadataVersion)
+ .setSoftwareLayoutVersion(softwareVersion)
+ .build());
+ assertEquals(1, SCMUpgradeFinalizer.getNumFinalizedDatanodes(nodeManager)
+ .getNumFinalizedDatanodes(),
+ "Restored metadata layout version should restore finalized count");
+ }
+ }
+
+ @Test
+ public void testDatanodeFinalizedCounterTracksRegistrationAndRemoveNode()
+ throws IOException, AuthenticationException, NodeNotFoundException {
+ try (SCMNodeManager nodeManager = createNodeManager(getConf())) {
+ DatanodeDetails finalizedNode =
+ registerWithCapacity(nodeManager, CORRECT_LAYOUT_PROTO, success);
+ assertEquals(1, SCMUpgradeFinalizer.getNumFinalizedDatanodes(nodeManager)
+ .getNumFinalizedDatanodes(),
+ "Finalized registration should increment finalized count");
+
+ DatanodeDetails nonFinalizedNode =
+ registerWithCapacity(nodeManager, SMALLER_MLV_LAYOUT_PROTO, success);
+ assertEquals(1, SCMUpgradeFinalizer.getNumFinalizedDatanodes(nodeManager)
+ .getNumFinalizedDatanodes(),
+ "Non-finalized registration should not increment finalized count");
+
+ nonFinalizedNode.setPersistedOpState(
+ HddsProtos.NodeOperationalState.DECOMMISSIONED);
Review Comment:
We should have test coverage for all ineligible health and op states to make
the intent clear. It should be fast to enumerate all of them.
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/upgrade/SCMUpgradeFinalizer.java:
##########
@@ -177,4 +146,76 @@ private void
waitForDatanodesToFinalize(SCMUpgradeFinalizationContext context)
}
}
}
+
+ public static DatanodeFinalizationCounts getNumFinalizedDatanodes(
+ NodeManager nodeManager) {
+ int finalizedNodes = 0;
+ int totalHealthyNodes = 0;
+ int unfinalizedNodes = 0;
+
+ for (DatanodeDetails dn : nodeManager.getAllNodes()) {
+ try {
+ // Only check HEALTHY nodes. STALE/DEAD nodes will be told to
+ // finalize when they recover.
+ if (!nodeManager.getNodeStatus(dn).isHealthy()) {
+ continue;
+ }
+ totalHealthyNodes++;
+ DatanodeInfo datanodeInfo = nodeManager.getDatanodeInfo(dn);
+ if (datanodeInfo == null) {
+ LOG.warn("Could not get DatanodeInfo for {}, skipping in " +
+ "finalization wait.", dn.getHostName());
+ continue;
+ }
+
+ LayoutVersionProto dnLayout = datanodeInfo.getLastKnownLayoutVersion();
+ int dnMlv = dnLayout.getMetadataLayoutVersion();
+ int dnSlv = dnLayout.getSoftwareLayoutVersion();
+
+ if (dnMlv < dnSlv) {
+ // Datanode has not yet finalized
+ unfinalizedNodes++;
+ LOG.debug("Datanode {} has not yet finalized: MLV={}, SLV={}",
+ dn.getHostName(), dnMlv, dnSlv);
+ } else {
+ finalizedNodes++;
+ }
+ } catch (NodeNotFoundException e) {
+ // Node was removed while we were iterating. This is OK, skip it.
+ LOG.debug("Node {} not found while waiting for finalization, " +
+ "skipping.", dn);
+ }
+ }
+
+ return new DatanodeFinalizationCounts(finalizedNodes, totalHealthyNodes,
unfinalizedNodes);
+ }
+
+ /**
+ * Class to store the number finalized, unfinalized and healthy datanodes.
+ */
+ public static final class DatanodeFinalizationCounts {
+ private final int numFinalizedDatanodes;
+ private final int totalHealthyDatanodes;
+ private final int numUnfinalizedDatanodes;
Review Comment:
Do we actually need three fields here? I think we can just track total
healthy in-service nodes and number of finalized nodes. The number of
unfinalized nodes can be derived from that. These are the only two fields the
upgrade status API uses.
The finalization wait code can be changed to check that the healthy node
count equals the finalized count. We could also wrap that in an
`allNodesFinalized` method inside `DatanodeFinalizationCount`.
`getNumUnfinalizedDatanodes` has no test coverage, but I think it should be
removed. `getTotalHealthyDatanodes` also has no test coverage, which should be
added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]