smengcl commented on code in PR #11962:
URL: https://github.com/apache/ozone/pull/11962#discussion_r4509152804
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeQueueMetrics.java:
##########
@@ -60,6 +61,16 @@ public final class DatanodeQueueMetrics implements
MetricsSource {
"ContainerActionQueue";
public static final String PIPELINE_ACTION_QUEUE_PREFIX =
"PipelineActionQueue";
+ public static final String COMMAND_STATUS_REPORT_QUEUE_PREFIX =
+ "CommandStatusReportQueue";
+ public static final String COMMAND_STATUS_REPORT_SENT_PREFIX =
+ "CommandStatusReportSent";
+ public static final String COMMAND_STATUS_REPORT_DEFERRED_PREFIX =
Review Comment:
`deferredBytes` comes from `getCommandStatusReportQueueBytes(endpoint)`
right after selection, so `CommandStatusReportDeferred` reports the same value
as the `CommandStatusReportQueue` gauge, only sampled at a different time.
Maybe drop one of them? That also removes two of the new maps.
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java:
##########
@@ -73,6 +73,10 @@ public final class HddsConfigKeys {
"hdds.heartbeat.report.max.limit";
public static final int HDDS_HEARTBEAT_REPORT_MAX_LIMIT_DEFAULT =
10000;
+ public static final String HDDS_HEARTBEAT_MAX_COMMAND_STATUS_REPORT_SIZE =
+ "hdds.heartbeat.max.command.status.report.size";
Review Comment:
nit: the related keys put the limit last (`hdds.heartbeat.report.max.limit`,
`hdds.container.action.max.limit`). How about
`hdds.heartbeat.command.status.report.max.size` to match? Easy to change while
this is still a draft.
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/states/endpoint/HeartbeatEndpointTask.java:
##########
@@ -148,25 +170,34 @@ public EndpointStateMachine.EndPointStates call() throws
Exception {
requestBuilder = SCMHeartbeatRequestProto.newBuilder()
.setDatanodeDetails(datanodeDetailsProto)
.setDatanodeVersion(versionInfo);
- addReports(requestBuilder);
+ int reportCount = addNonCommandStatusReports(requestBuilder);
addContainerActions(requestBuilder);
addPipelineActions(requestBuilder);
addQueuedCommandCounts(requestBuilder);
+ commandStatusSelection = addCommandStatusReports(
+ requestBuilder, maxReportsPerHB - reportCount);
SCMHeartbeatRequestProto request = requestBuilder.build();
LOG.debug("Sending heartbeat message : {}", request);
SCMHeartbeatResponseProto response = rpcEndpoint.getEndPoint()
.sendHeartbeat(request);
processResponse(response, datanodeDetailsProto);
rpcEndpoint.setLastSuccessfulHeartbeat(ZonedDateTime.now());
rpcEndpoint.zeroMissedCount();
- if (context.hasPendingReports(rpcEndpoint.getAddress())) {
- // addReports was capped at maxReportsPerHB and left reports queued.
Trigger immediately.
- context.getParent().setNextHB(Time.monotonicNow());
+ if (context.getQueueMetrics() != null) {
Review Comment:
Removing the immediate follow-up heartbeat changes behavior for all queued
reports, not only command statuses. An ICR backlog, or a full report left
behind by the count cap, now drains one `hdds.heartbeat.report.max.limit` batch
per heartbeat interval instead of right away, which is what HDDS-16723 added
the follow-up for. Could we keep it here, or move this part to its own JIRA?
If it stays, the `hdds.heartbeat.report.max.limit` description in
ozone-default.xml still says reports beyond the limit "are sent in an immediate
follow-up heartbeat", so it needs an update. Also `hasPendingReports()` and
`getAllAvailableReports(endpoint, maxLimit)` would only be used by tests after
this, so they could go instead of updating the javadoc.
##########
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/states/endpoint/TestHeartbeatEndpointTask.java:
##########
@@ -441,16 +451,169 @@ public void
testHeartbeatCapsReportsAndSchedulesFollowup() throws Exception {
// Only the capped number of ICRs is sent in this heartbeat.
SCMHeartbeatRequestProto heartbeat = argument.getValue();
assertEquals(2, heartbeat.getIncrementalContainerReportCount());
- // The remainder stays queued and a follow-up heartbeat is scheduled now.
+ // The remainder stays queued for the next regular heartbeat.
assertTrue(context.hasPendingReports(TEST_SCM_ENDPOINT));
- verify(datanodeStateMachine).setNextHB(anyLong());
+ verify(datanodeStateMachine, never()).setNextHB(anyLong());
- // The follow-ups drain the rest; the last one sends 1 ICR and schedules
nothing.
+ // Subsequent regular heartbeats drain the rest.
endpointTask.call();
endpointTask.call();
assertEquals(1, argument.getValue().getIncrementalContainerReportCount());
assertFalse(context.hasPendingReports(TEST_SCM_ENDPOINT));
- verify(datanodeStateMachine, times(2)).setNextHB(anyLong());
+ verify(datanodeStateMachine, never()).setNextHB(anyLong());
+ }
+
+ @Test
+ public void testHeartbeatCapsCommandStatusReportsByBytes()
Review Comment:
Could we add a case where non command status reports use up the count limit?
For example `hdds.heartbeat.report.max.limit=2` with 2 ICRs and 1 command
status report should send no command status and keep it queued. That covers the
`maxReportsPerHB - reportCount` path. The new metrics (sent, requeued,
oversized bytes) are not checked by any test either.
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java:
##########
@@ -392,6 +395,71 @@ public List<Message> getAllAvailableReports(
return getAllAvailableReportsUpToLimit(endpoint, maxLimit);
}
+ /**
+ * Returns available reports other than command status reports. Full reports
+ * are selected before incremental reports, and the total number returned is
+ * capped by {@code maxLimit}.
+ */
+ public List<Message> getAvailableNonCommandStatusReports(
+ HostAndPort endpoint, int maxLimit) {
+ List<Message> reports = getFullReports(endpoint, maxLimit);
+ reports.addAll(getIncrementalReportsExcludingCommandStatus(
+ endpoint, maxLimit - reports.size()));
+ return reports;
+ }
+
+ /**
+ * Selects command status reports within both the report count and serialized
+ * byte limits. An aggregate report may be split between complete command
+ * statuses, but an individual command status is never split.
+ */
+ public CommandStatusReportSelection getCommandStatusReports(
+ HostAndPort endpoint, int maxLimit, long maxBytes,
+ long maxConfiguredBytes) {
+ List<CommandStatusReportsProto> selected = new LinkedList<>();
+ long selectedBytes = 0;
+ long oversizedBytes = 0;
+ synchronized (incrementalReportsQueue) {
+ List<Message> reportsForEndpoint =
+ incrementalReportsQueue.get(endpoint);
+ if (reportsForEndpoint != null) {
+ ListIterator<Message> iterator = reportsForEndpoint.listIterator();
+ while (iterator.hasNext() && selected.size() < maxLimit) {
+ Message message = iterator.next();
+ if (!(message instanceof CommandStatusReportsProto)) {
+ continue;
+ }
+
+ CommandStatusReportsProto report =
+ (CommandStatusReportsProto) message;
+ long remainingBytes = maxBytes - selectedBytes;
+ long reportBytes = getCommandStatusReportWireSize(report);
+ if (reportBytes <= remainingBytes) {
+ selected.add(report);
+ selectedBytes += reportBytes;
+ iterator.remove();
+ continue;
+ }
+
+ CommandStatusReportSplit split =
+ splitCommandStatusReport(report, remainingBytes);
+ if (split.getSelected() != null) {
+ selected.add(split.getSelected());
+ selectedBytes +=
+ getCommandStatusReportWireSize(split.getSelected());
+ iterator.set(split.getRemainder());
+ } else if (getFirstCommandStatusWireSize(report)
Review Comment:
When a single CommandStatus does not fit the budget, it stays at the head of
the queue and the `break` holds back every command status behind it, on every
heartbeat. `CommandStatusReportPublisher` keeps appending a report each
interval, so the queue grows and SCM gets no command ACKs from this datanode
until the config is raised.
The oversized check also compares only against the configured limit. If the
limit is set above 90% of `ipc.maximum.data.length`, a status larger than
`safeHeartbeatSize` is stuck the same way but never reported as oversized.
Would it work to send such a status alone when it still fits in the safe RPC
size, and to check oversized against the effective cap (min of configured limit
and safe size)?
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java:
##########
@@ -73,6 +73,10 @@ public final class HddsConfigKeys {
"hdds.heartbeat.report.max.limit";
public static final int HDDS_HEARTBEAT_REPORT_MAX_LIMIT_DEFAULT =
10000;
+ public static final String HDDS_HEARTBEAT_MAX_COMMAND_STATUS_REPORT_SIZE =
+ "hdds.heartbeat.max.command.status.report.size";
+ public static final String
+ HDDS_HEARTBEAT_MAX_COMMAND_STATUS_REPORT_SIZE_DEFAULT = "64MB";
Review Comment:
nit: this fits on one line within the 120 char limit. Same for many other
wrapped lines in the patch, such as `testHeartbeatCapsReports()` / `throws
Exception` and the `addCommandStatusReports(requestBuilder, maxReportsPerHB -
reportCount)` call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]