Copilot commented on code in PR #11962:
URL: https://github.com/apache/ozone/pull/11962#discussion_r4643861140
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/report/CommandStatusReportPublisher.java:
##########
@@ -63,21 +65,31 @@ protected long getReportFrequency() {
@Override
protected CommandStatusReportsProto getReport() {
+ return null;
+ }
+
+ @Override
+ protected List<CommandStatusReportsProto> getIncrementalReports() {
Map<Long, CommandStatus> map = this.getContext()
.getCommandStatusMap();
Iterator<Long> iterator = map.keySet().iterator();
- CommandStatusReportsProto.Builder builder = CommandStatusReportsProto
- .newBuilder();
+ List<CommandStatusReportsProto> reports = new ArrayList<>();
- iterator.forEachRemaining(key -> {
+ while (iterator.hasNext()) {
+ Long key = iterator.next();
CommandStatus cmdStatus = map.get(key);
+ if (cmdStatus == null) {
+ continue;
+ }
+ reports.add(CommandStatusReportsProto.newBuilder()
+ .addCmdStatus(cmdStatus.getProtoBufMessage())
+ .build());
// If status is still pending then don't remove it from map as
// CommandHandler will change its status when it works on this command.
if (!cmdStatus.getStatus().equals(Status.PENDING)) {
map.remove(key);
}
Review Comment:
A command can finish between building this report and the subsequent
live-status check. In that interleaving, the queued protobuf still contains
PENDING, but the check sees EXECUTED and removes the command from the map, so
its final delete ACK is never published. StateContext.updateCommandStatus
updates through computeIfPresent; please serialize and decide whether to remove
the entry atomically through the same per-key operation.
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/states/endpoint/HeartbeatEndpointTask.java:
##########
@@ -256,26 +272,54 @@ private void addReports(SCMHeartbeatRequestProto.Builder
requestBuilder) {
} else {
requestBuilder.setField(descriptor, report);
}
+ heartbeatSize.add(descriptor.getNumber(), report);
break;
}
}
}
+ return reportCount;
+ }
+
+ private void addCommandStatusReports(
+ SCMHeartbeatRequestProto.Builder requestBuilder,
+ int remainingReportCount, long currentHeartbeatSize) {
+ long remainingSafeBytes = Math.max(
+ 0, safeHeartbeatSize - currentHeartbeatSize);
+ CommandStatusReportSelection selection =
+ context.getCommandStatusReports(rpcEndpoint.getAddress(),
+ remainingReportCount, remainingSafeBytes, safeHeartbeatSize);
Review Comment:
This applies only the remaining IPC budget; the proposed
hdds.heartbeat.max.command.status.report.size setting is neither defined nor
read. Consequently, configuring a smaller command-status limit has no effect,
and the default permits roughly 115 MiB instead of the stated 64 MB when other
heartbeat fields are small. Please add the setting with its default and pass
the minimum of that limit and remainingSafeBytes to the selector.
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java:
##########
@@ -393,38 +395,66 @@ public int clearCommandStatusReports(HostAndPort
endpoint) {
*/
public List<Message> getAllAvailableReports(
HostAndPort endpoint) {
- return getAllAvailableReportsUpToLimit(endpoint, Integer.MAX_VALUE);
+ List<Message> reports = getFullReports(endpoint, Integer.MAX_VALUE);
+ reports.addAll(getIncrementalReports(endpoint, Integer.MAX_VALUE -
reports.size()));
+ return reports;
}
/**
- * Returns up to {@code maxLimit} available reports for the endpoint,
- * removing them from the queue. Any reports beyond the limit stay queued
- * and are returned by a later call; callers can use {@link
- * #hasPendingReports(HostAndPort)} to detect that case and schedule a
- * follow-up heartbeat.
- *
- * @return List of reports
+ * Returns available reports other than command status reports. Full reports
+ * are selected before incremental reports, and the total number returned is
+ * capped by {@code maxLimit}.
*/
- public List<Message> getAllAvailableReports(
+ public List<Message> getAvailableNonCommandStatusReports(
HostAndPort endpoint, int maxLimit) {
- return getAllAvailableReportsUpToLimit(endpoint, maxLimit);
+ List<Message> reports = getFullReports(endpoint, maxLimit);
+ reports.addAll(getIncrementalReportsExcludingCommandStatus(
+ endpoint, maxLimit - reports.size()));
+ return reports;
}
/**
- * Returns true if any reports are still queued for the endpoint.
+ * Selects command status reports within both the report count and serialized
+ * byte limits.
*/
- public boolean hasPendingReports(HostAndPort endpoint) {
+ public CommandStatusReportSelection getCommandStatusReports(
+ HostAndPort endpoint, int maxLimit, long maxBytes,
+ long oversizedThreshold) {
+ List<CommandStatusReportsProto> selected = new LinkedList<>();
+ long selectedBytes = 0;
+ long oversizedBytes = 0;
synchronized (incrementalReportsQueue) {
- List<Message> reportsForEndpoint = incrementalReportsQueue.get(endpoint);
- if (reportsForEndpoint != null && !reportsForEndpoint.isEmpty()) {
- return true;
+ List<Message> reportsForEndpoint =
+ incrementalReportsQueue.get(endpoint);
+ if (reportsForEndpoint != null) {
+ Iterator<Message> iterator = reportsForEndpoint.iterator();
+ while (iterator.hasNext() && selected.size() < maxLimit) {
+ Message message = iterator.next();
+ if (!(message instanceof CommandStatusReportsProto)) {
+ continue;
+ }
+
+ CommandStatusReportsProto report =
+ (CommandStatusReportsProto) message;
+ long remainingBytes = maxBytes - selectedBytes;
+ long reportBytes = getCommandStatusReportWireSize(report);
+ if (reportBytes <= remainingBytes) {
+ selected.add(report);
+ selectedBytes += reportBytes;
+ iterator.remove();
+ if (reportBytes > oversizedThreshold) {
+ oversizedBytes = reportBytes;
+ }
+ continue;
+ }
+ if (reportBytes > oversizedThreshold) {
+ oversizedBytes = reportBytes;
+ }
+ break;
Review Comment:
A command-status report larger than safeHeartbeatSize can never fit, but
this break leaves it queued and blocks all later command-status reports on
every heartbeat. Since those heartbeats can succeed, the failure-triggered
queue reset does not recover the backlog. Handle permanently unsendable entries
explicitly, by bounding ACK size upstream or providing a recovery path that
lets later reports progress, rather than indefinitely deferring them.
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/states/endpoint/HeartbeatEndpointTask.java:
##########
@@ -153,21 +169,22 @@ public EndpointStateMachine.EndPointStates call() throws
Exception {
requestBuilder = SCMHeartbeatRequestProto.newBuilder()
.setDatanodeDetails(datanodeDetailsProto)
.setDatanodeVersion(versionInfo);
- addReports(requestBuilder);
- addContainerActions(requestBuilder);
- addPipelineActions(requestBuilder);
- addQueuedCommandCounts(requestBuilder);
+ SerializedSizeAccumulator heartbeatSize = new
SerializedSizeAccumulator();
+ heartbeatSize.add(SCMHeartbeatRequestProto.DATANODEDETAILS_FIELD_NUMBER,
datanodeDetailsProto);
+ heartbeatSize.add(SCMHeartbeatRequestProto.DATANODEVERSION_FIELD_NUMBER,
versionInfo);
+ int reportCount = addNonCommandStatusReports(requestBuilder,
heartbeatSize);
+ addContainerActions(requestBuilder, heartbeatSize);
+ addPipelineActions(requestBuilder, heartbeatSize);
+ addQueuedCommandCounts(requestBuilder, heartbeatSize);
+ addCommandStatusReports(
+ requestBuilder, maxReportsPerHB - reportCount, heartbeatSize.get());
Review Comment:
Selecting all non-command reports first can indefinitely starve
command-status reports: if ICRs keep filling maxReportsPerHB on every
heartbeat, remainingReportCount is always zero, even for an ACK queued before
those ICRs. Unlike the previous shared FIFO selection, later ICRs continually
overtake the ACK, preventing SCM from observing completed deletions. Please
reserve a fair share of report slots for queued command statuses (while still
adding their fields last for byte budgeting), and cover repeated heartbeats
with replenished ICRs.
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/states/endpoint/HeartbeatEndpointTask.java:
##########
@@ -256,26 +272,54 @@ private void addReports(SCMHeartbeatRequestProto.Builder
requestBuilder) {
} else {
requestBuilder.setField(descriptor, report);
}
+ heartbeatSize.add(descriptor.getNumber(), report);
Review Comment:
XceiverServerRatis.sendPipelineReport queues PipelineReportsProto as
incremental messages, so several selected reports can overwrite the same
non-repeated heartbeat field. The accumulator counts all of them even though
only the last is serialized, which can defer command ACKs despite available
space. Subtract the previous field's wire size before replacing it, or measure
the assembled non-command payload.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]