m-trieu commented on code in PR #30764:
URL: https://github.com/apache/beam/pull/30764#discussion_r1559030285
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/computations/ActiveWorkState.java:
##########
@@ -124,11 +128,40 @@ private static Stream<HeartbeatRequest>
toHeartbeatRequestStream(
.setWorkToken(work.getWorkItem().getWorkToken())
.setCacheToken(work.getWorkItem().getCacheToken())
.addAllLatencyAttribution(
- work.getLatencyAttributions(
- /* isHeartbeat= */ true,
work.getLatencyTrackingId(), sampler))
+ work.getLatencyAttributions(/* isHeartbeat= */ true,
sampler))
.build());
}
+ private static Stream<DirectHeartbeatRequest>
toHeartbeatRequestStreamDirectPath(
+ Entry<ShardedKey, Deque<Work>> shardedKeyAndWorkQueue,
+ Instant refreshDeadline,
+ DataflowExecutionStateSampler sampler) {
+ ShardedKey shardedKey = shardedKeyAndWorkQueue.getKey();
+ Deque<Work> workQueue = shardedKeyAndWorkQueue.getValue();
+
+ return workQueue.stream()
+ .filter(work -> work.getStartTime().isBefore(refreshDeadline))
+ .peek(
+ work -> {
+ if (work.getProcessWorkItemClient().getDataStream().isClosed()) {
+ work.setFailed();
+ }
+ })
+ // Don't send heartbeats for queued work we already know is failed.
+ .filter(work -> !work.isFailed())
+ .map(
+ work ->
+ DirectHeartbeatRequest.create(
Review Comment:
changed how this works, ActiveWorkState/ComputationState just expose the
work map as a readonly map and have the ActiveWorkRefresher components manage
creating this heartbeats internally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
