FrankChen021 commented on code in PR #20741:
URL: https://github.com/apache/druid/pull/20741#discussion_r4388800627
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:
##########
@@ -4283,23 +4333,25 @@ private void checkPendingCompletionTasks()
for (Entry<Integer, CopyOnWriteArrayList<TaskGroup>> pendingGroupList :
pendingCompletionTaskGroups.entrySet()) {
- boolean stopTasksInTaskGroup = false;
+ // Partitions of the groups in this list that will not publish their
segments
+ final Set<PartitionIdType> failedPartitions = new HashSet<>();
Integer groupId = pendingGroupList.getKey();
CopyOnWriteArrayList<TaskGroup> taskGroupList =
pendingGroupList.getValue();
List<TaskGroup> toRemove = new ArrayList<>();
for (TaskGroup group : taskGroupList) {
boolean foundSuccess = false, entireTaskGroupFailed = false;
- if (stopTasksInTaskGroup) {
- // One of the earlier groups that was handling the same partition
set timed out before the segments were
- // published so stop any additional groups handling the same
partition set that are pending completion.
+ // After a scale action, the groups in this list may have been created
with different taskCounts, so they do
+ // not necessarily handle the same partition set. Only stop the groups
that overlap with a failed group.
+ if (!Collections.disjoint(failedPartitions,
group.startingSequences.keySet())) {
Review Comment:
[P1] Propagate failed partitions across pending-group lists
**Finding:** After scaling changes the partition-to-group mapping, an
earlier pending completion group can be stored under one map key while a later
overlapping group is stored under another. failedPartitions is recreated for
each pendingGroupList, so the later group bypasses this overlap check and is
not stopped. The helper resets the offset and onCompleted() then releases any
waiting task, allowing that later task to publish from the earlier group's
unpublished offset and potentially skip data.
**Suggestion:** Track failed partitions and overlapping pending groups
across the whole pending-completion map, or explicitly fail and unblock their
waiting tasks before invoking completion listeners; add a regression test
covering overlap across different group IDs after scaling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
