Anton Kalashnikov created FLINK-40379:
-----------------------------------------
Summary: AdaptiveScheduler is in restart loop when task
cancellation consistently times out
Key: FLINK-40379
URL: https://issues.apache.org/jira/browse/FLINK-40379
Project: Flink
Issue Type: Bug
Reporter: Anton Kalashnikov
When task cancellation keeps timing out during a scale-up, the affected TM is
killed, the cluster ends up with fewer slots than before, and the
AdaptiveScheduler loops forever in Restarting -> WaitingForResources ->
CreatingExecutionGraph -> Restarting, never making progress
*Root cause*
WaitingForResources, doesn't wait for resources long enough after the TM was
canceled(if TM is stuck in canceled, it will be terminated in a hard way and it
needs some time to restart it). So the stabilization timeout for after restart
is normally small to avoid downtime in normal cases, and "sufficient resources"
means slots >= lowerBound but lowerBound is hard-coded to 1, so the gate is
always open
*result:*
The scheduler immediately re-enters CreatingExecutionGraph -> Executing,
cancellation times out again, the TM is killed, and the loop continues with the
same or fewer slots
State transition entry point: Restarting.java:116 in confluentinc/flink
(onGloballyTerminalState handler)
*Fix proposal:*
Adding `AdaptiveScheduler#getFreeSlotsVertexParallelism()`, computed from free
slots only (`DeclarativeSlotPool#getFreeSlotTracker`), excluding slots still
reserved by the execution that is being canceled. WaitingForResources now uses
this (via a targetParallelism gate) to redefine "desired resources" only for
rescale-triggered restarts: desired means the free-slot-achievable parallelism
is at least what was free right before the restart. The onFailure (TM/slot
loss) and reconfigure paths are untouched: they never carry a restart target,
so they keep the original all-slots-based
hasSufficientResources()/hasDesiredResources() unchanged
Adding a new dedicated option
`jobmanager.adaptive-scheduler.rescale.resource-stabilization-timeout` (default
2 min), used only for rescale-triggered restarts. It bounds how long the
scheduler waits, after a rescale-restart, for free slots to reach the
pre-restart parallelism before falling back to running with whatever's
currently sufficient. Keeping it separate from the submission path means
initial job-submission latency is unaffected
*Behavior comparison:*
||Scenario||Resource situation||master||This branch (final)||Difference||
|Initial submission|Full desired reached in time|Don't wait as soon as
sufficient|Identical - this path never carries a restart target, code
untouched|None|
|Initial submission|Only sufficient (< desired) available|After sufficient
resources appear, waits 10s for desired, then runs with
sufficient|Identical|None|
|Scale up (RESCALE)|Free slots catch up to pre-restart target quickly|All-slots
check usually (incorrectly) reports "unchanged" right after cancel and jumps
straight to {{{}CreatingExecutionGraph{}}}, potentially racing on slots not yet
actually free|Free-slots check correctly detects the gap, enters
{{{}WaitingForResources{}}}, but exits the stabilizing phase almost as fast
once slots genuinely free up|Similar wall-clock time in the good case, but no
longer racing ahead of real slot availability|
|Scale up (RESCALE)|Sufficient but < pre-restart target for a while|No
dedicated wait tied to the rescale target: either proceeds near-immediately on
the (possibly wrong) all-slots view, or - if the mismatch is caught - waits
only 10s against the job's generic desired resources (not the specific
pre-restart target), then runs at whatever's sufficient|Waits up to 2 min
specifically for free slots to reach the pre-restart target before falling back
to "sufficient"|New code waits materially longer (2 min vs ~0-10s) for the
actual intended target before settling for less - fewer premature under-scaled
restarts, at the cost of up to ~2 min extra restart latency in the worst case|
|Scale down (RESCALE)|Same two sub-cases - identical code path,
direction-agnostic (only checks whether free slots reach what was achievable
pre-restart)|Same as scale-up row|Same as scale-up row|Same as scale-up row|
|TaskManager/slot failure (task exception restart)|Desired reached quickly /
only sufficient available|Restart target is {{null}} here in both versions -
untouched, uses 10s + all-slots desired/sufficient checks|Identical|None|
|Reconfigure (JAR/config change restart)|Desired reached quickly / only
sufficient available|Restart target is {{null}} here in both versions -
untouched|Identical|None|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
