[
https://issues.apache.org/jira/browse/FLINK-40379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-40379:
----------------------------------
Affects Version/s: 2.1.3
2.2.1
2.3.0
2.0.2
> AdaptiveScheduler is in restart loop when task cancellation consistently
> times out
> -----------------------------------------------------------------------------------
>
> Key: FLINK-40379
> URL: https://issues.apache.org/jira/browse/FLINK-40379
> Project: Flink
> Issue Type: Bug
> Affects Versions: 2.0.2, 2.3.0, 2.2.1, 2.1.3
> Reporter: Anton Kalashnikov
> Assignee: Anton Kalashnikov
> Priority: Major
>
> When task cancellation keeps timing out during a scale-up, the affected TM is
> killed, the cluster ends up with fewer slots than before, and the
> AdaptiveScheduler loops forever in Restarting -> WaitingForResources ->
> CreatingExecutionGraph -> Restarting, never making progress
> *Root cause*
> WaitingForResources, doesn't wait for resources long enough after the TM was
> canceled(if TM is stuck in canceled, it will be terminated in a hard way and
> it needs some time to restart it). So the stabilization timeout for after
> restart is normally small to avoid downtime in normal cases, and "sufficient
> resources" means slots >= lowerBound but lowerBound is hard-coded to 1, so
> the gate is always open
> *result:*
> The scheduler immediately re-enters CreatingExecutionGraph -> Executing,
> cancellation times out again, the TM is killed, and the loop continues with
> the same or fewer slots
> State transition entry point: Restarting.java:116 in confluentinc/flink
> (onGloballyTerminalState handler)
> *Fix proposal:*
> Adding `AdaptiveScheduler#getFreeSlotsVertexParallelism()`, computed from
> free slots only (`DeclarativeSlotPool#getFreeSlotTracker`), excluding slots
> still reserved by the execution that is being canceled. WaitingForResources
> now uses this (via a targetParallelism gate) to redefine "desired resources"
> only for rescale-triggered restarts: desired means the free-slot-achievable
> parallelism is at least what was free right before the restart. The onFailure
> (TM/slot loss) and reconfigure paths are untouched: they never carry a
> restart target, so they keep the original all-slots-based
> hasSufficientResources()/hasDesiredResources() unchanged
> Adding a new dedicated option
> `jobmanager.adaptive-scheduler.rescale.resource-stabilization-timeout`
> (default 2 min), used only for rescale-triggered restarts. It bounds how long
> the scheduler waits, after a rescale-restart, for free slots to reach the
> pre-restart parallelism before falling back to running with whatever's
> currently sufficient. Keeping it separate from the submission path means
> initial job-submission latency is unaffected
> *Behavior comparison:*
> ||Scenario||Resource situation||master||This branch (final)||Difference||
> |Initial submission|Full desired reached in time|Don't wait as soon as
> sufficient|Identical - this path never carries a restart target, code
> untouched|None|
> |Initial submission|Only sufficient (< desired) available|After sufficient
> resources appear, waits 10s for desired, then runs with
> sufficient|Identical|None|
> |Scale up (RESCALE)|Free slots catch up to pre-restart target
> quickly|All-slots check usually (incorrectly) reports "unchanged" right after
> cancel and jumps straight to {{{}CreatingExecutionGraph{}}}, potentially
> racing on slots not yet actually free|Free-slots check correctly detects the
> gap, enters {{{}WaitingForResources{}}}, but exits the stabilizing phase
> almost as fast once slots genuinely free up|Similar wall-clock time in the
> good case, but no longer racing ahead of real slot availability|
> |Scale up (RESCALE)|Sufficient but < pre-restart target for a while|No
> dedicated wait tied to the rescale target: either proceeds near-immediately
> on the (possibly wrong) all-slots view, or - if the mismatch is caught -
> waits only 10s against the job's generic desired resources (not the specific
> pre-restart target), then runs at whatever's sufficient|Waits up to 2 min
> specifically for free slots to reach the pre-restart target before falling
> back to "sufficient"|New code waits materially longer (2 min vs ~0-10s) for
> the actual intended target before settling for less - fewer premature
> under-scaled restarts, at the cost of up to ~2 min extra restart latency in
> the worst case|
> |Scale down (RESCALE)|Same two sub-cases - identical code path,
> direction-agnostic (only checks whether free slots reach what was achievable
> pre-restart)|Same as scale-up row|Same as scale-up row|Same as scale-up row|
> |TaskManager/slot failure (task exception restart)|Desired reached quickly /
> only sufficient available|Restart target is {{null}} here in both versions -
> untouched, uses 10s + all-slots desired/sufficient checks|Identical|None|
> |Reconfigure (JAR/config change restart)|Desired reached quickly / only
> sufficient available|Restart target is {{null}} here in both versions -
> untouched|Identical|None|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
