[
https://issues.apache.org/jira/browse/FLINK-40800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18143590#comment-18143590
]
Giannis Polyzos commented on FLINK-40800:
-----------------------------------------
The same stuck state also occurs without any manual rescale, through a redeploy
on Flink 2.x.
*Setup:* Flink Kubernetes Operator 1.16.1, Flink 2.3.0, application mode,
adaptive scheduler, last-state upgrades.
*What happened:*
# Under load, the autoscaler scaled a job out in place.
# The job was redeployed (a configuration change, a restart nonce, and a
suspend and resume were each tested).
# On Flink 2.x the parallelism overrides are not applied at submission
(FLINK-38770), so the job came back at its configured parallelism.
# The stored overrides were unchanged, so the operator saw no difference in the
spec. For more than 10 minutes the autoscaler reported an active backlog every
cycle but made no new decision.
# Only resetting the autoscaler state (autoscalerResetNonce) recovered it.
With a build that includes the FLINK-38770 fix the parallelism survives all of
these redeploys, but the underlying gap remains: the autoscaler trusts its
stored decision without checking the running job.
*Suggested fix covering both cases:* at each evaluation, compare each vertex's
actual running parallelism with the stored override. When they differ, emit a
warning event naming the vertices and both values, then re-apply the override
(or clear that vertex's scaling history so a new decision can be made). This
would recover from a manual rescale, a lost override after a redeploy, or any
other path that changes parallelism outside the autoscaler.
> When scaling in Flink is performed Autoscaler decision is not applied anymore
> -----------------------------------------------------------------------------
>
> Key: FLINK-40800
> URL: https://issues.apache.org/jira/browse/FLINK-40800
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
> Affects Versions: 1.16.1
> Environment: Real kubernetes cluster with Flink Kubernetes Operator
> 1.16.1 ({{{}ghcr.io/apache/flink-kubernetes-operator:1c895a3{}}}).
> FlinkDeployment with JobSpec ({{{}flink:1.20.5{}}}) running WindowJoin
> example.
> Reporter: Sergey Kononov
> Priority: Major
>
> I encountered a problem on a Kubernetes cluster with Operator with Autoscaler
> enabled. If user scaled a job with {{jobmanager.scheduler: adaptive}} using
> Flink Web UI (or REST) Autoscaler can't apply its scaling decision anymore.
> In logs user can see only repeating SCALINGREPORT like this:
>
> {noformat}
> o.a.f.k.o.l.AuditUtils [INFO ][fko/test-windowjoin-autoscaler] >>>
> Event[Job] | Info | SCALINGREPORT | Scaling execution enabled,
> begin scaling vertices:{ Vertex ID 17e79f7e136f9548649fb1b735f40eae |
> Parallelism 10 -> 12 | Processing capacity 1501097.47 -> 1801316.96 | Target
> data rate 1490558.47}{noformat}
> But no in-place scaling trigger after that. It doesn't looks like an intended
> behavior.
> My investigation showed that it happens because parallelism overrides
> evaluated by Autoscaler stay the same and the resource reconciler doesn't
> see a difference in specs and skips a scaling phase. One needs to redeploy
> the job or apply {{autoscalerResetNonce}} (that leads to a job redeployment)
> to force scaling by Autoscaler. That may be undesired. Or one can wait when
> Autoscaler evaluates a different desired parallelism then reconciler will see
> a diff in specs and apply the parallelism overrides.
>
>
> I propose a simple solution for this. One can add a scaling counter to
> JobSpec. Whenever Autoscaler evaluates a new scaling decision it increments
> it. This way the reconciler can see a difference in specs and applies the
> parallelism overrides. I can submit a PR if assigned.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
