Maximilian Michels created FLINK-33710:
------------------------------------------
Summary: Autoscaler redeploys pipeline for a NOOP parallelism
change
Key: FLINK-33710
URL: https://issues.apache.org/jira/browse/FLINK-33710
Project: Flink
Issue Type: Bug
Components: Autoscaler, Kubernetes Operator
Affects Versions: kubernetes-operator-1.7.0, kubernetes-operator-1.6.0
Reporter: Maximilian Michels
Assignee: Maximilian Michels
Fix For: kubernetes-operator-1.8.0
The operator supports two modes to apply autoscaler changes:
# Use the internal Flink config {{pipeline.jobvertex-parallelism-overrides}}
# Make use of Flink's Rescale API
For (1), a string has to be generated for the Flink config with the actual
overrides. This string has to be deterministic for a given map. But it is not.
Consider the following observed log:
{noformat}
>>> Event | Info | SPECCHANGED | SCALE change(s) detected (Diff:
FlinkDeploymentSpec[flinkConfiguration.pipeline.jobvertex-parallelism-overrides
:
92542d1280187bd464274368a5f86977:3,9f979ed859083299d29f281832cb5be0:1,84881d7bda0dc3d44026e37403420039:1,1652184ffd0522859c7840a24936847c:1
->
9f979ed859083299d29f281832cb5be0:1,84881d7bda0dc3d44026e37403420039:1,92542d1280187bd464274368a5f86977:3,1652184ffd0522859c7840a24936847c:1]),
starting reconciliation.
{noformat}
The overrides are identical but the order is different which triggers a
redeploy. This does not seem to happen often but some deterministic string
generation (e.g. sorting by key) is required to prevent any NOOP updates.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
