[
https://issues.apache.org/jira/browse/FLINK-40531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-40531:
-----------------------------------
Labels: pull-request-available (was: )
> K8s Operator Autoscaler Parallelism overrides serialized in standard-YAML
> format break the deployed job
> -------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40531
> URL: https://issues.apache.org/jira/browse/FLINK-40531
> Project: Flink
> Issue Type: Bug
> Components: Autoscaler, Kubernetes Operator
> Reporter: Juliusz Nadberezny
> Priority: Major
> Labels: pull-request-available
>
> {{Affects unreleased 1.17.0 version}}
> {{}}
> Flink can write map-valued config settings in two styles: the old {{a:1,b:2}}
> and the newer {{{}{a: '1', b: '2'}{}}}. Every Flink version can read the old
> style; only newer parsers can read the new one.
> {{}}
> The autoscaler lets Flink choose the style automatically, and that choice
> depends on which config file the operator itself was started with. So if the
> operator runs with {{{}config.yaml{}}}, it writes parallelism overrides in
> the new style into every FlinkDeployment it scales. Any JobManager still
> reading a legacy {{flink-conf.yaml}} then fails to start, because it cannot
> parse the value. The operator itself reports no problem — only the job breaks.
> {{KubernetesScalingRealizer}} and the autoscaler state stores serialize the
> parallelism overrides map with {{{}ConfigurationUtils.convertValue(overrides,
> String.class){}}}. The two-arg overload picks its format from
> {{{}GlobalConfiguration.isStandardYaml(){}}}, which is derived from the
> config file the _operator's own JVM_ loaded.
> If the operator runs with {{config.yaml}} (standard YAML) instead of
> {{{}flink-conf.yaml{}}}, it writes
> {{pipeline.jobvertex-parallelism-overrides: \{a: '1', b: '2'}}} into every
> FlinkDeployment it scales. A JobManager loading a legacy {{flink-conf.yaml}}
> cannot parse that and fails on startup with {{{}Could not split string.
> Illegal quoting at position: 5{}}}. The operator logs nothing — only the
> deployed job breaks.
> The legacy {{a:1,b:2}} form is parseable by all Flink versions, including 2.x
> (which reads both forms even though it serializes the new one). The flow form
> is not. So serialization should be pinned to the portable format rather than
> inherited from ambient operator state.
> Made worse by a self-inflicted trap: {{getOverrideString}} reuses the
> previous string verbatim when the override map is unchanged, to avoid a no-op
> redeploy. That pins a broken string in place permanently, so an affected job
> cannot recover on its own.
> *Steps to reproduce*
> # Deploy the operator with {{defaultConfiguration.config.yaml}} instead of
> {{{}flink-conf.yaml{}}}.
> # Let the autoscaler scale a {{v1_20}} FlinkDeployment.
> # Inspect {{spec.flinkConfiguration}} — the override string is {{{...}}} and
> the JobManager crash-loops.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
