[
https://issues.apache.org/jira/browse/FLINK-40380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-40380:
-----------------------------------
Labels: pull-request-available (was: )
> Document the runtime configuration and stop jobs from overriding operator
> config through it
> -------------------------------------------------------------------------------------------
>
> Key: FLINK-40380
> URL: https://issues.apache.org/jira/browse/FLINK-40380
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
> Reporter: Dennis-Mircea Ciupitu
> Priority: Major
> Labels: pull-request-available
>
> h1. Summary
> FLINK-35746 added runtime configuration observation, where the operator reads
> a job's effective settings from the Flink REST API and layers them over the
> observed configuration. Two follow-ups are needed: the operator documentation
> still describes the previous configuration model, and the global job
> parameters ingested from the REST response are not filtered.
> h1. Documentation is out of date
> The {{Internals -> Startup -> Configuration}} section still describes the
> model as it was before FLINK-35746, and three statements there are now wrong.
> h2. The configuration count and the Observe row
> The section opens with "Three configurations matter throughout the operator"
> and lists Default, Deploy and Observe. There is now a fourth source, the
> runtime configuration, fetched from the cluster and cached separately from
> the derived configurations.
> The Observe row describes that configuration as derived purely from the last
> reconciled spec. {{FlinkResourceContext#getObserveConfig}} now layers the
> runtime configuration on top, so REST-observed values win over the spec.
> h2. The warning is now false
> The section closes with:
> {quote}
> None of these is the running configuration. The operator derives its view
> from the spec and its own defaults, while the cluster can pick up settings
> the operator never sees: a config.yaml baked into the image, environment
> overrides, or properties the job sets programmatically. The configuration a
> pipeline actually runs with can therefore differ from everything the operator
> tracks.
> {quote}
> Reading back settings that the job sets programmatically is exactly what
> FLINK-35746 implemented, so this tells the reader the operator cannot do the
> thing it now does. A narrower version of the caution is still true and worth
> keeping, because the mapping covers only a subset of settings and only while
> the job runs.
> h2. The mechanism is undocumented
> Nothing describes when the runtime configuration is fetched (once per job by
> the job status observer, skipped for globally terminal jobs), how it is
> cached (per resource and job id, under the existing config cache limits),
> which settings it covers, or what happens when a fetch fails.
> h1. Global job parameters are ingested unfiltered
> {{FlinkRuntimeConfigurationUtils#mapJobConfiguration}} copies the job's
> global parameters wholesale:
> {code:java}
> jobConfig.putAll(execInfo.getGlobalJobParameters());
> {code}
> Every other value produced by that class is an explicit, curated
> {{ConfigOption}}. This one is unbounded and set by the job, and the result is
> written over the observed configuration through {{observeConfig::setString}}.
> A job that declares a global parameter matching an operator key, for example
> under {{kubernetes.operator.}} or {{job.autoscaler.}}, therefore changes how
> the operator manages that resource, with no indication in the logs. Keys in
> those namespaces have no legitimate reason to arrive from a job's global
> parameters, so they should be dropped.
> h1. Expected outcome
> The Configuration section describes the runtime configuration as a
> first-class source, states that it overrides the observed configuration, and
> replaces the warning with an accurate account of what the operator can and
> cannot observe.
> Global job parameters in the operator and autoscaler namespaces are ignored
> rather than applied, and the drop is logged so it is diagnosable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
