sharpca opened a new pull request, #1347:
URL: https://github.com/apache/flink-agents/pull/1347
Linked issue: N/A (Python counterpart of #1336 / #1337)
### Purpose of change
A Python job configured through `flink-conf.yaml` with flat `agent.<key>`
entries now starts with those settings instead of silently running on
defaults,
and an empty configuration file no longer aborts the job.
The Python loader only read the nested `agent` mapping. A legacy
`flink-conf.yaml` normally declares settings as flat entries, which PyYAML
returns as literal `agent.<key>` keys, so the legacy path loaded nothing.
Flat
entries are also what the Java loader accepts (#1337).
Runtime flow: `RemoteAgentBuilder.__load_config_from_flink_conf_dir` selects
the
file through `__find_config_file` (unchanged: `flink-conf.yaml` if present,
else
`config.yaml`) and calls `AgentConfiguration.load_from_file`, which now
parses
the document with `yaml.safe_load`, hands it to the new
`_extract_agent_config`,
then flattens and normalizes the result into `conf_data`.
Key decisions:
- A flat entry wins over the same key in the nested mapping, matching the
Java
loader; the reverse order was rejected to keep the two languages aligned.
- A document that parses to `None` (empty file) is treated as no settings
rather
than an error, because an empty file is a valid "nothing configured" input.
- Wrong-shaped documents raise `TypeError` naming the file, following the
project's `TRY004` lint rule for wrong types; they previously escaped as an
unrelated `AttributeError`.
### Behavioral Semantics
#### Interaction decisions
| Document | `agent` section | Result |
|---|---|---|
| empty / only comments | absent | no settings |
| mapping | absent | no settings |
| mapping | `null` | no settings |
| mapping | mapping | nested settings, then flat `agent.*` entries override |
| mapping | scalar or list | `TypeError` naming the file |
| list or scalar | not reachable | `TypeError` naming the file |
| Declaration | Result |
|---|---|
| flat only | flat settings |
| nested only | nested settings |
| both, same key | flat value |
| both, different keys | union of both |
#### Behavioral contracts
- Flat top-level `agent.<key>` entries are loaded under `<key>`.
- Nested `agent` mapping entries are still loaded and nested maps are still
flattened to dot-separated paths.
- When both forms set the same key, the flat value is used.
- Top-level keys without the `agent.` prefix are ignored.
- An empty document or an empty `agent` section contributes no settings.
- A non-mapping document or a non-mapping `agent` section raises `TypeError`
naming the file.
- File selection order is unchanged.
#### Failure behavior
- Missing file: `FileNotFoundError` (unchanged).
- Invalid YAML: `yaml.YAMLError` (unchanged).
- Empty document: falls back to no settings (previously `AttributeError`).
- Non-mapping document or `agent` section: raises `TypeError` (previously
`AttributeError` from `flatten_dict`).
- No retries and no other absorbed errors.
### Tests
| Contract | Test |
|---|---|
| Flat entries loaded and stripped |
`test_load_configuration_from_flat_agent_keys` |
| Nested form unchanged, nested maps flattened |
`test_load_configuration_from_file` (existing) |
| Flat wins over nested |
`test_flat_agent_keys_override_nested_agent_section` |
| Non-`agent.` keys ignored | `test_load_configuration_from_flat_agent_keys`
|
| Empty file / empty section | `test_load_empty_configuration_file`,
`test_load_configuration_with_empty_agent_section` |
| Wrong document or section shape |
`test_load_configuration_rejects_non_mapping_document`,
`test_load_configuration_rejects_non_mapping_agent_section` |
| File selection order | no test; `__find_config_file` is untouched |
Not verified: the in-tree pytest module imports `AgentPlan`, which imports
`pyflink`, and `pyflink` is not installed in this environment. The six new
cases
were executed with pytest against `AgentConfiguration` directly (6 passed);
CI
runs the in-tree module. The `FLINK_CONF_DIR` path through
`RemoteExecutionEnvironment` was not exercised end to end.
<details>
<summary>Implementation invariants and supporting detail</summary>
- Flat-key extraction skips keys that are not strings and keys whose suffix
after `agent.` is empty, so a literal `agent.` key cannot create an empty
option name.
- The `agent` key itself is never treated as a flat entry.
- Validation happens before flattening, so a malformed document fails before
any setting reaches `conf_data`.
</details>
### API
Compatibility impact: no signature change. Nested-form users see no
difference.
Flat-form users gain settings that were previously dropped. An empty file no
longer raises. A wrongly shaped file now raises `TypeError` instead of
`AttributeError`; neither was documented. The Java loader and the file
selection order do not change.
### Documentation
- [ ] `doc-needed`
- [ ] `doc-not-needed`
- [x] `doc-included`
`docs/content/docs/operations/configuration.md` now states that agent
settings
can use either the nested mapping or flat `agent.`-prefixed keys, and that a
flat key wins on a conflict. Reconcile the wording with #1337 if that PR
lands
first.
### Was this patch authored or co-authored using generative AI tooling?
- [x] Yes
- [ ] No
Generated-by: WPS Lingxi AI agent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
