Zim95 opened a new pull request, #76220:
URL: https://github.com/apache/airflow/pull/76220
Temporal mappers only parse keys with `input_format`, so the output of one
mapper can't be consumed by another. A chain such as `StartOfHourMapper` ->
`StartOfDayMapper` fails with a `ValueError`, because `2026-07-30T14` doesn't
match the default `%Y-%m-%dT%H:%M:%S`.
Setting `input_format` works when a mapper receives one known format, as
`example_asset_partition.py` does. It doesn't cover a mapper that receives keys
from different mappers. It also can't parse the default quarter key, because
`{quarter}` isn't a `strptime` directive, or reliably read week keys, because
`%V` can't be round-tripped through `strptime` on its own.
closes: #70762
related: #67119
This builds on the discussion in #70781.
### Changes
- **`to_downstream` falls back to `_parse_default_key`**
- Why: `strptime` with `input_format` is the failure in the issue.
- Goal: accept other mappers' keys. `input_format` is still tried first,
so a key that already parses gets the same result, and a key neither step can
read raises the original error.
- **`_to_downstream_key` is extracted**
- Why: the fallback and the rollup path both need the same timezone,
`normalize` and `format` steps.
- Goal: share that code without changing it.
- **`_DEFAULT_KEY_PATTERNS` is added**
- Why: a key is a plain string with no record of which mapper produced it,
so every default format has to be recognised. Regex also handles the week and
quarter formats.
- Goal: build one pattern per mapper with the existing
`_compile_output_format_regex`, so the patterns follow each mapper's
`default_output_format`.
- **`_parse_default_key` is added**
- Why: a short key has to become a complete `datetime` before it can be
normalized.
- Goal: fill missing fields with the start of the period, for example
`2026-Q3` -> `2026-07-01T00:00:00`. Matching is stricter than `strptime`:
unpadded fields, impossible dates and week keys that aren't a Monday in the
stated ISO week are rejected.
- **`to_rollup_downstream` is added to temporal mappers**
- Why: a rollup waits for keys rendered in `input_format`. If it accepted
`2026-02-10T14`, the key would map to `2026-02-10` but never match an expected
upstream key, and the Dag run would be held forever.
- Goal: keep rollups strict, so these keys remain a logged mapping failure.
- **`PartitionMapper.to_rollup_downstream` is added and
`RollupMapper.to_downstream` calls it**
- Why: `RollupMapper` needs a way to ask for strict parsing from any
upstream mapper.
- Goal: the base method calls `to_downstream`, so other mappers are
unaffected.
- **Docs are updated in `assets.rst`**
- Why: chaining is now supported and has limits users need to know about.
- Goal: document chaining, the need for a matching `input_format` with a
custom `output_format`, and strict rollups.
- **Tests are added to `test_temporal.py`**
- Why: the parser accepts a new set of keys, and the existing behaviour
has to stay the same.
- Goal: cover chaining, malformed keys, `input_format` priority, timezone
handling and both rollup cases.
### Alternatives considered
- **Change the default `output_format` to a full timestamp:** this would
make chaining work, but it would change partition keys for existing Dags, could
split queued keys, and would replace readable keys such as `2026-Q3`.
- **Fall back to the mapper's own `output_format`:** a mapper could read its
own keys, but not those of other mappers.
- **Pad the key as text:** this can't handle quarter or week keys, and it
can turn a malformed key into a different valid date.
### Feedback welcome
- Should rollups stay strict through `to_rollup_downstream`?
- Coarse-to-fine mapping selects the start of the period, so `2026` maps to
`2026-01-01` through `StartOfDayMapper`. Should this be allowed?
- A week crossing a month boundary uses its Monday, so `2026-03-30 (W14)`
maps to `2026-03` through `StartOfMonthMapper`. Is that the expected result?
- Only default output formats are recognised. A key with a custom
`output_format` still needs a matching `input_format`.
This overlaps with #67119 in `temporal.py`, `base.py` and
`test_temporal.py`, so whichever PR lands second will need a rebase.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Claude Opus 5.5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
