rmittal46 opened a new pull request, #77263:
URL: https://github.com/apache/airflow/pull/77263
closes: #77256
### Problem
When a Dag is scheduled on an `AssetAny` condition and the condition is met,
the scheduler consumes every queued asset event for the Dag, including events
belonging to branches that were not satisfied.
For a condition like `(a & b) | (c & d)`:
1. `a` is updated → nothing runs (correct).
2. `c` and `d` are updated → the Dag runs, and the pending `a` event is
consumed too.
3. `b` is updated → nothing runs, because `a` was already consumed.
So the `a & b` branch can effectively never trigger if the other branch
completes first.
### Change
`AssetAny` gets an opt-in `consume` parameter:
```python
schedule = AssetAny(
AssetAll(asset_a, asset_b),
AssetAll(asset_c, asset_d),
consume="branch",
)
```
- `consume="all"` (default): current behaviour, unchanged.
- `consume="branch"`: a triggered run consumes only the events of the
satisfied branch(es). Events of other branches stay queued for a later run.
Implementation:
- **Task SDK:** `AssetAny.__init__` accepts and validates `consume`. It is
included in `__eq__`, `__hash__` and `__repr__`, and preserved by `|`.
- **Serialization:** `consume` is written only when it is not `"all"`, so
existing Dags serialize byte-for-byte as before (no Dag version churn).
`SerializedAssetAny.as_expression()` includes it, and
`expand_composite_asset_expr` keeps extra keys on `any` nodes.
- **`AssetEvaluator`:**
- `uses_branch_consumption()` checks whether any `AssetAny` in the
condition opts in.
- `find_satisfied_keys()` returns the queued asset keys a run should
consume.
- **Scheduler:**
- `_is_asset_condition_met` is replaced by `_find_consumable_asset_ids`,
which returns whether the condition is met plus an optional set of asset ids.
- `_select_consumed_asset_events` filters by those ids only when they are
given, i.e. only for Dags that opt in.
- Since `_delete_consumed_asset_records` already deletes only the queue
rows whose events were consumed, the other branches' rows stay queued.
### Compatibility
- **Existing Dags:** no change. They take the exact previous code path, and
their serialized form is unchanged.
- **Database and API:** no migration and no change to the REST API.
- **Mixed versions:**
- A Dag using `consume="branch"` on a scheduler without this change runs
with the old behaviour; the option is silently ignored. The same happens after
a downgrade.
- airflow-core with this change and an older task-sdk is safe: the encoder
reads `consume` with a fallback, since an older `AssetAny` has no such
attribute.
- **UI:** the next-run assets endpoint does not yet expose `consume`. That
can be a follow-up if reviewers want the mode shown in the UI.
### Testing
- **Unit tests:**
- `find_satisfied_keys` and `uses_branch_consumption` in
`test_evaluation.py`.
- The serialization round-trip.
- SDK `AssetAny(consume=...)`.
- A scheduler test with `a`, `c` and `d` queued. With `"branch"`, `c` and
`d` are consumed and `a` stays queued; with `"all"`, all three are consumed.
This test fails without the scheduler change.
- **Manually with `airflow standalone`:** two Dags on `(a & b) | (c & d)`,
one with `"branch"` and one with `"all"`. I emitted `a`, then `c` and `d`, then
`b`.
- The `"branch"` Dag ran twice: first on c and d, then on a and b.
- The `"all"` Dag ran once, and `b` was left waiting alone.
- Only tested on SQLite so far.
Docs for the new parameter will follow once reviewers agree on the API shape.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Opus 5.5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]