rmittal46 opened a new issue, #77256:
URL: https://github.com/apache/airflow/issues/77256
### Description
When a Dag is scheduled on a conditional asset expression that nests
`AssetAll` inside `AssetAny`, e.g.
```python
from airflow.sdk import DAG, Asset, AssetAll, AssetAny
with DAG(
dag_id="consumer",
schedule=AssetAny(
AssetAll(Asset("a"), Asset("b")),
AssetAll(Asset("c"), Asset("d")),
),
):
...
```
the scheduler clears **all** `asset_dag_run_queue` rows for the Dag as soon
as **any** branch is satisfied. Progress towards the branch that did not fire
is lost.
**Steps to reproduce**
1. Emit an event for `a` (branch `a & b` is now half satisfied; queue = {a}).
2. Emit events for `c` and `d` → branch `c & d` is satisfied, a Dag run is
created.
3. The queue is now empty. The pending `a` is gone (it is attached to the
run created in step 2 as a triggering event).
4. Emit an event for `b` → nothing happens. The `a & b` branch needs a
**new** `a` event, even though `a` was updated after the last run that `a & b`
actually satisfied.
**Current behaviour:** queued events are cleared per Dag, not per condition
branch.
**Expected / desired behaviour:** an option to track queued events per
branch of the condition, so that firing one `AssetAny` branch consumes only the
events of that branch. Unrelated, partially satisfied `AssetAll` branches keep
their queued events and can complete later.
### Use case/motivation
Data platforms commonly model "rebuild X when (source A **and** source B)
have both refreshed, **or** when (source C **and** source D) have both
refreshed". Each `AND` group represents a consistent pair of inputs that must
land together.
With the current behaviour, the `AND` semantics become unreliable as soon as
another branch fires. The only workaround today is to create an intermediate
"gate" Dag per `AssetAll` branch, each publishing its own asset, and schedule
the consumer on `AssetAny(gate_1, gate_2)`. That works, because each Dag has
its own queue, but it:
- adds extra Dags purely for scheduling bookkeeping
- makes the asset graph harder to read
- has to be applied by every user who discovers the behaviour (often only in
production)
### Possible approaches
1. **Per-branch consumption (opt-in):** when a branch evaluates true, delete
only the queue rows for assets referenced by that branch. Rows for other assets
remain. Could be gated by a Dag-level flag or a parameter on `AssetAny` (e.g.
`AssetAny(..., consume="branch")`) to keep backward compatibility.
2. **Documentation:** at minimum, document clearly in the conditional asset
scheduling docs that the whole queue is cleared when any condition is
satisfied, and describe the gate-Dag workaround.
Open questions for maintainers:
- With per-branch consumption, should events consumed by a branch still
appear as triggering events on runs created by other branches? Probably not, to
keep the lineage accurate.
- How should this interact with `max_active_runs` and event coalescing when
several branches become satisfied in the same scheduler loop?
### Related issues
- #56050 Most recent Asset events ignored when max_active_runs = 1
- #55956 Asset event scheduling - introduce max_asset_events parameter
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]