rmittal46 opened a new pull request, #77263:
URL: https://github.com/apache/airflow/pull/77263

   closes: #77256
   
   ### Problem
   
   When a Dag is scheduled on an `AssetAny` condition and the condition is met, 
the scheduler consumes every queued asset event for the Dag, including events 
belonging to branches that were not satisfied.
   
   For a condition like `(a & b) | (c & d)`:
   
   1. `a` is updated → nothing runs (correct).
   2. `c` and `d` are updated → the Dag runs, and the pending `a` event is 
consumed too.
   3. `b` is updated → nothing runs, because `a` was already consumed.
   
   So the `a & b` branch can effectively never trigger if the other branch 
completes first.
   
   ### Change
   
   `AssetAny` gets an opt-in `consume` parameter:
   
   ```python
   schedule = AssetAny(
       AssetAll(asset_a, asset_b),
       AssetAll(asset_c, asset_d),
       consume="branch",
   )
   ```
   
   - `consume="all"` (default): current behaviour, unchanged.
   - `consume="branch"`: a triggered run consumes only the events of the 
satisfied branch(es). Events of other branches stay queued for a later run.
   
   Implementation:
   
   - **Task SDK:** `AssetAny.__init__` accepts and validates `consume`. It is 
included in `__eq__`, `__hash__` and `__repr__`, and preserved by `|`.
   - **Serialization:** `consume` is written only when it is not `"all"`, so 
existing Dags serialize byte-for-byte as before (no Dag version churn). 
`SerializedAssetAny.as_expression()` includes it, and 
`expand_composite_asset_expr` keeps extra keys on `any` nodes.
   - **`AssetEvaluator`:**
     - `uses_branch_consumption()` checks whether any `AssetAny` in the 
condition opts in.
     - `find_satisfied_keys()` returns the queued asset keys a run should 
consume.
   - **Scheduler:**
     - `_is_asset_condition_met` is replaced by `_find_consumable_asset_ids`, 
which returns whether the condition is met plus an optional set of asset ids.
     - `_select_consumed_asset_events` filters by those ids only when they are 
given, i.e. only for Dags that opt in.
     - Since `_delete_consumed_asset_records` already deletes only the queue 
rows whose events were consumed, the other branches' rows stay queued.
   
   ### Compatibility
   
   - **Existing Dags:** no change. They take the exact previous code path, and 
their serialized form is unchanged.
   - **Database and API:** no migration and no change to the REST API.
   - **Mixed versions:**
     - A Dag using `consume="branch"` on a scheduler without this change runs 
with the old behaviour; the option is silently ignored. The same happens after 
a downgrade.
     - airflow-core with this change and an older task-sdk is safe: the encoder 
reads `consume` with a fallback, since an older `AssetAny` has no such 
attribute.
   - **UI:** the next-run assets endpoint does not yet expose `consume`. That 
can be a follow-up if reviewers want the mode shown in the UI.
   
   ### Testing
   
   - **Unit tests:**
     - `find_satisfied_keys` and `uses_branch_consumption` in 
`test_evaluation.py`.
     - The serialization round-trip.
     - SDK `AssetAny(consume=...)`.
     - A scheduler test with `a`, `c` and `d` queued. With `"branch"`, `c` and 
`d` are consumed and `a` stays queued; with `"all"`, all three are consumed. 
This test fails without the scheduler change.
   - **Manually with `airflow standalone`:** two Dags on `(a & b) | (c & d)`, 
one with `"branch"` and one with `"all"`. I emitted `a`, then `c` and `d`, then 
`b`.
     - The `"branch"` Dag ran twice: first on c and d, then on a and b.
     - The `"all"` Dag ran once, and `b` was left waiting alone.
   - Only tested on SQLite so far.
   
   Docs for the new parameter will follow once reviewers agree on the API shape.
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes (please specify the tool below)
   
   Generated-by: Claude Code (Opus 5.5) following [the 
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] Allow AssetAny to consume... rmittal46 (via GitHub) via commits

Reply via email to