AP24110011695 opened a new issue, #77346:
URL: https://github.com/apache/airflow/issues/77346

   ### Apache Airflow version
   3.1.x (main @ bee092f4, 2026-11-19)
   
   ### If "Airflow 2" (experimental)
   
   _No response_
   
   ### What happened
   
   With a custom XCom backend configured (`[core] xcom_backend`), XComs written 
by a trigger through the task-end event API bypass the backend, while XComs 
written by a running task go through it. The two write paths produce 
differently-shaped rows for the same key namespace:
   
   - **Task-runtime writes** go through the supervisor, which runs the backend 
in a helper process and stores an externalized reference in the XCom row 
(`XComBackendCall.store(...)` → the row holds the backend's returned reference).
   - **Trigger-written writes** (`Trigger._push_xcoms_if_necessary` → 
`XComModel.set_for_attempt(serialize=False)`, in 
`airflow-core/src/airflow/models/trigger.py`) store the raw value in the row. 
This was pinned deliberately by #76596, and the triggerer has no task context 
or backend access.
   
   `TaskSuccessEvent(xcoms=...)` / `TaskFailureEvent(xcoms=...)` 
(`airflow-core/src/airflow/triggers/base.py`) are documented public trigger 
API, so this path is reachable by user code.
   
   The consequence shows on reads. When a downstream task pulls such a key, the 
supervisor's get path resolves row values through the backend's load hook 
whenever a custom backend is configured (see #77136 for the current gate). For 
a trigger-written row the "row value" is the raw value itself — never 
externalized — so a backend that assumes reference-shaped input (the common 
case: fetching from external storage, parsing a pointer) either fails with 
`XCOM_BACKEND_ERROR` or silently mis-resolves. Backends that fall back to the 
serde path on foreign values happen to work, which is why this has not surfaced 
widely yet.
   
   ### What you think should happen instead
   
   The two write paths should produce consistent rows for the same key, or 
reads should distinguish provenance. Options we can see:
   
   1. Route trigger-written xcoms through a backend-aware path (the triggerer 
has no backend instance or task-instance coordinate today, so this needs 
plumbing — and the helper process model assumes a live task).
   2. Mark rows with their provenance (backend-externalized vs inline) so reads 
can skip the load hook for inline rows.
   3. Document that custom backends must tolerate raw JSON values in `load` 
(the base-class fallback does; strict backends do not).
   
   We are not sure which of these the project prefers — 1 is the semantically 
cleanest but the heaviest, 2 is targeted, 3 acknowledges the current behavior.
   
   ### How to reproduce
   
   1. Configure a custom XCom backend whose `load`/`load_value` assumes 
externalized references (e.g. raises on non-reference values — any backend that 
fetches from remote storage).
   2. Run a Dag whose task defers; its trigger yields 
`TaskSuccessEvent(xcoms={"k": "v"})`.
   3. A downstream task pulls `k` through the Task SDK.
   4. The supervisor routes the row through the backend's load hook with the 
raw value `v`, and the backend fails (or mis-resolves) — while the same pull of 
a task-written key of the same shape resolves correctly.
   
   ### Operating system
   
   macOS / Linux (main branch code path; observed while reviewing #77136)
   
   ### Versions of Apache Airflow Providers
   
   n/a — core task-sdk / triggerer paths
   
   ### Anything else?
   
   Related but distinct: #76212 (transaction timing on the same write path), 
#76596 (the double-encoding fixed there, which also pinned raw storage), #77136 
(the read-gate key exclusions this issue complements). No existing issue tracks 
the backend bypass itself.
   
   AI disclosure: this report was compiled with AI assistance from the code 
paths cited above; the flow is traceable from the referenced sources.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to