dabla opened a new pull request, #76777:
URL: https://github.com/apache/airflow/pull/76777
A value that `XComObjectStorageBackend` writes to object storage still
travels through the task that pulls it: `xcom_pull` opens the file,
deserializes it and hands the Python object over. When the next step is a tool
that can read the file itself, for example a query engine reading a large JSON
extract straight from object storage, the task only needs to know where the
file is. Loading and re-serializing the value in between costs memory and time
for nothing.
This adds:
- `XComObjectStorageBackend.get_path(*, key, dag_id, task_id, run_id,
map_index=None)`, which returns the `ObjectStoragePath` of a stored value
without reading it, or `None` when there is no such XCom or the value is stored
in the database (smaller than `xcom_objectstorage_threshold`). It reads the
database value through `BaseXCom.get_one`, whose own deserialization stops at
the stored path.
- A provider plugin, `common_io`, with an `xcom_path` macro for templates.
Unlike the method, the macro raises instead of returning `None`, since a `None`
rendered into a query fails later with a less helpful error: `TypeError` when
another XCom backend is configured, `LookupError` when there is no such XCom,
`ValueError` when the value is stored in the database. `protocol=False` returns
the path without its scheme, for a tool that is given the file system
separately.
Example, an SQL operator running on DuckDB:
```sql
SELECT *
FROM read_json('{{ macros.common_io.xcom_path(ti, "extract_invoices") }}',
format = 'array')
```
```python
@task
def extract_invoices() -> list[dict]:
return fetch_all_invoices() # large: written to object storage by the
backend
flatten = SQLExecuteQueryOperator(task_id="flatten_invoices",
conn_id="duckdb_default", sql="flatten_invoices.sql")
extract_invoices() >> flatten
```
We have run the same pattern in production through a custom backend subclass
for a while; having it upstream lets us drop that subclass, whose copies of
`get_one`/`get_all` no longer match Airflow 3.4's signatures.
The plugin follows the same pattern as the Hive provider's macros
(`macros.hive.max_partition`). The documentation in `xcom_backend.rst`
describes the use case, the threshold rule and the errors. Looking up the path
requires Airflow 3; the provider's Airflow 2 support is unchanged.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Opus 5.5
Generated-by: Claude Opus 5.5 following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
