hussein-awala opened a new pull request, #77198:
URL: https://github.com/apache/airflow/pull/77198

   ## Why
   
   The HITL review plugin reads the review session and the agent outputs 
straight from the XCom table (`row.value`). A custom XCom backend that keeps 
values outside the metadata database stores only a reference there. For 
example, with `XComObjectStorageBackend` from `common.io` and 
`xcom_objectstorage_threshold = 0`, every worker-written review XCom is a path 
such as `s3://.../xcom-store/...` or `file:///...`. The plugin then fails to 
parse the session (`AgentSessionData.model_validate` on a string), so the 
review tab and the `find`, `feedback`, `approve` and `reject` endpoints fail. 
Workers are unaffected because they read through the backend.
   
   ## How
   
   - `_read_xcom` and `_read_xcom_by_prefix` pass each row to the configured 
backend's `deserialize_value`, which is how workers read these values. 
`_read_xcom_by_prefix` now selects whole XCom rows instead of the key and value 
columns. The Airflow 3.4 region scoping (`region_id`, 
`include_node_regions=False`) is unchanged.
   - `XCom` comes from `common.compat.sdk` (`airflow.sdk.execution_time.xcom`, 
which resolves `[core] xcom_backend`), available on every Airflow version the 
plugin loads on (3.1+) and in the minimum `common.compat` the provider requires.
   - With the default backend nothing changes: the stored values are plain JSON 
(strings and `model_dump(mode="json")` dicts), which 
`BaseXCom.deserialize_value` returns unchanged. The existing tests, including 
the JSON-looking string outputs covered after #70070, still pass.
   - Plugin writes stay native. The plugin's own entries (human action, 
feedback, updated session) are written to the database as before, and 
`XComObjectStorageBackend.deserialize_value` returns a value that is not a 
reference unchanged, both on the worker and in the plugin. Writing through the 
backend would also need serialization and write access to the backend's storage 
from the API server, and it is not needed to fix the bug.
   - Docs: the HITL review page now says the plugin reads review XComs through 
the configured backend, so such a backend must also be installed and configured 
on the API server with access to its storage. One sentence in the observability 
page that said the API server never runs a backend's `deserialize_value` now 
refers to the public XCom endpoint only.
   
   Note: this runs the configured backend's `deserialize_value` on the API 
server for these review XComs only. That is needed because the plugin has to 
parse the session, not only display it. The public XCom endpoint keeps 
returning what is stored in the database.
   
   ## Tests
   
   New `TestReferenceStoringXComBackend` in `test_hitl_review.py` patches the 
plugin's `XCom` with a stub `BaseXCom` subclass that resolves references from 
an in-memory store, like object-storage backends do. The session and the latest 
output are stored as references, an earlier output and human feedback are 
stored natively. It covers `find` and the `feedback`, `approve` and `reject` 
actions, whose responses also read back the session the plugin has just 
rewritten natively. All four tests fail on `main` with the `AgentSessionData` 
validation error.
   
   - `uv run --project providers/common/ai pytest 
providers/common/ai/tests/unit/common/ai/plugins/` (main sources): 113 passed, 
1 skipped
   - `breeze shell --use-airflow-version 3.3.2 --mount-sources 
providers-and-tests`, `test_hitl_review.py` and `test_hitl_review_compat.py`: 
78 passed, 22 skipped (3.4-only tests); this covers the pre-3.4 query path
   - Same on `--use-airflow-version 3.1.8`, the oldest version the plugin loads 
on: 68 passed, 32 skipped
   - `uv run --project providers/common/ai pytest 
providers/common/ai/tests/unit`: 5019 passed, 45 skipped, 2 failed. The 2 
failures are `sandbox/test_base.py` list-directory tests that need GNU `find 
-printf`, which the macOS host lacks; they are unrelated to this change
   - `prek run --stage pre-commit --files <changed files>`: all passed
   - `prek run --stage manual --skip compile-ui-assets-dev --skip 
view-skill-eval --skip run-skill-eval-codex --from-ref upstream/main` (includes 
`mypy-providers`): all passed
   - `breeze build-docs common.ai`: successful
   
   Not run locally: the full provider suites `breeze verify` lists for 
dependent providers (amazon, google, standard and others); CI covers them.
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes — Claude Code (Opus 5.5)
   
   Generated-by: Claude Code (Opus 5.5) following [the 
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to