jason810496 opened a new pull request, #75184: URL: https://github.com/apache/airflow/pull/75184
- closes: #51067 ## Why TypeScript, Go and Java tasks skip the configured `[core] xcom_backend`, because their XComs go from the supervisor straight to the API server. In a Python Dag with stub TaskHandler that uses `XComObjectStorageBackend`, a TypeScript task reads the `s3://...` reference that a Python task stored instead of the value, its own values stay in the metadata database, and clearing a try leaves the stored objects behind. ## How - The supervisor of a language SDK task runs the backend on the runtime's XCom requests (`SetXCom`, `GetXCom`, `GetXComSequenceItem`, `GetXComSequenceSlice`, `DeleteXCom`) and on the XComs it clears when a try starts. With the default `BaseXCom`, requests go to the API server unchanged. - The backend runs in a helper process, started per task the same way a Python task process is started (bare fork, or a new interpreter on macOS or with `[core] execute_tasks_new_python_interpreter`). So the long-lived worker process keeps none of the backend's state, such as a cached `S3FileSystem` that a Python task forked from it later could not use. - Runtimes get what the default backend would give them: the value is decoded with serde before `store_value`, and a loaded value is encoded with serde again. `skipmixin_key` bypasses the backend, as it does for Python tasks. - The backend looks up Connections and Variables through the supervisor's API client. - While a backend call runs, the supervisor keeps heartbeating, and stops the call when the task reaches its `execution_timeout` or the server stops the task. - A Go task now fails when its return value cannot be stored, as TypeScript and Java tasks already do. No SDK change is needed, since all three runtimes only talk to the supervisor. With this config, TypeScript, Go and Java tasks read and write offloaded XComs like Python tasks: ```ini [core] xcom_backend = airflow.providers.common.io.xcom.backend.XComObjectStorageBackend ``` ## Performance Default backend, mocked API, median of 3 alternating runs: | | base | this PR | |---|---|---| | `SetXCom` handler | 65.8 µs | 65.6 µs | | `GetXCom` handler | 67.7 µs | 67.1 µs | | `GetXComCount` handler (control) | 58.2 µs | 57.8 µs | | `_find_own_coordinate` | 0.645 µs | 0.641 µs | ## Verification - task-sdk unit tests, including a real helper process that looks up a Connection through the supervisor. - `go test ./...` and `go test -race ./...`. - The `xcom_object_storage` e2e mode now also runs `typescript_taskflow_example` and checks the objects in S3. It was not run locally (no Docker). --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
