Andrushika opened a new issue, #74400:
URL: https://github.com/apache/airflow/issues/74400
### Description
`[workers] state_store_backend` lets a deployment keep large task and asset
state store values in external storage. In a Python task, the task process
calls the backend. On `set`, it stores the value and writes only a marker like
`{"__airflow_state_ref__": "<ref>"}` to the metadata DB. On `get`, it reads the
value back from that ref (`task-sdk/src/airflow/sdk/execution_time/context.py`).
Java and Go tasks do not run this code. They send `SetTaskStateStore` /
`GetTaskStateStore` to the supervisor, and the supervisor stores and returns
the value as-is. The Java and Go docs list this as a limitation, and the
TypeScript SDK in #73618 has the same note.
### Use case/motivation
With a backend configured, a Dag that mixes Python and Java or Go tasks
behaves like this:
- A Java or Go task reads a key that a Python task wrote, and gets the
marker dict instead of the value.
- A Java or Go task writes a large value, and it goes to the metadata DB
instead of the external storage.
The backend is a Python class, so implementing it again in each SDK looks
hard. The solution could be like letting the supervisor calls the backend when
it handles the state store messages. (Like how remote log backend works)
Then every language SDK gets it without its own code. The backend would run
in the supervisor process instead of the task process, both on the same worker.
### Related issues
related: #73464, #73420, #73618, #74217
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
