jason810496 opened a new pull request, #72131: URL: https://github.com/apache/airflow/pull/72131
- related: #68232 - related: #68283 ## Why `StateStoreObjectStorageBackend` raised `NotImplementedError` from all four of its async methods, so the async state store accessors added in #68232 fail against the shipped object storage backend. `adelete`/`aclear` clear the DB reference before calling the backend, so the call raised only after the reference was gone, orphaning the stored object. ## What - Implement `aget`, `aset`, `adelete`, `aclear` by offloading the existing sync methods with `asyncio.to_thread`. - Add tests for all four, including a parametrized check that the blocking work does not run on the loop thread. - Show the offload pattern in the custom backend guide, which requires the four async methods but only demonstrated sync `get()`. ## Why `asyncio.to_thread` and not fsspec's async API fsspec does have one, but it is not reachable from where this backend sits: - The coroutines live on `AsyncFileSystem` and are all private (`_cat_file`, `_pipe_file`, `_rm`, `_glob`). `open_async` is the only public one. - They exist only on async implementations. The base path is user configured, and `LocalFileSystem` sets `async_impl = False`, so a native path still needs an `isinstance` branch and a sync fallback. - `ObjectStoragePath` exposes no coroutines at all, so going native means bypassing it and re-implementing the path building and compression that `_write_to_object_storage`/`_read_from_object_storage` already handle. - fsspec's sync methods already dispatch those coroutines onto fsspec's own IO loop via `run_coroutine_threadsafe` and block the caller. `asyncio.to_thread` only moves that block off Airflow's loop, so this follows fsspec's own concurrency model rather than working around it. --- ##### Was generative AI tooling used to co-author this PR? - [x] Yes, with help of Claude Code Opus 5 following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
