jingi723 opened a new pull request, #74649: URL: https://github.com/apache/airflow/pull/74649
Async task and asset state-store reads currently call synchronous reference deserialization after awaiting the supervisor response. With an object-storage backend, the read can stall other coroutines on the event loop. Add non-abstract async reference-read hooks and use them from task and asset `aget`. Their defaults call the existing synchronous hooks on the configured backend instance and caller thread, preserving compatibility with thread-bound custom backends. For an already initialized standard s3fs filesystem, the common.io backend reads plain and gzip objects through the filesystem's existing loop. Unsupported filesystem configurations retain the synchronous path. **Design feedback requested:** The narrow S3 reader uses the private `s3fs.core._inner_fetch` helper to preserve conditional reads (`If-Match`/`FileExpired`) and response-body cleanup. Is that acceptable here, or should a public async-read abstraction in the filesystem layer come first? A simpler `_cat_file` path lost the existing object-replacement behavior in a controlled test. This is a read-only partial improvement related to #72129. It does not change `aset`, synchronous callers, blocking fallbacks for custom backends, or cold filesystem/credential initialization. It does not meet the issue's full read/write non-blocking acceptance criteria. Validation on current `main` (`a52e9f96`): - Three affected test modules: 294 passed on Python 3.12 in Breeze. - Local S3-compatible HTTP contract tests: 44 passed, 6 earlier write-experiment cases deselected. They cover task/asset reads, conditional requests, lineage on errors, cancellation cleanup, and loop progress. This is not live AWS validation. - Changed-file pre-commit checks passed; provider source-file mypy passed in Breeze. The full manual provider mypy hook could not start because the local `breeze` shim is absent from PATH. --- ##### Was generative AI tooling used to co-author this PR? - [x] Yes — Codex (GPT-6) Generated-by: Codex (GPT-6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
