bujjibabukatta opened a new pull request, #71064: URL: https://github.com/apache/airflow/pull/71064
**Summary** WASB task logs are rewritten from scratch on every upload -- the whole existing blob is downloaded, the new segment appended in memory, and the full result re-uploaded. For a long-running task this means O(N^2) traffic across N log writes instead of O(N). **Root cause** WasbRemoteLogIO.write() always calls hook.load_string(..., overwrite=True), which replaces the entire blob object regardless of how much of it is actually new. **Fix** Added an opt-in write_mode="append_blob" that uses Azure's native AppendBlob API (append_block() with a position guard) to append only the new segment, instead of rewriting the whole object. Falls back to the existing block-blob rewrite path for any blob already written as a block blob, since Azure doesn't support converting a blob's type in place. Segments over the 4 MiB per-block limit are chunked automatically, and a concurrent-writer position mismatch fails safely rather than writing at the wrong offset. closes: #70867 Was generative AI tooling used ? - [X] Yes - Claude Generated-by: Claude following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
