dheerajturaga opened a new pull request, #66450:
URL: https://github.com/apache/airflow/pull/66450
LogStreamAccumulator spills the in-flight log stream to a
NamedTemporaryFile(delete=False) in /tmp once a task's output exceeds
5000 lines. Cleanup was hooked only into the stream generator's
finally block, so the file leaked whenever the caller did not iterate
the generator to completion: client disconnect on the NDJSON streaming
endpoint, exception during list() materialisation on the JSON path, or
the caller using only total_lines and never accessing .stream.
A single production host accumulated ~39 GB of orphaned files from this
code path, with individual files up to 2.5 GB on tasks emitting
~10M log lines.
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
ClaudeCode Opus 4.7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
