GitHub user potiuk edited a comment on the discussion: Airflow Triggerer memory
leak
Two things are likely contributing here, and they're separate fixes.
**1. pathlib `sys.intern` memory growth — already fixed, ships in 3.2.2**
CPython's `pathlib.PurePath._parse_path()` calls `sys.intern()` on every
path component (CPython issue 119518). Interned strings live for the
lifetime of the process, so a long-running triggerer that opens a new
log file per trigger via `init_log_file()` keeps accumulating interned
path components and never gives the memory back. CPython 3.14 removed
the `sys.intern` call upstream; #65706 backports the fix as a
`_PatchedPath` used inside `init_log_folder` / `init_log_file`.
That fix is on `main` and was already cherry-picked into `v3-2-test` as
`34de075a1b`, so you'll pick it up when 3.2.2 is released. It's not in
3.2.0. The "slow continuous growth that never drops" shape in your
screenshot fits this pattern — every trigger contributes a bit, and
nothing the process does can reclaim it.
**2. File handle leak when remote log upload fails — fix proposed in #66675**
Separate from the pathlib issue, @arkadiuszbach is right that
`structlog.BytesLogger` was contributing in some setups. In
`triggerer_job_runner.py`, when a trigger finishes the supervisor does:
```python
if factory := self.logger_cache.pop(id, None):
factory.upload_to_remote()
factory.close()
```
If `upload_to_remote()` raises (e.g. transient S3/GCS error, throttling)
the factory has already been popped, so `close()` is never called — the
underlying `BufferedWriter` (8 KiB buffer + open fd) leaks for every
failed upload, and the exception escapes into `handle_requests`. #66675
wraps the cleanup in `try/except/finally` so the fd is always closed and
the upload failure is logged instead of propagating.
This one only bites you if remote logging is enabled and uploads
occasionally fail. If you have remote logging off, you can ignore (2)
and the pathlib fix alone should account for what you're seeing.
**What to do**
- Wait for 3.2.2 (or test `v3-2-test`) — that picks up #65706 and should
flatten the slope you're plotting.
- If you can run the memray trace @wjddn279 set up (#65994 + #65996),
please attach a flamegraph — that confirms whether what's left is
string interning, leftover trigger log fds, or something else.
---
Drafted-by: Claude Opus 4.7 (1M context); reviewed by @potiuk before posting
GitHub link:
https://github.com/apache/airflow/discussions/65985#discussioncomment-16872869
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
