dheerajturaga opened a new pull request, #64326:
URL: https://github.com/apache/airflow/pull/64326
DBDagBag._dags is an unbounded in-memory cache causing steady memory
growth in the API server.
DBDagBag was designed for the scheduler, which works with a bounded set
of currently-active DAG versions. As an API server singleton, it is
exposed to
the full history of DAG versions in the database with no bound on how
many it will cache
Replace the plain dict in DBDagBag._dags with a bounded OrderedDict-based
LRU cache. In long-running API server processes, every unique
dag_version_id
accessed is inserted and never evicted, causing unbounded RSS growth
(observed:
9.4 GiB after 7 days with ~70k DAG versions in DB).
The cache is now capped at 512 entries by default (configurable via
core.max_dag_version_cache_size). Cache hits promote the entry to MRU so
frequently-accessed versions are retained over stale historical ones.
<img width="1408" height="227" alt="image"
src="https://github.com/user-attachments/assets/8a235479-7d6d-4b17-b7f1-8c4f58354408";
/>
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
