github-actions[bot] opened a new pull request, #71821: URL: https://github.com/apache/airflow/pull/71821
The scheduler kept every Dag version it deserialized in a mapping that never evicted, so a long-running scheduler grew with the number of versions it had ever seen until it was restarted or OOM killed. Deployments that redeploy Dags frequently accumulate versions fastest and hit this soonest. A least-recently-used cap is the only thing that bounds this outright. An idle timeout would not: the scheduler re-checks an entry on each lookup, which re-arms its expiry, so a timeout reclaims a version only once its runs finish and it stops being requested, leaving memory a function of the concurrently active set rather than a fixed ceiling. Deliberately not configurable here, so the fix stays small enough to cherry-pick. Cache activity currently reports under the existing api_server.dag_bag.* metrics; a scheduler-specific namespace, along with configuration, follows separately. (cherry picked from commit 29dd99d0034c3cff0fe37882f6b210ffca902d29) Co-authored-by: Jason(Zhe-You) Liu <[email protected]> closes: #69001 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
