The GitHub Actions job "Tests (AMD)" on airflow.git/fix/celery-executor-app-cache-memory-leak has failed. Run started by GitHub user arkadiuszbach (triggered by arkadiuszbach).
Head commit for run: e51915a36601e596d5383656d617cbdfd1dfe9e4 / bach.ab <[email protected]> fix(celery): cache Celery apps to stop per-send memory leak create_celery_app() rebuilt a brand-new Celery app on every workload send. When sends run in-process (a single task per heartbeat, or sync_parallelism == 1) this happens in the long-lived scheduler/executor process, so each rebuilt app — and its broker/result-backend connection pools — accumulates and is never released, leaking memory over time. PR #60675 assumed sends always run in throwaway ProcessPoolExecutor subprocesses, where a fresh app per send is harmless. The in-process hot path in _send_workloads_to_celery() breaks that assumption. Cache apps by their resolved (team-aware) name so each distinct app is built once per process and reused. Config is static per app-name per process, so this is safe for both call sites: subprocesses still build their own app once, and the in-process path stops recreating it on every send. Add regression tests for app caching/reuse and per-team app isolation. Report URL: https://github.com/apache/airflow/actions/runs/28036006123 With regards, GitHub Actions via GitBox --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
