harel-sht opened a new issue, #72838:
URL: https://github.com/apache/airflow/issues/72838
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.1.7
### What happened and how to reproduce it?
**Issue Description**
On Airflow 3.1.7 with `LocalExecutor` and a versioned `GitDagBundle`, the
scheduler job dies every `dag_processor.stale_bundle_cleanup_interval` seconds
(default 1800). `BundleUsageTrackingManager._remove_stale_bundle` calls
`shutil.rmtree(bundle_version_path)` and only catches `BlockingIOError`. If the
`versions/<sha>` directory is already gone but the tracking/lock file under
`_tracking/<bundle>/<sha>` remains, `rmtree` raises `FileNotFoundError`. That
exception is invoked from `SchedulerJobRunner._run_scheduler_loop` via
`timers.run()`, is logged as `Exception when executing
SchedulerJob._run_scheduler_loop`, and kills the scheduler loop.
After that, `LocalExecutor.end()` waits on in-flight children with an
unbounded `join()`. Heartbeats stop, the scheduler health endpoint returns 503,
and Docker `restart: always` does not recycle the container because the process
is still alive. We saw 43 failed `SchedulerJob` rows in 24 hours from this
single orphan SHA.
The **dag-processor** already wraps the same call:
```python
# airflow/dag_processing/manager.py (_cleanup_stale_bundle_versions)
try:
BundleUsageTrackingManager().remove_stale_bundle_versions()
except Exception:
self.log.exception("Error removing stale bundle versions")
```
The scheduler timer does not.
Still present on `main` (`_remove_stale_bundle` excepts only
`BlockingIOError`). Confirmed `_filter_for_min_versions` is `sorted(found, dt
desc)[min_versions:]` and reads `conf.getint` at call time (default
`stale_bundle_cleanup_min_versions=10`).
**How the orphan tracking file is created**
`_remove_stale_bundle` deletes the tracking/lock file **only after**
`rmtree` succeeds:
```python
with open(info.lock_file_path, "a") as f:
flock(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
shutil.rmtree(bundle_version_path)
os.remove(info.lock_file_path)
except BlockingIOError:
...
```
Two processes share the bundle storage (scheduler timer + dag-processor):
1. Both list tracking files and pick the same candidate past `min_versions`.
2. A `rmtree`s the version dir and then `os.remove`s the lock.
3. B's `open(lock, "a")` recreates the lock file, `flock` succeeds, `rmtree`
raises `FileNotFoundError`, the lock file stays.
4. Every subsequent cleanup tick retries the same delete.
Unpaired `rm -rf versions/<sha>` without the tracking file produces the same
state.
**Steps to reproduce**
1. Run 3.1.7 with `LocalExecutor` and a versioned `GitDagBundle`. Scheduler
and dag-processor must share `dag_bundle_storage_path`.
2. Keep default `stale_bundle_cleanup_min_versions=10`.
3. Create 11 tracking files under `_tracking/<bundle>/` and only 10
directories under `versions/` — the extra tracking file points at a SHA whose
version dir is missing (or let the TOCTOU above mint one).
4. Wait `stale_bundle_cleanup_interval` (default 30 minutes), or temporarily
lower it.
5. Scheduler logs `Exception when executing
SchedulerJob._run_scheduler_loop` and the job dies. Dag-processor logs `Error
removing stale bundle versions` and continues.
### What you think should happen instead?
`_remove_stale_bundle` should treat a missing version directory (and a
missing lock file) as "already gone": catch `FileNotFoundError` / `OSError`
with errno ENOENT, remove the leftover tracking file if it still exists, and
return. The scheduler timer should not be able to take down
`_run_scheduler_loop` for a cleanup race the dag-processor already swallows.
Alternatively, wrap `bundle_cleanup_mgr.remove_stale_bundle_versions` in the
scheduler the same way `DagFileProcessorManager._cleanup_stale_bundle_versions`
already wraps it (`except Exception: log`).
```
2026-09-07T20:40:24.213855Z [info] removing stale bundle.
bundle_name=amiio-repo bundle_version=1443d9ca4dfdeb69b68b6bc6ea0cc710abd34738
bundle_path=/opt/airflow/dag-bundles/amiio-repo/versions/1443d9ca4dfdeb69b68b6bc6ea0cc710abd34738
lock_file=/opt/airflow/dag-bundles/_tracking/amiio-repo/1443d9ca4dfdeb69b68b6bc6ea0cc710abd34738
loc=base.py:158
2026-09-07T20:40:24.214757Z [error] Exception when executing
SchedulerJob._run_scheduler_loop loc=scheduler_job_runner.py:1086
Traceback (most recent call last):
File ".../airflow/jobs/scheduler_job_runner.py", line 1082, in _execute
self._run_scheduler_loop()
File ".../airflow/jobs/scheduler_job_runner.py", line 1412, in
_run_scheduler_loop
next_event = timers.run(blocking=False)
File ".../airflow/dag_processing/bundles/base.py", line 229, in
remove_stale_bundle_versions
self._remove_stale_bundle_versions_for_bundle(bundle_name=bundle.name)
File ".../airflow/dag_processing/bundles/base.py", line 210, in
_remove_stale_bundle_versions_for_bundle
self._remove_stale_bundle(bundle_name=bundle_name, info=info)
File ".../airflow/dag_processing/bundles/base.py", line 172, in
_remove_stale_bundle
shutil.rmtree(bundle_version_path)
File "/usr/python/lib/python3.12/shutil.py", line 759, in rmtree
_rmtree_safe_fd(stack, onexc)
File "/usr/python/lib/python3.12/shutil.py", line 669, in _rmtree_safe_fd
FileNotFoundError: [Errno 2] No such file or directory:
PosixPath('/opt/airflow/dag-bundles/amiio-repo/versions/1443d9ca4dfdeb69b68b6bc6ea0cc710abd34738')
```
### Operating System
Ubuntu 24.04 (GCE VM), official Airflow Docker image, Python 3.12
### Deployment
Docker-Compose
### Versions of Apache Airflow Providers
apache-airflow-providers-git (GitDagBundle). Core bug is in
`airflow.dag_processing.bundles.base`, not the git provider.
### Official Helm Chart version
Not Applicable
### Kubernetes Version
Not Applicable
### Helm Chart configuration
Not Applicable
### Docker Image customizations
Official `apache/airflow:3.1.7` image plus Python deps. Not required to
reproduce — stock 3.1.7 source of `_remove_stale_bundle` is enough.
### Anything else?
- Default `min_versions=10`. With 11 tracking files and 10 version dirs,
`_filter_for_min_versions` returns exactly the orphan, which is why `rmtree`
runs at all.
- Workaround: raise
`AIRFLOW__DAG_PROCESSOR__STALE_BUNDLE_CLEANUP_MIN_VERSIONS` so the candidate
slice stays empty. Do not set `stale_bundle_cleanup_interval=0` as the primary
workaround (3.1.7 does guard `if check_interval > 0` on the scheduler, but
starving cleanup is the safer knob).
- Related but different: #62512 (Celery spawn / local function), #47567
(slash in bundle name / lock path).
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
