potiuk opened a new pull request, #74282: URL: https://github.com/apache/airflow/pull/74282
Follow-up to #73836, which gave each git worktree its own Breeze Compose project and sweeps resources of deleted worktrees before Docker-backed commands. This builds on its label-based ownership to close the remaining gaps: - **Resources are released soon after a worktree is deleted.** The first Docker-backed command in an isolated worktree starts a small detached watcher. It checks the directory every 30s and removes the worktree's containers, volumes and networks once the directory is gone. It uses only the standard library, imports everything at start-up (deleting the worktree can delete the venv it runs from), allows one instance per worktree via a file lock, and exits after an hour with no containers. It is not started in CI or with `--dry-run`. Until now, CPU and memory were only freed when some later Breeze command ran the sweep. - **A shared Docker daemon no longer loses another machine's resources.** Resources are labelled `org.apache.airflow.breeze.host`, and only the host that created them treats them as stale. Resources without the label (created before this change) keep the old local-path check. - **Same-named worktrees no longer collide.** The default project is `breeze-<dirname>-<6-char path hash>`. - **Opt-out.** `breeze setup config --no-worktree-isolation` (stored in the main checkout's `.build`, so it applies to every worktree) or `BREEZE_WORKTREE_ISOLATION=true|false`. With isolation off, worktrees share the `breeze` project and their resources aren't tied to the worktree. - **`breeze down` lists other worktrees** that still hold containers or volumes. - **ADR 0019** records the reasoning for per-worktree isolation as a whole: label ownership, naming, what stays shared, cleanup triggers, safety rules, opt-out. Note: the path hash renames existing worktree projects, so a worktree that already had a database volume starts with a fresh one once. `breeze down` in that worktree still removes the old project, because it's selected by its worktree label. A second PR will scope the names that are still fixed: the `breeze-docs`/`breeze-db`/`breeze-constraints*`/`breeze-providers*` projects, the airflow-ctl and task-sdk test conftest projects, fixed `container_name`s, kind cluster names and the pycache volume. Checks run locally: - `uv run pytest` in `dev/breeze`: 1297 passed. Two port-binding tests in `test_run_utils.py` fail only because my local sandbox blocks binding. - `prek run --from-ref main --stage pre-commit`, `prek run mypy-dev --all-files`: passed. - `breeze setup regenerate-command-images --check-only` and `check-all-params-in-groups`: passed. - The watcher was tested as a real standalone `python -I` process against a fake `docker` executable. It was not run against a live Docker daemon. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5.5) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
