potiuk opened a new pull request, #65549:
URL: https://github.com/apache/airflow/pull/65549
…icated .build/ venvs (#65492)
* Isolate mypy prek hooks, cover all non-provider dirs, and clean up type
errors
Each non-provider mypy prek hook now builds and caches its own virtualenv at
.build/mypy-venvs/<hook>/ and its own mypy cache at .build/mypy-caches/<hook>/.
UV_PROJECT_ENVIRONMENT redirects uv away from the project's .venv, so running
the hook never mutates a contributor's regular development environment while
still matching CI's frozen dependency set. Mypy runs with
--follow-imports=silent so each hook only reports errors for files it owns;
transitive code is covered by its own hook and different venvs no longer
produce divergent results on shared code.
Adds mypy hooks for the non-provider directories that were previously
uncovered: airflow-ctl-tests, helm-tests, airflow-e2e-tests,
task-sdk-integration-tests, docker-tests, kubernetes-tests, and shared. The
mypy-shared hook iterates every shared/<dist> workspace distribution and builds
a separate venv + cache per distribution so each shared library is type-checked
against its own dependency set.
breeze down --cleanup-mypy-cache additionally removes .build/mypy-venvs/ and
.build/mypy-caches/ so all per-hook state is wiped alongside the existing
.mypy_cache and mypy-cache-volume.
Also fixes pre-existing type errors surfaced by the newly added and
cleaned-up checks: platform-specific ignores for Linux-only os.posix_fadvise in
the shared logging helper, narrower types and type: ignore where appropriate in
shared configuration/observability/ timezones/secrets_backend/secrets_masker,
Liskov override markers on the AirflowConfigParser subclass methods, and small
correctness fixes in dev/breeze and the docker-tests / kubernetes-tests helpers
so the full non-provider mypy suite runs clean on macOS and in CI.
* Move mypy prek hooks to their respective distribution configs
The new mypy hooks for airflow-ctl-tests, helm-tests, airflow-e2e-tests,
task-sdk-integration-tests, docker-tests, and kubernetes-tests now live in each
distribution's own .pre-commit-config.yaml, matching the pattern already used
by airflow-core, task-sdk, and airflow-ctl. New .pre-commit- config.yaml files
are added to distributions that didn't have one. prek auto-discovers nested
configs, so the hooks remain part of the default check set.
mypy-dev (covers dev + scripts), mypy-devel-common, and mypy-shared stay at
the repo root: dev/scripts/devel-common don't have their own configs, and
mypy-shared iterates every shared/<dist> distribution so has no single home.
* Split mypy-dev and mypy-scripts, each with its own pyproject.toml config
Previously the mypy-dev prek hook ran mypy against dev/ and scripts/ in a
single invocation under the dev project's virtualenv. The two now get
independent hooks — mypy-dev in dev/.pre-commit-config.yaml and mypy-scripts in
scripts/.pre-commit-config.yaml — so each can evolve its own dependency set and
check its own folder.
Copy the full [tool.mypy] section from the root pyproject.toml into both
dev/pyproject.toml and scripts/pyproject.toml so each sub-project owns its mypy
configuration. Paths inside mypy_path are rewritten from $MYPY_CONFIG_FILE_DIR/
to $MYPY_CONFIG_FILE_DIR/../ so they still resolve to the repo-root siblings
from the sub-project location. The decorator/ outputs plugins are scoped to dev
only (scripts does not author DAG code).
mypy_local_folder.py now passes --config-file <project>/pyproject.toml when
the folder maps to one of these sub-project configs, so mypy uses the
sub-project's configuration rather than the root one.
* Teach selective-checks about the new non-provider mypy hooks
Add FileGroupForCi entries and regex patterns for helm-tests,
airflow-e2e-tests, docker-tests, kubernetes-tests, scripts, and shared Python
files, then wire them into skip_prek_hooks so the corresponding mypy-* prek
hook is only kept when its folder changed:
- mypy-scripts (split off from the old combined mypy-dev)
- mypy-airflow-ctl-tests, mypy-helm-tests, mypy-airflow-e2e-tests,
mypy-task-sdk-integration-tests, mypy-docker-tests, mypy-kubernetes-tests
- mypy-shared
Update test_selective_checks.py skip-list constants and per-case inline skip
lists to include the new hooks. Targeted test cases for files under the
new-hook directories override skip-prek-hooks to leave the matching hook out of
the skip set, confirming it will run when its folder changes.
* Trim dev/scripts pyproject mypy_path to just relevant distributions
Drop the 200+ provider path entries that were blindly copied from the root
pyproject.toml. dev and scripts only import from other non-provider workspace
members, so listing every provider src/tests directory under mypy_path just
adds noise. The remaining non-provider entries cover everything dev or scripts
plausibly import from.
* Install mypy into per-hook venvs from uv.lock via a `mypy` dep group
Each non-provider distribution with a mypy prek hook now declares a `mypy`
dependency group in its pyproject.toml resolving to
`apache-airflow-devel-common[mypy]`. mypy_local_folder.py syncs each dedicated
virtualenv with `uv sync --frozen --project <X> --group mypy` and runs mypy
with `uv run --frozen --project <X> --group mypy` — so mypy and its type stubs
come from the workspace uv.lock, not from an ephemeral `--with` overlay whose
resolution is independent of the main lockfile. uv.lock is refreshed to include
the new group.
Covers airflow-core, task-sdk, airflow-ctl, devel-common, dev, scripts,
airflow-ctl-tests, helm-tests, airflow-e2e-tests, task-sdk-integration- tests,
docker-tests, kubernetes-tests, and every shared/<dist> workspace member.
* Drop mypy_path from dev/scripts pyprojects — venv site-packages is enough
After the switch to installing mypy (and every transitive workspace
dependency) directly into each hook's virtualenv via the `mypy` dep group,
workspace packages like airflow, airflow.sdk, airflowctl, airflow_breeze,
tests_common are all available via the venv's site-packages. mypy resolves them
without needing mypy_path entries, so drop the copied list and leave a short
comment explaining why.
* Split mypy-shared into per-distribution hooks and enforce the pattern
Each shared/<dist> workspace member now owns a mypy-shared-<dist> prek hook
backed by its own shared/<dist>/.pre-commit-config.yaml. The single mypy-shared
iterator is gone — mypy_local_folder.py accepts shared/<dist> as a first-class
folder and the per-hook virtualenv now lives at
.build/mypy-venvs/shared-<dist>/ (slash in the folder name is replaced with a
dash in the venv/cache path).
Adds a new check-shared-mypy-hooks prek hook that fails when a shared/<dist>
workspace member is missing its dedicated .pre-commit- config.yaml, printing
the exact YAML to add. Selective-checks emits one skip entry per dist,
enumerated from shared/ at run time. Contributing docs cover the two-step
process for adding a new shared library.
* Pin minimum_prek_version to 0.3.4 consistently across all configs
All .pre-commit-config.yaml files now require prek >= 0.3.4 (the version
already declared by the root config). Previously the nested configs pinned a
mix of 0.2.0, 0.3.2, and 0.3.4, so a contributor could pass the root's version
check and still trip on stale subproject pins as they moved between directories.
* Refresh uv.lock after rebase to reflect the `mypy` dep groups
The rebase onto main resolved the uv.lock conflict by taking main's version,
so `uv sync --group mypy` would fail against uv.lock until the groups added to
the per-distribution pyprojects were re-resolved. Regenerates the lockfile to
include them.
* Add explicit selective-checks test for per-shared-dist mypy hook skipping
Verifies that when a file under shared/logging/ changes, only
mypy-shared-logging is kept among the thirteen mypy-shared-* hooks; all other
shared distributions' hooks land in the skip list. Pins the contract that the
runtime enumeration over shared/*/pyproject.toml works as intended.
* Refresh mypy docs to match the per-hook venv + --group mypy workflow
Fills in the docs that still referenced the pre-split workflow:
- AGENTS.md: mentions `mypy-shared-<dist>` per shared workspace member and
the `uv sync --group mypy` install path for mypy itself.
- scripts/ci/prek/AGENTS.md: clarifies that non-provider mypy hooks run
locally through mypy_local_folder.py (Breeze image only needed for the
providers hook).
- dev/breeze/doc/03_developer_tasks.rst: renames stale `mypy-airflow` to
`mypy-airflow-core`, and expands the cache note to cover the per-hook
virtualenvs and caches under .build/.
- dev/breeze/doc/ci/04_selective_checks.md: expands the file-group and
skip-reason lists so every new mypy hook (scripts, task-sdk, airflow-ctl, the
six test-dir hooks, and mypy-shared-<dist> enumerated at runtime) is documented.
* Rename mypy_local_folder.py to
run_mypy_full_dist_local_venv_or_breeze_in_ci.py
Updates every .pre-commit-config.yaml entry and prose references so they
point at the new script name. Two shared configs use YAML folded-scalar entries
to stay under the 110-char yamllint limit; updates the validation script's
expected template to match.
(cherry picked from commit 4f3b228be1a785ea5df7b89d8d3291ded78fea4d)
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
http://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name] following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
