nailo2c opened a new pull request, #65511:
URL: https://github.com/apache/airflow/pull/65511
closes: #54394
<br>
Hi folks, I know this is a big PR. I'll try my best to explain it concise
and clear.
## Summary
Add build constraints support to Airflow's constraints pipeline.
Airflow now generates and publishes `build-constraints-{python}.txt` next to
the existing runtime constraints, and install flows pass them to `uv pip
install --build-constraints` or `pip install --build-constraint` when available.
## How it works
```mermaid
flowchart LR
A["Existing inputs:<br/>uv.lock + workspace pyproject.toml"]
B["Existing runtime constraints"]
C["Existing constraints branch"]
D["Existing install commands (uv pip install / pip install)"]
X["NEW build constraints generation"]
Y["NEW build-constraints-3.x.txt"]
Z["NEW --build-constraints flags"]
A --> B --> C --> D
A --> X --> Y --> C
C --> Z --> D
classDef existing fill:#ffffff,stroke:#777777,color:#111111;
classDef added
fill:#e7f5ff,stroke:#1971c2,stroke-width:2px,color:#111111;
class A,B,C,D existing;
class X,Y,Z added;
```
**Key design decisions:**
- Build constraints are a single file per Python version, shared across all
3 constraint modes.
- `uv sync` paths are excluded because `uv sync` does not support
`--build-constraints`.
- Fallback no-constraints paths never include build constraints, matching
runtime constraints behavior.
- Auto-inferred missing build constraints are skipped for backward
compatibility with old constraint branches
- Explicit `AIRFLOW_BUILD_CONSTRAINTS_LOCATION` values fail fast if the file
or URL is missing.
- The generator targets build requirements relevant to Airflow's default
wheel-preferred install path, rather than forcing source-build coverage for
every locked package.
<details>
<summary>Source-scan scope validation</summary>
A separate full-scan script scans **all** ~749 packages with sdist URLs
(no wheel filtering, no cache). Running both against the same `uv.lock`:
```
Production (cold): 24.4s (110 sdists) → 24 unique build deps
Production (warm cache): 0.4s (0 downloads)
Full scan: 35.9s (749 sdists) → 61 unique build deps
Only in full scan (production skipped): 37
poetry, poetry-core, pdm-backend, mypy, jupyterlab, twine, ...
Only in production (should be 0): 0
```
All 37 extra deps come from packages with universal wheels. Under the
default wheel-preferred install path, those packages are not expected to enter
build isolation, so the production scanner covers the build deps relevant to
normal Airflow installs while keeping generation fast.
</details>
## Review guide
Suggested review order follows the data flow:
| Area | Main files | What to review |
|------|------------|----------------|
| Generator | scripts/in_container/run_generate_constraints.py | Collect
workspace build requirements and upstream build requirements from uv.lock, then
resolve pinned build dependency versions with uv pip compile. |
| Install helpers | scripts/docker/common.sh | Resolve/download build
constraints and emit the correct uv/pip install flag. |
| Install consumers |
scripts/docker/*.sh<br/>scripts/in_container/install_airflow_and_providers.py<br/>scripts/in_container/install_development_dependencies.py
| Pass build constraints to relevant install commands and omit them from
fallback paths. |
| Breeze plumbing | dev/breeze/src/airflow_breeze/... | Expose
--airflow-build-constraints-location and pass it through env vars / Docker
build args. |
| Publishing |
.github/workflows/generate-constraints.yml<br/>scripts/ci/constraints/*.sh |
Upload, diff, and commit build-constraints-*.txt, including first-time
untracked files. |
| Docs |
installing-from-pypi.rst<br/>MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md<br/>09_release_management_tasks.rst
| Document generated files and user-facing uv / pip usage. |
| Tests | scripts/tests/in_container/test_build_constraints.py | Unit
coverage for resolver behavior, helper behavior, command assembly, and fallback
invariants. |
Generated/synced files:
- Dockerfile, Dockerfile.ci were regenerated from the command `prek
update-inlined-dockerfile-scripts --all-files`.
- Breeze command help docs/images were regenerated after adding the new CLI
option.
## Verification
- 64 focused unit tests cover generator, resolver, command assembly, and
fallback behavior.
- Manual Breeze generation produced `build-constraints-*.txt` for all three
constraints modes.
<details>
<summary>Generation: workspace scan, upstream scan, and constraint
resolution</summary>
```console
$ uv run --project scripts python -c "
import sys; sys.path.insert(0, 'scripts/in_container')
from run_generate_constraints import _collect_workspace_build_reqs
from pathlib import Path
reqs = _collect_workspace_build_reqs(Path('.'))
print(f'{len(reqs)} deps: {sorted(reqs)}')
"
10 deps: ['flit-core', 'gitdb', 'gitpython', 'hatchling', 'packaging',
'pathspec', 'pluggy', 'smmap', 'tomli', 'trove-classifiers']
$ uv run --project scripts python -c "
import sys; sys.path.insert(0, 'scripts/in_container')
from run_generate_constraints import _parse_uv_lock
from pathlib import Path
pkgs = _parse_uv_lock(Path('uv.lock'))
targets = [p for p in pkgs if not p.has_universal_wheel and p.sdist_url]
print(f'{len(pkgs)} total, {len(targets)} targets')
"
893 total, 110 targets
$ uv run --project scripts python -c "
import sys; sys.path.insert(0, 'scripts/in_container')
from run_generate_constraints import _stream_build_reqs_from_sdist,
_parse_uv_lock
from pathlib import Path
pkgs = _parse_uv_lock(Path('uv.lock'))
pkg = next(p for p in pkgs if p.name == 'pydantic-core')
reqs = _stream_build_reqs_from_sdist(pkg.sdist_url)
print(f'pydantic-core: {reqs}')
"
pydantic-core: ['maturin>=1.9.4,<2']
```
</details>
<details>
<summary>Integration: 3 constraint modes all produce
build-constraints-*.txt</summary>
```console
$ breeze ci-image build --python 3.12 --upgrade-to-newer-dependencies
--answer yes
$ breeze release-management generate-constraints \
--airflow-constraints-mode constraints --run-in-parallel --answer yes
$ breeze release-management generate-constraints \
--airflow-constraints-mode constraints-source-providers
--run-in-parallel --answer yes
$ breeze release-management generate-constraints \
--airflow-constraints-mode constraints-no-providers --run-in-parallel
--answer yes
$ ls files/constraints-3.12/
build-constraints-3.12.txt constraints-3.12.txt
diff-constraints-3.12.md
build-deps-cache.json constraints-no-providers-3.12.txt
diff-constraints-no-providers-3.12.md
constraints-source-providers-3.12.txt
diff-constraints-source-providers-3.12.md
original-constraints-3.12.txt
original-constraints-no-providers-3.12.txt
original-constraints-source-providers-3.12.txt
$ cat files/constraints-3.12/build-constraints-3.12.txt
# (header omitted)
beniget==0.4.2.post1
flit-core==3.12.0
hatchling==1.29.0
maturin==1.13.1
meson-python==0.18.0
setuptools==82.0.1
wheel==0.46.3
... (31 pinned build deps total)
```
</details>
<details>
<summary>Negative & positive tests: build constraints actually block/allow
installation</summary>
```console
# Create a test package requiring setuptools>=70
$ mkdir -p /tmp/test-pkg && cat > /tmp/test-pkg/pyproject.toml << 'EOF'
[build-system]
requires = ["setuptools>=70"]
build-backend = "setuptools.build_meta"
[project]
name = "test-build-constraint"
version = "0.1.0"
EOF
$ echo "from setuptools import setup; setup()" > /tmp/test-pkg/setup.py
$ cd /tmp/test-pkg && uv build --sdist && cd -
# Negative: conflicting build constraints → correctly fails
$ echo "setuptools==60.0.0" > /tmp/bad-build-constraints.txt
$ uv pip install /tmp/test-pkg/dist/test_build_constraint-0.1.0.tar.gz \
--build-constraints /tmp/bad-build-constraints.txt
× Failed to build `test-build-constraint`
├─▶ Failed to resolve requirements from `build-system.requires`
╰─▶ Because you require setuptools>=70 and setuptools==60.0.0,
we can conclude that your requirements are unsatisfiable.
# Positive: matching build constraints → succeeds
$ echo "setuptools==82.0.1" > /tmp/good-build-constraints.txt
$ uv pip install /tmp/test-pkg/dist/test_build_constraint-0.1.0.tar.gz \
--build-constraints /tmp/good-build-constraints.txt
Installed 1 package in 59ms
+ test-build-constraint==0.1.0
```
</details>
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [v] Yes (please specify the tool below)
Claude Opus 4.6 + Codex 5.4
<!--
Generated-by: [Tool Name] following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
