kaxil opened a new pull request, #63269:
URL: https://github.com/apache/airflow/pull/63269
Adds `breeze registry backfill` — a new breeze subcommand that extracts
runtime parameters and connection types for previously released provider
versions. Uses `uv run --with` to install the specific version in a temporary
environment, so **no Docker or breeze CI image needed**.
```bash
# Single version
breeze registry backfill --provider amazon --version 9.15.0
# Multiple versions
breeze registry backfill --provider amazon --version 9.15.0 --version 9.14.0
```
The command:
1. Reads `provider.yaml` + `pyproject.toml` to find the package name and
optional extras
2. Patches `dev/registry/providers.json` to the target version (restores on
completion)
3. Runs `extract_parameters.py` and `extract_connections.py` via `uv run
--with {pkg}=={ver}`
4. Falls back to installing without extras if the full spec fails (handles
cases like Google's `leveldb` C dependency)
Output goes to
`registry/src/_data/versions/{provider}/{version}/parameters.json` and
`connections.json`.
## Design decisions
**Why `uv run --with` instead of Docker?** The extraction scripts just need
the provider package installed to discover modules via `issubclass()`. `uv`
handles this in seconds vs minutes for Docker image builds. Docker is still
used by `breeze registry extract-data` for the full (latest version) extraction.
**Why sequential per-provider?** `extract_parameters.py` and
`extract_connections.py` write to shared files (`modules.json`,
`providers.json`). Parallel local runs would clobber each other. The new
`registry-backfill.yml` workflow handles parallelization via GitHub Actions
matrix strategy — each provider runs in its own job.
**Why retry without extras?** Some providers have extras that require native
C libraries (e.g., Google's `leveldb` extra requires `plyvel` which needs
`leveldb` headers). The fallback ensures extraction still works for the base
package.
Working on sequential and retry with extras in a separate PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
