kaxil opened a new pull request, #65223:
URL: https://github.com/apache/airflow/pull/65223
The registry backfill workflow had two problems preventing it from
backfilling intermediate provider versions:
1. **Flat version list applied to all providers**: Input like
`providers="amazon google" versions="9.24.0 21.0.0"` tried all versions against
all providers, failing because amazon doesn't have 21.0.0 and google doesn't
have 9.24.0.
2. **Missing system dependencies**: Extraction ran `uv run --with` on a bare
GitHub runner, which lacks C libraries (krb5-dev, libxml2-dev) needed by
providers like amazon and google.
## Changes
**Workflow (`registry-backfill.yml`):**
- Replace separate `providers` + `versions` inputs with `provider-versions`
accepting `provider/version` pairs (e.g. `amazon/9.24.0 google/21.0.0
amazon/9.23.0`)
- jq-based matrix builder groups pairs by provider, so each job gets only
its relevant versions
- Add `build-ci-image` job and `prepare_breeze_and_image` step (same pattern
as `registry-build.yml`)
**Breeze command (`registry_commands.py`):**
- Add `_backfill_docker()` that runs extraction inside the Breeze CI
container via `pip install` + `execute_command_in_shell`
- Refactor existing host-based extraction into `_backfill_uv()` as
`--no-docker` fallback
- Default is Docker; local dev can use `--no-docker` for faster iteration
## Example usage
```bash
# Backfill multiple providers with different versions
gh workflow run "Registry Backfill" \
-f destination=live \
-f provider-versions="amazon/9.24.0 google/21.0.0 celery/3.17.2"
```
This creates parallel jobs: one for amazon (9.24.0), one for google
(21.0.0), one for celery (3.17.2). Multiple versions per provider are grouped
into a single job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]