kaxil opened a new pull request, #65984: URL: https://github.com/apache/airflow/pull/65984
## Summary `dev/registry/extract_metadata.py` took the top entry of `provider.yaml`'s `versions:` list as a provider's "latest" version with no verification that a real release tag exists. Two concrete failure modes shipped phantom pointers to the live registry: - `providers/celery/provider.yaml` lists `3.19.0` at the top, but only `providers-celery/3.19.0rc1` and `rc2` tags exist — no final. The registry would advertise `celery 3.19.0` as latest while `pip install apache-airflow-providers-celery` resolves to `3.18.0`. - `providers/akeyless/` is brand-new in-tree with `versions: [1.0.0]` but no `providers-akeyless/*` tag. The provider would appear on the registry with broken outbound links to its non-existent PyPI release, GitHub tag, and docs page. ## Design rationale **One-shot tag load**: `load_release_tags()` runs `git tag --list 'providers-*'` once and returns a set, so per-provider lookups are O(1). Wave tags (`providers/<YYYY-MM-DD>`, with a slash, no dash) don't match the glob, so only per-provider release tags are loaded. **Walk newest-first**: `find_latest_released_version` walks `versions:` in document order and returns the first entry with a matching tag. All current `provider.yaml` files are newest-first; if any drifts to non-monotonic ordering in the future, the function still picks the first *tagged* entry it encounters — degraded but never wrong. **Filter the list, not just the latest pointer**: this PR filters both `Provider.version` (singular, the latest) AND `Provider.versions` (the list). The latter is read by `extract_versions.py`'s backfill, which would otherwise try to `git show` from non-existent phantom tags. **Skip vs. fallback**: when no entry in `versions:` has a tag (truly unreleased provider), the provider is skipped from `providers.json` rather than emitted with `version="0.0.0"`. The old `0.0.0` fallback shipped a registry page with broken outbound links — strictly worse than the page not existing. **Workflow change**: `registry-build.yml`'s checkout now sets `fetch-tags: true`. The default `fetch-depth: 1` checkout has no tags, which would silently return an empty set from `load_release_tags()` and trigger the same `versions[0]` fallback the rest of this PR is removing. `registry-backfill.yml`'s primary checkout already uses `fetch-depth: 0`, so tags are present there. ## Behaviour change - Previously: provider with no tagged version → emitted with `version="0.0.0"` and broken links. - Now: provider with no tagged version → skipped from the registry entirely, with a per-provider "skipping" log line and an end-of-run summary. Inspecting the current main HEAD (`upstream/main` at the time of writing): exactly two providers fall into the new "skip" category — `akeyless` (brand-new, never released) and `celery` if you read it as "no `3.19.0` final" (caught by the latest-pointer fallback to `3.18.0`). ## Gotchas - Run on a machine without `git` installed → `load_release_tags()` returns an empty set, falls back to the previous behaviour with a one-line warning. Acceptable for local dev; production runs through CI which always has `git`. - A bogus tag (e.g. `providers-amazon/99.99.0` exists but `provider.yaml` doesn't exist at that commit) would still pass the existence check. Defending against that requires a `git show` validation per tag and is left as follow-up — current tag hygiene is good enough that no real provider has this issue today. ## Known follow-ups (not in scope here) - `extract_parameters.py` runs against `provider.yaml` directly and doesn't yet honour the skip signal — an unreleased provider's modules can still leak into `modules.json`. Tracked separately. - The `dev/registry/AGENTS.md` doc explains the four extractors but doesn't mention this filter; happy to add a paragraph in a doc-only follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
