kaxil opened a new pull request, #74284:
URL: https://github.com/apache/airflow/pull/74284
The registry ignores the `filesystems`, `asset-uris` and `remote-logging`
sections of `provider.yaml`, so a provider page never says which URI schemes
the provider handles. Nothing on the Microsoft Azure page tells you that
`sharepoint://`, `onedrive://` and `msgraph://` paths work with
`ObjectStoragePath`, and nothing on the Amazon page says `cloudwatch://` is a
remote logging target. These three sections are the ones Airflow resolves by
URI scheme at runtime, so this adds a **URI Schemes** card to the provider page
built from them.
23 providers register at least one scheme (41 schemes in total) and get the
card. Providers that register none render nothing new.
Microsoft Azure (dark):
Amazon (dark), with all three columns in use:
Google (light), including `gcp://`, which is registered with a null asset
handler:
Phone width: the table scrolls inside the card.
Each column header links to the matching Airflow docs page. Hovering a check
mark shows the registered module, class or function paths.
## Registry API changes
Additive only. Every provider in `/api/providers.json` gains a `uri_schemes`
array with one entry per scheme:
```json
{
"scheme": "s3",
"filesystem": "airflow.providers.amazon.aws.fs.s3",
"asset": {
"handler": "airflow.providers.amazon.aws.assets.s3.sanitize_uri",
"factory": "airflow.providers.amazon.aws.assets.s3.create_asset",
"to_openlineage_converter":
"airflow.providers.amazon.aws.assets.s3.convert_asset_to_openlineage"
},
"remote_logging":
"airflow.providers.amazon.aws.log.s3_task_handler.S3RemoteLogIO"
}
```
- `filesystem`, `asset` and `remote_logging` are present only when the
provider registers that feature for the scheme.
- `asset.handler: null` keeps its `provider.yaml` meaning: the scheme is
registered with Airflow's no-op normalizer (Google's `gcp://`). An `asset-uris`
entry with no `handler` key is skipped, as `ProvidersManager` skips it.
- The field defaults to `[]`, so a `providers.json` already on S3 still
validates when an incremental build merges into it.
- The OpenAPI spec picks up the new `UriSchemeContract` and
`AssetUriContract` schemas from the shared contracts, with a description on
each field.
- The per-provider endpoints (modules, parameters, connections, versions)
are unchanged. Per-version `metadata.json` carries the same field for older
release pages, but it is build input, not an API endpoint.
## Design rationale
- **One table keyed by scheme, not three new module types.** Filesystem
modules expose a `get_fs` function and asset handlers are plain functions, so
neither fits the class-based module grid. Three extra filter tabs covering 4 to
19 providers each would also crowd the module tabs. Keyed by scheme, the three
sections answer one question: what Airflow does with a path or URI that has
this scheme.
- **Filesystem schemes come from the module source.** `provider.yaml` lists
only the module; the schemes are its module-level `schemes` list, which
`airflow.sdk.io.fs` reads after import. The extractor reads that list with AST,
like the rest of `extract_metadata.py`, so it still runs on the host without
providers installed. For older versions it reads the file from the release tag.
- **The table shows what a release registers, which can be less than what it
supports.** The `remote-logging` section is recent (#69816 added it for
Amazon), so a release like amazon 9.17.0 ships `S3RemoteLogIO` without
registering it. The intro line says the table reflects `provider.yaml`, and an
empty cell reads "Not registered" to screen readers, not "Not supported".
## Gotchas
- Older version pages get the card the next time the backfill workflow
rebuilds them. Until then they show no card, because the field defaults to
empty.
- Scheme search is not included. Adding schemes to the search index needs a
collision rule like the one used for external services: `ssh` is a scheme of
the SFTP provider and would compete with the SSH provider in ranking.
- Other `provider.yaml` sections the registry still ignores (`config`,
`email-backends`, `xcom`, `cli`, `auth-backends`) are out of scope here.
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
