kaxil commented on code in PR #71477: URL: https://github.com/apache/airflow/pull/71477#discussion_r4042141511
########## dev/registry/registry_tools/docs_guides.py: ########## @@ -0,0 +1,124 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Map a provider's classes to the how-to guide sections that document them. + +A module's ``docs_url`` points at generated API reference, which tells a reader +what the arguments are but not how the thing is meant to be used. The prose +guides carry that, and they already mark it: a how-to guide documents one class +per section, titled with the class name (``HookToolset``, ``SQLToolset``). + +So the mapping is read back out of the guides rather than curated anywhere: a +hand-maintained class-to-guide table would rot silently every time a guide is +split, renamed, or a class is dropped, and a rotten link is worse than none. +Callers supply the reST they can see (a git tag, or the working tree) and get +back only the anchors those sources actually contain. +""" + +from __future__ import annotations + +import re +from collections.abc import Mapping +from typing import Any + +# reST underlines an (optionally overlined) section title with a run of one +# punctuation character, at least as long as the title itself. +_ADORNMENT_CHARS = "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~" + +# Only titles opening with a class name as an inline literal are treated as +# documenting it, so prose headings ("Bounded query results") never produce a link. +_LEADING_CLASS_LITERAL = re.compile(r"^``([A-Za-z_][A-Za-z0-9_]*)``") Review Comment: The catalog carries decorators as modules named `@task.agent`, `@task.llm` and so on, and `operators/agent.rst` titles its section ``AgentOperator`` & ``@task.agent`` precisely because it documents both. Only the leading literal is captured, so the decorator half never gets a Guide link even though the title names it (`llm_file_analysis.rst` is the same shape). Could this collect every inline literal in the title, with the pattern widened enough to admit the `@task.x` form? ########## dev/registry/extract_parameters.py: ########## @@ -453,6 +455,16 @@ def _resolve_dotted_path(class_path: str) -> tuple[str, str, object] | None: return module_path, name, obj +def read_guide_docs(docs_dir: Path) -> dict[str, str]: + """Read a provider's authored reST docs from the working tree, keyed by path relative to ``docs_dir``.""" + if not docs_dir.is_dir(): + return {} + return { + path.relative_to(docs_dir).as_posix(): path.read_text(encoding="utf-8") + for path in sorted(docs_dir.rglob("*.rst")) Review Comment: `rglob` picks up everything under `docs/`, including the autoapi-generated `_api/` tree. common/ai has 93 `.rst` on disk but only 36 tracked, so the anchor set here depends on whether a docs build has run in the tree this executes against. Include-only files have the same problem (`providers/amazon/docs/_partials/` is never built as a page), and since `_`-prefixed paths sort first they would win the first-page tie-break in `collect_guide_anchors` and emit a link to an `.html` that does not exist. Skipping path parts starting with `_` closes both. ########## dev/registry/extract_versions.py: ########## @@ -181,6 +197,26 @@ def get_source_file_path(layout: str, dir_path: str, module_path: str) -> str: return f"providers/src/{rel_file}" +def read_guide_docs(tag: str, layout: str, dir_path: str) -> dict[str, str]: + """Read a provider's authored reST docs at a tag, keyed by path relative to its docs dir. + + Only the per-provider layout keeps docs beside the provider; under the old flat + layout they lived in a top-level ``docs/`` tree, so those tags get no guide + links rather than links guessed from a path that moved. + """ + if layout != "new": + return {} + + docs_prefix = f"providers/{dir_path}/docs/" + docs: dict[str, str] = {} + for path in git_ls_tree(tag, docs_prefix): + if not path.endswith(".rst"): Review Comment: This spawns one `git show` per `.rst`, and most of that set can never yield an anchor. At `providers-amazon/9.17.0` it is 102 files and 1.45s, of which `changelog.rst` alone is 140KB, and `--all-versions` pays that per provider-version. Filtering to the pages that are actually built (skip `_`-prefixed dirs plus `changelog.rst` and `commits.rst`) would cut most of the cost and fixes the same dead-page tie-break as the working-tree reader. ########## dev/registry/extract_parameters.py: ########## @@ -95,6 +96,7 @@ class Module: provider_id: str provider_name: str supports_durable_execution: bool + guide_url: str | None = None Review Comment: That makes 13 fields, so `discover_classes_from_provider`'s docstring below ("all 12 Module fields") is now off by one. ########## registry/AGENTS.md: ########## @@ -462,6 +462,24 @@ They run inside Breeze where all providers are installed. `extract_metadata.py` the CI workflow can run the fast scripts (metadata, ~30s per provider) without spinning up Breeze, while parameter/connection extraction is a separate step. +### How a module gets a "Guide" link + +A module card links to the how-to guide section that documents it, alongside the +generated API reference. Nothing declares that link: `registry_tools/docs_guides.py` +reads the provider's own `docs/*.rst` and matches a class to a section when the +section's title *opens with the class name as an inline literal* — ``` ``HookToolset`` ``` +or ``` ``AgentOperator`` & ``@task.agent`` ```. The anchor is derived from the whole +title the way docutils derives its HTML id. + +That convention is what the guides already do, and it is deliberately the only Review Comment: I checked this across the tree: common/ai is the only provider whose guides title sections with a leading inline literal, 17 of them. informatica has 3, but they name config options rather than classes, and the other 92 providers have none. Worth saying so here, since it means extending Guide links to another provider is a matter of editing that provider's headings rather than touching this extractor. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
