Lee-W commented on code in PR #71477: URL: https://github.com/apache/airflow/pull/71477#discussion_r4130314553
########## dev/registry/registry_tools/docs_guides.py: ########## @@ -0,0 +1,163 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Map a provider's classes to the how-to guide sections that document them. + +A module's ``docs_url`` points at generated API reference, which tells a reader +what the arguments are but not how the thing is meant to be used. The prose +guides carry that, and they already mark it: a how-to guide documents one class +(or a class and its task-flow decorator) per section, titled with the name(s) +(``HookToolset``, ``SQLToolset``, ``AgentOperator`` & ``@task.agent``). + +So the mapping is read back out of the guides rather than curated anywhere: a +hand-maintained name-to-guide table would rot silently every time a guide is +split, renamed, or a class is dropped, and a rotten link is worse than none. +Callers supply the reST they can see (a git tag, or the working tree) and get +back only the anchors those sources actually contain. +""" + +from __future__ import annotations + +import re +from collections.abc import Mapping +from pathlib import PurePosixPath +from typing import Any + +# reST underlines an (optionally overlined) section title with a run of one +# punctuation character, at least as long as the title itself. +_ADORNMENT_CHARS = "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~" + +_SKIPPED_PAGE_NAMES = frozenset({"changelog.rst", "commits.rst"}) + + +def is_guide_page(relative_path: str) -> bool: + """Whether a path relative to a provider's docs directory is a how-to guide page. + + Callers hand every ``.rst`` they can see to this before it ever reaches + ``collect_guide_anchors``. Two kinds of real, built pages must not go + further: + + - Anything under a ``_``-prefixed path segment, at any depth + (``_api/hook/index.rst``, ``operators/_partials/foo.rst``, top-level + ``_partials/foo.rst``): Sphinx/autoapi output and partials are directive + markup, not the hand-written, reST-underlined titles this module's + leading-inline-literal convention parses. + - ``changelog.rst`` and ``commits.rst``: real release-note pages, not + how-to guides, that can carry inline-literal-formatted headings by + coincidence. + """ + path = PurePosixPath(relative_path) + if any(part.startswith("_") for part in path.parts): + return False + return path.name not in _SKIPPED_PAGE_NAMES + + +# A single inline-literal name: a class (``HookToolset``) or a task-flow +# decorator (``@task.llm_file_analysis``) -- narrow enough that it still can't +# match arbitrary prose wrapped in backticks. +_INLINE_LITERAL_NAME = r"``(@?[A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)*)``" +_INLINE_LITERAL_NAME_RE = re.compile(_INLINE_LITERAL_NAME) + +# Only titles opening with a run of inline-literal names are treated as +# documenting them, so prose headings ("Bounded query results") never produce a +# link. A run is one or more names joined by "&", "," or "/" -- how guides write +# a section that covers both an operator and its decorator +# (``AgentOperator`` & ``@task.agent``). The run stops at the first thing that +# is neither a name nor a separator, so it never reaches into prose. +_LEADING_LITERAL_NAME_RUN = re.compile(rf"^{_INLINE_LITERAL_NAME}(?:\s*[&,/]\s*{_INLINE_LITERAL_NAME})*") Review Comment: Both title shapes are read now: a title that opens with the name run (what older release tags still use), and one that ends with a colon followed by the name run. A page's own title beats a subsection on another page, so `HookToolset` lands on `toolsets/hook.html` and `LLMBatchOperator` on its page title. Every resolved anchor exists in a `common.ai` docs build, and no other provider's result changed. The fixtures use the new shape, with the leading shape kept for the tags, and the AGENTS.md paragraph now describes both shapes instead of the "most thoroughly" claim. The reorg also left four decorators (`@task.llm_file_analysis`, `@task.llm_sql`, `@task.llm_branch`, `@task.llm_schema_compare`) under a "TaskFlow Decorator" subsection whose title does not name them, so their pages' titles now name them next to the operator, the way the `LLMOperator` and `AgentOperator` pages do. That brings `common.ai` to 31 names. I kept titles rather than `objects.inv` labels: labels would have to be added to every guide section, and a release tag's classes would resolve against the `stable` inventory instead of that tag's own docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
