kaxil opened a new issue, #74376: URL: https://github.com/apache/airflow/issues/74376
Coding agents (Claude Code, Cursor, Copilot, OpenCode) read our docs to write Dags, and today they get only HTML. A provider guide page is about 150 KB of HTML for about 20 KB of content, and fetch tools often summarise it down to a paragraph, losing the code samples. Projects such as Temporal, Pydantic AI, Ray and Camel publish an `llms.txt` index and a Markdown copy of each page. `https://airflow.apache.org/llms.txt` and every `/docs/.../llms.txt` return 404. This issue covers doing the same for Airflow's docs, starting with the Common AI provider (`apache-airflow-providers-common-ai`), in three parts. A working proof of concept exists for the first two, and the third was tested on the staging site. > [!IMPORTANT] > **This needs a committer.** Take it on only if you are a committer, or are pairing with one who will test and verify it with you. Parts 2 and 3 cannot be checked from a pull request alone: > - **Testing the `.htaccess` rule** means pushing to airflow-site's `staging` branch and probing `airflow.staged.apache.org`. > - **Publishing docs** to the live bucket is a committer workflow. > - **The production rollout** has to be checked on the live site: the `.md` copies exist, the `Vary` and content-type headers are right, and browsers still get HTML. ## 1. Docs build (this repo) Proof of concept: [`astronomer/airflow` branch `llms-txt-poc`](https://github.com/astronomer/airflow/tree/llms-txt-poc), one commit. - [`sphinx-llm`](https://github.com/NVIDIA/sphinx-llm) runs a second Sphinx build with a Markdown builder and writes `page.html.md` next to every `page.html`. It also adds `<link rel="alternate" type="text/markdown" href="page.html.md">` to each HTML page, which Codex CLI follows. - A new extension, `devel-common/src/sphinx_exts/airflow_llms_txt.py`, enabled for Common AI only through `PROVIDER_PACKAGES_WITH_LLMS_TXT` in `devel-common/src/docs/provider_conf.py`: - **Fixes the Markdown for Airflow docs:** mermaid diagrams, code-block captions, admonitions, compact tables, same-page links, license-header comments, and `|version|` in URLs. - **Links each `exampleinclude` excerpt to the full example file on GitHub,** because the excerpts leave out imports. - **Writes `llms.txt`** grouped by the docs navigation, with absolute versioned URLs and a description per page. - **Writes `llms-full.txt`:** the same pages in one file. - Build time for Common AI is unchanged at about 45 s locally, and nothing changes for other packages. Before merging: - [ ] **Cut the extension down.** About 90 of its ~360 lines use private internals of `sphinx-markdown-builder` (tables, admonitions, same-page links) or work around a `sphinx-llm` bug (in sequential mode the Markdown build reuses the HTML build's doctree cache). Upstream those to `sphinx-markdown-builder` and `sphinx-llm`, and keep only the Airflow-specific parts that use public Sphinx APIs here. - [ ] **Write a `.html.md` for each redirect stub** generated from `docs/redirects.txt` (seven in Common AI), saying where the page moved. - [ ] **Fail the build if a published `.html` page has no `.html.md`,** excluding `_modules/`, `genindex`, `search` and `py-modindex`. The serving rule in part 3 relies on every other page having one. - [ ] **Add unit tests for the extension.** ## 2. Publishing (`dev/breeze`) - [ ] **Stop `stable/` going backwards.** `stable/` is a full copy (`aws s3 sync --delete`) of whichever version was published last. Republishing or patching an older version with the default `skip-write-to-stable-folder: false` replaces `stable/` with an older build, and `docs_publisher.py` writes `stable.txt` without comparing versions. Make the publisher refuse to overwrite `stable/` with a version lower than `stable.txt`. - [ ] **Add a post-publish check** that `stable/index.html.md` returns 200 with a `text/markdown` content type. S3 stores the content type the AWS CLI guesses from the file extension, and older Python `mimetypes` tables have no entry for `.md`, which leaves `binary/octet-stream`. ## 3. Serving: `Accept: text/markdown` on airflow.apache.org (apache/airflow-site) Claude Code sends `Accept: text/markdown, text/html, */*` on every fetch; Cursor, Copilot and OpenCode send similar headers ([support table](https://acceptmarkdown.com/status)). If `airflow.apache.org` answers those requests with the Markdown copy, agents get it from the URLs they already use, with no need to find `llms.txt`. Read the Docs and Cloudflare-hosted sites already do this. No ASF project does it on purpose yet. `/docs` is proxied to the docs CloudFront in `landing-pages/site/static/.htaccess`, so this is a `.htaccess` change placed before that proxy rule: ```apache # Phase 1: the Common AI provider's stable docs only. Deploy this only after # https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/index.html.md returns 200. RewriteCond %{HTTP:Accept} text/markdown [NC] RewriteCond %{HTTP:Accept} !text/markdown\s*;\s*q=0(\.0*)?\s*(,|$) [NC] # Match the raw request line, so the rewrite and the Vary header below decide on the same string. RewriteCond %{THE_REQUEST} " /docs/apache-airflow-providers-common-ai/stable/" RewriteCond %{THE_REQUEST} !"/(_modules/|genindex\.html|search\.html|py-modindex\.html)" RewriteRule ^(docs/apache-airflow-providers-common-ai/stable/.+\.html)$ https://d7fnmbhf26p21.cloudfront.net/$1.md [P,L] RewriteCond %{HTTP:Accept} text/markdown [NC] RewriteCond %{HTTP:Accept} !text/markdown\s*;\s*q=0(\.0*)?\s*(,|$) [NC] RewriteCond %{THE_REQUEST} " /docs/apache-airflow-providers-common-ai/(stable/(?!_modules/)(.+/)?)?[ ?]" RewriteRule ^docs/apache-airflow-providers-common-ai/(?:(stable/(?:.+/)?))?$ https://d7fnmbhf26p21.cloudfront.net/docs/apache-airflow-providers-common-ai/stable/$1index.html.md [P,L] <IfModule mod_headers.c> Header merge Vary Accept "expr=%{THE_REQUEST} =~ m# /docs/apache-airflow-providers-common-ai/#" Header set X-Robots-Tag "noindex" "expr=%{THE_REQUEST} =~ m#\.html\.md[ ?]#" Header edit Content-Type "^(text/(markdown|plain))$" "$1; charset=utf-8" "expr=%{THE_REQUEST} =~ m# /docs/#" </IfModule> ``` What was tested on `airflow.staged.apache.org`, with each page's `_sources/*.rst.txt` standing in for the not-yet-published `.html.md`: - **Agents:** a request with `Accept: text/markdown` got the text copy with `charset=utf-8`. - **Browsers:** the same URL returned HTML. - **`Vary`:** every response for the package carried `Vary: Accept-Encoding,Accept`. - **Other packages:** unchanged. - **End to end:** Claude Code's fetch tool received the text copy from an ordinary `.html` URL. ASF's Fastly cache honours `Vary: Accept`; the same check on `apisix.apache.org` returned separate cache hits for Markdown and HTML requests. That staging test used an earlier version of the rule. The rule above adds fixes for problems it had: - **Crafted URLs bypassed `Vary`.** `…/common-%61i/…` and `//docs/…` matched the rewrite but not the `Vary` condition, so Fastly could have cached Markdown for browsers. - **The directory rule lacked the `_modules/` exclusion.** - **`text/markdown;q=0` still matched,** even though a client sending it is refusing Markdown. Re-test the rule above on staging before production. Rollout constraints: - **A missing `.md` returns S3's 404, and Fastly caches it** for up to an hour. Publishing invalidates CloudFront, not Fastly. So turn the rule on for a package only once its `stable/` has `.md` copies, and keep an explicit package list: removed providers (`sqoop`, `plexus`, `qubole`) still serve `stable/` and will never get `.md` files. - **Fastly doesn't normalise `Accept`,** so `Vary: Accept` stores one cache entry per distinct `Accept` string, up to 50 per URL. That's fine for one package. Talk to ASF Infra before widening the rule to all of `/docs`. - **Rollback** is reverting the `.htaccess` commit; Fastly may take up to an hour to drop cached copies. ## Later - A root `https://airflow.apache.org/llms.txt` in airflow-site, pointing at each package's `stable/llms.txt`. - The same for the core `apache-airflow` docs, which get far more traffic than any provider. - Versioned offline bundles (a zip of every `.md` per release) for agents without internet access, as [Camel](https://issues.apache.org/jira/browse/CAMEL-23788) does. ## References - [llms.txt format](https://llmstxt.org/) - [Read the Docs: Markdown for agents](https://docs.readthedocs.com/platform/latest/reference/markdown-for-agents.html) - [Ray's in-repo Sphinx `llms.txt` extension](https://github.com/ray-project/ray/blob/master/doc/source/_ext/llms_txt.py) - [SPARK-53528](https://issues.apache.org/jira/browse/SPARK-53528): Spark's SPIP for `llms.txt` in Spark docs, which also uses Sphinx - [INFRA-26812](https://issues.apache.org/jira/browse/INFRA-26812): Fastly caching in front of Airflow's `/docs` proxy -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
