andygrove opened a new issue, #7684:
URL: https://github.com/apache/datafusion-comet/issues/7684

   ## What is the problem?
   
   `https://datafusion.apache.org/comet/user-guide/latest/` serves the 
unreleased docs built from `main`. There is no stable URL for the newest 
release, and several places send users to `latest/` expecting release docs:
   
   - `CometConf.COMPAT_GUIDE` and `CometConf.TUNING_GUIDE` put 
`user-guide/latest/...` links into config descriptions and fallback messages, 
so a released 1.2 jar sends users to docs that describe unreleased behavior.
   - The home page buttons link to `user-guide/latest/...`.
   - The user guide index lists the main-branch guide as "Development Snapshot 
<latest/index>".
   
   Apache Spark and Apache Iceberg serve the newest release at `latest/`, which 
is what users expect from that name.
   
   ## Proposal
   
   | URL                  | Content                                   |
   | -------------------- | ----------------------------------------- |
   | `user-guide/latest/` | The newest release                        |
   | `user-guide/dev/`    | The development snapshot, built from main |
   | `user-guide/1.2/`    | Each release, pinned (as today)           |
   
   Only the versioned URLs are permanent. Pages under `latest/` and `dev/` may 
be added, renamed or removed between releases, and we do not add redirects to 
keep old `latest/` deep links alive. The user guide index should say so. 
Expected consequences: existing links to `/user-guide/latest/...` start showing 
release content, and pages that exist only on main will 404 under `latest/` 
until the next release.
   
   ## Prior work
   
   I opened #7112 for this. It has one approval and the review comments were 
addressed, but I do not have time to finish it. It is 109 files and now 
conflicts with `main`. About 50 of those files are the mechanical `latest` to 
`dev` source directory rename, so it is easier to redo on current `main` than 
to resolve the conflicts. The branch is 
`andygrove:documentation-publishing-54d88243` and still has the working 
implementation. Its diff also carries unrelated hunks from an older base (for 
example the Spark 3.4 profile and the release build notes), so take only the 
docs and publishing changes.
   
   ## Design that worked
   
   1. **Rename the source directory `docs/source/user-guide/latest` to `dev`.** 
This is required, not cosmetic. Sphinx resolves the contributor guide's 
`../user-guide/latest/...` links by docname, and the `latest` alias cannot also 
be a source directory. About 70 files outside the directory reference it, so 
grep for `user-guide/latest` after the rename. Known spots: 
`.github/labeler.yml`, `.prettierignore`, `dev/generate-release-docs.sh`, 
`docs/source/conf.py` (the `exclude_patterns` entry), `docs/README.md`, 
`contrib/delta-spark/README.md`, the contributor guide pages, the expression 
audit pages, `spark/pom.xml` (the `generate-docs` output paths), and the agent 
skills under `.ai/skills/` (`wire-datafusion-function`, 
`review-comet-expression-pr`).
   2. **`docs/generate-versions.py`.**
      - Move the release versions to two constants at the top of the file 
(`LATEST_RELEASED_VERSION`, `PREVIOUS_VERSIONS`), so promoting a release also 
moves `latest`.
      - Build the main-branch guide as `dev`.
      - Detect each release branch's guide directory instead of switching on 
the version number. `branch-1.1` and `branch-1.2` both keep it under 
`user-guide/latest/`, and branches cut after the rename will use 
`user-guide/dev/`. Raise if neither exists, since falling back to the whole 
`user-guide` tree hides a failed `git clone`.
      - Fail the build if `docs/source/user-guide/latest` exists on main, 
because the alias step deletes whatever Sphinx built there.
   3. **Alias after the Sphinx build.** Add `generate-versions.py 
--alias-latest <html dir>`, called from `docs/build.sh` after Sphinx, which 
copies `user-guide/<major.minor>/` to `user-guide/latest/`. Copy the finished 
HTML instead of building the sources twice, so Sphinx sees each document once. 
Use a copy, not a symlink, because the site is published by rsync into 
`asf-site` and ASF's web server is not known to follow symlinks. Every link in 
the pages is relative, so the copy works one directory over.
   4. **Redirects in `docs/source/conf.py`.** The five `user-guide/latest/...` 
redirects covered old dev-tree URLs and should go. 1.2 has real pages for 
`compatibility/scans.html`, `tuning/scans.html` and `iceberg-writes.html`, but 
not for `compatibility.html` or 
`compatibility/parquet-iceberg-known-issues.html`, so those two will 404 under 
`latest/` until the next release. sphinx-reredirects writes its stub pages 
during `html-collect-pages`, and the alias step then removes the whole 
`latest/` directory, so keeping a redirect entry has no effect as written. If 
we want to keep those two, copy with `shutil.copytree(source, target, 
dirs_exist_ok=True)` instead of deleting the target first (`build.sh` already 
starts from an empty `build/`).
   5. **Version-aware links in the jar.** Add a small `CometDocs` object that 
maps the running Comet version to a guide directory: `1.2.3` gives `1.2`, and a 
snapshot or unknown version gives `dev`. Build `COMPAT_GUIDE` and 
`TUNING_GUIDE` from it. The plan stability suites embed these URLs in the 
TPC-DS goldens (`q24a`, `q24b` and `q24` `extended.txt`), so 
`CometPlanStabilitySuite` has to normalize the version to `{version}` or every 
release bump rewrites the goldens. Add a `CometDocsSuite` for the version 
mapping.
   6. **Docs.** Update the user guide index (a URL table, the "current stable 
release" sentence, the toctree entry), `docs/README.md`, `release_process.md` 
(promoting a release now means editing the two constants, and `index.md` has a 
new URL-table row to update) and `backporting.md`.
   
   ## Review findings to carry over
   
   - `CometDocsSuite` must be registered in 
`.github/workflows/pr_build_linux.yml`. `python3 dev/ci/check-suites.py` exits 
non-zero otherwise.
   - The agent skill instructions 
(`.ai/skills/wire-datafusion-function/SKILL.md` and the others) must point at 
`user-guide/dev/` in the same change. Following the old text recreates the 
`latest` source directory that the guard rejects.
   - PR CI never runs `docs/build.sh`. The `docs` job runs only on push to 
`main`, so the first CI run of the alias step is the post-merge deploy. A 
failure there stops before rsync, leaving the site stale rather than broken, 
but I would still run the full `docs/build.sh` locally first.
   - After the rename, `iceberg-writes.md` no longer exists (it was merged into 
`iceberg.md`), so the benchmark source comments that cite it should point at 
the "Accepted divergences" section of `dev/iceberg.md`. Also run `make format` 
so spotless is happy with the rewrapped comment lines.
   - The backporting text must not claim that 1.2 and later branches use `dev`. 
`branch-1.2` was cut before the rename and has `latest`. Say that branches cut 
after the rename use `dev`, and that git's rename detection usually carries a 
cherry-pick across it. The release branch freeze script, 
`dev/generate-release-docs.sh`, also has to work on both names.
   - Mermaid diagrams need `mmdc` to render. I had it missing locally, so check 
that CI or the first publish draws them.
   
   ## How I validated it
   
   - Ran the full `docs/build.sh` (clones the release branches, the 
`generate-docs` Maven profiles, Sphinx, then the alias step). `latest/` was 
byte-identical to the newest release directory (`diff -rq`).
   - Checked about 165,000 `href` and `src` attributes in the built HTML for 
existence. The only misses were unrendered theme template fragments in 
`_static/webpack-macros.html`.
   - Checked that `user-guide/installation.html` still redirects to 
`latest/installation.html`, that the home page hero links resolve, and that a 
page under `latest/` links to `../dev/index.html` from the sidebar.
   - Checked that no conditional block markers or literal `$COMET_VERSION` 
leaked into `dev/` or `latest/`.
   - Not checked: how the sidebar renders and highlights from a page under 
`latest/`. Look at that on the first publish.
   
   ## Suggested approach for whoever picks this up
   
   1. Do the rename with `git mv` on current `main` in a quiet period. It 
touches about 120 files, so it conflicts with any open docs PR. Announce it on 
the PR or dev list first.
   2. Set `LATEST_RELEASED_VERSION` to the newest release at that time (1.2.0 
now) and update `index.md` and `older-versions.md` to match.
   3. Reapply the generator, `conf.py`, `CometDocs`, plan stability and docs 
changes from the branch above.
   4. Run `docs/build.sh` locally and repeat the link check.
   5. Open the PR with the `documentation` label.
   
   ## Acceptance criteria
   
   - `user-guide/latest/` serves the newest release and `user-guide/dev/` 
serves main.
   - A released jar's config descriptions and fallback messages link to its own 
`user-guide/<major.minor>/` pages, and a snapshot build links to `dev`.
   - The user guide index states that only versioned URLs are stable.
   - Promoting a release at release time is a one-place change in 
`generate-versions.py`, documented in `release_process.md`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to