This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 08980bdbf7c [doc-tools](skill) Add doris-release-docs skill (#4160)
08980bdbf7c is described below
commit 08980bdbf7cf1c1238970467f24f6b86c2b93b1e
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Mon Sep 21 18:31:57 2026 +0800
[doc-tools](skill) Add doris-release-docs skill (#4160)
## What this adds
`doc-tools/skills/doris-release-docs/` — a skill for auditing and
updating the user documentation of a Doris release from the commit diff
between two refs (for example `4.1.3..4.1.4-rc04`). It is distilled from
the 4.1.4 round (#4123), including every mistake made there and the step
that now prevents it.
## Workflow
1. **Ask first** which doc branch gets the Chinese version (dev / 4.x /
...) and where to sync afterwards.
2. **Extract the hard user-facing surface** deterministically: FE/BE/MS
configs, session variables, grammar, builtin functions, system tables,
SHOW/ADMIN commands, HTTP endpoints, metrics, property analyzers,
shipped `conf/` files, dependency versions, build modules. This is the
baseline that subagent reports can add to but never override.
3. **Classify every production commit** with parallel subagents (split
by touched paths — the `[fix](test)` commit that added a session
variable is the reason subject tags are ignored).
4. **Decide per change**: FIX / ADD / NEW / NOTE / SKIP / ASK — feature
commits with no obvious home, large new capability areas and
"open-source build has no implementation" cases are collected into one
question for the user.
5. **Write the Chinese docs** for the chosen branch, run the static
validator, open a PR.
6. **After approval, sync** to the other branches and English — with a
master-vs-tag difference table first, because the dev docs describe
master (four variables, three defaults, the Paimon version and a module
rename differed in the 4.1.4 round).
## Iron rules
- Anything a user can perceive needs a doc.
- Every new feature, behavior change, default change or removal carries
a "since version X.Y.Z" note, verified against the release tags, on
every release line it shipped on.
- Every feature commit needs a documented home, or a question to the
user — never a silent "no location found".
- Facts come only from the code at the release tag: defaults,
mutability, gating conditions, error strings, column names, privilege
checks, counting semantics.
- Unreleased capabilities are recorded, not documented.
- Sidebars are shared across locales, so a new page waits for its
English twin; removed features keep their page as a removal notice with
a migration mapping; no `yarn build`, static checks only.
## Scripts
| Script | Purpose |
| --- | --- |
| `surface-diff.sh` | per-area diffs between two refs plus an identifier
list |
| `split-commits.sh` | production vs test-only commits, cut into batches
|
| `check-version-claims.py` | which tags contain an identifier; `--path`
for FE/BE name clashes |
| `compare-refs.py` | default value, mutability and presence of
configs/variables on two refs (handles both `@VariableMgr.VarAttr` and
master's `@VarAttrDef.VarAttr`) |
| `validate-docs.py` | front matter, relative links, bare JSX-like tags
in `.mdx` (import-aware), CJK in English trees, forbidden identifiers,
table column consistency |
All five were run against the real 4.1.3..4.1.4-rc04 range and against
the merged history of #4123; the validator reports 3 genuine
pre-existing issues over 488 files and nothing else.
## References
- the two subagent prompts (classification, cross-tree port)
- writing conventions: version-note wording per situation, FE/BE config
entry formats, the session-variable page format, removal pages, MDX
constraints, and a "claim → how to verify it" table
- a report template
- `pitfalls.md`: the 21 concrete mistakes from the 4.1.4 round
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01GRr7pw6ymrf99x3DUr2zxg
---------
Co-authored-by: morningman <[email protected]>
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
---
doc-tools/skills/doris-release-docs/SKILL.md | 204 +++++++++++++++++++++
.../references/classify-agent-prompt.md | 85 +++++++++
.../references/doc-conventions.md | 136 ++++++++++++++
.../doris-release-docs/references/pitfalls.md | 78 ++++++++
.../references/port-agent-prompt.md | 91 +++++++++
.../references/report-template.md | 38 ++++
.../scripts/check-version-claims.py | 74 ++++++++
.../doris-release-docs/scripts/compare-refs.py | 108 +++++++++++
.../doris-release-docs/scripts/split-commits.sh | 29 +++
.../doris-release-docs/scripts/surface-diff.sh | 65 +++++++
.../doris-release-docs/scripts/validate-docs.py | 139 ++++++++++++++
11 files changed, 1047 insertions(+)
diff --git a/doc-tools/skills/doris-release-docs/SKILL.md
b/doc-tools/skills/doris-release-docs/SKILL.md
new file mode 100644
index 00000000000..b9f6c52a287
--- /dev/null
+++ b/doc-tools/skills/doris-release-docs/SKILL.md
@@ -0,0 +1,204 @@
+---
+name: doris-release-docs
+description: Audit and update the user documentation for an Apache Doris
release from the commit diff between two refs (for example 4.1.3..4.1.4-rc04).
Extract the hard user-facing surface (FE/BE/MS configs, session variables,
grammar, functions, system tables, HTTP endpoints, metrics, shipped conf files,
dependency versions), classify every commit with parallel subagents, then add,
fix or create docs under three iron rules — anything a user can perceive needs
a doc, every new feature or [...]
+---
+
+# Release docs from the commit diff
+
+Goal: given `FROM_REF..TO_REF` (for example `4.1.3..4.1.4-rc04`), make sure
that **every user-perceivable
+change has documentation, every new feature or behavior change is
version-annotated, and every feature
+commit has a documented home**, then report the result to the user in three
buckets: done, needs your
+decision, deliberately skipped.
+
+Paths used below (confirm at the start):
+
+| Variable | Meaning |
+| --- | --- |
+| `DORIS` | a local `apache/doris` clone, read-only, with the release tags and
an `upstream` remote for `master` |
+| `SITE` | this repository (doris-website) |
+| `ZH` | the Chinese doc tree the user chose: `4.x` ->
`$SITE/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x`, `dev` ->
`.../current` (**ask first, never assume**) |
+| `OUT` | a scratch directory for diffs, batches and subagent reports |
+
+Scripts live in `doc-tools/skills/doris-release-docs/scripts/`, references in
`references/`.
+
+```bash
+S=doc-tools/skills/doris-release-docs/scripts
+$S/surface-diff.sh $DORIS 4.1.3 4.1.4-rc04 $OUT/surface # hard
user-facing surface
+$S/split-commits.sh $DORIS 4.1.3 4.1.4-rc04 $OUT/commits 30 #
production vs test-only, batches
+$S/check-version-claims.py $DORIS --tags 4.0.7 4.0.8 4.1.3 4.1.4-rc04 --
<id>... # before writing any version note
+$S/compare-refs.py $DORIS 4.1.4-rc04 upstream/master --from-file
$OUT/surface/identifiers.txt # before syncing dev
+$S/validate-docs.py $SITE --base upstream/master --forbidden
$OUT/forbidden.txt # before every commit
+```
+
+---
+
+## 0. Iron rules
+
+1. **A commit that adds a capability must have a documented home.** A subject
tagged `feat` / `feature`,
+ or a diff that introduces a new user-facing surface (new syntax, new
endpoint, a new group of configs,
+ a new data source, a new runtime mechanism such as cross-compute-group peer
cache reads) gets
+ documentation: a new section on an existing page, or a new page. If there
is no obvious home, or you
+ are not sure where it belongs, **ask the user** — never silently park it in
a "no location found" list.
+2. **New features, behavior changes, default changes and removals are always
version-annotated**
+ ("Supported since version 4.1.4"). Every version number must be checked
against the tags with
+ `check-version-claims.py`; a change that shipped on several release lines
(4.0.x and 4.1.x) is
+ annotated for each line.
+3. **Anything a user can perceive needs a doc**: configs, session variables,
SQL syntax, functions, system
+ table / SHOW columns, HTTP endpoints, metrics and profile counters, error
messages, privilege
+ requirements, type mappings, support matrices, dependency versions, and the
shipped `conf/` files,
+ start scripts and jar directory names. Pure refactors, logging and
optimizations with no interface
+ change are not documented.
+4. **Facts come only from the code at `TO_REF`** — not from commit messages,
PR titles, subagent
+ reports or the existing docs. Default values, mutability, gating
conditions, error strings, column
+ names, privilege checks and counting semantics are verified with `git show`
/ `git grep` before they
+ are written (see `references/doc-conventions.md` §7).
+5. **Unreleased capabilities are not documented, but they are recorded.**
Signals: the PR says the
+ open-source build only ships a framework / no-op, the user names the
feature, or it is visibly
+ half-finished. For a large new capability area, **ask the user before
writing** — writing a whole
+ chapter and then pulling it is wasted work.
+6. **The dev trees describe `master`, not `TO_REF`.** Before syncing to dev,
run `compare-refs.py` and
+ build a difference table; master can lack backported variables, use
different defaults, dependency
+ versions, module names and error strings. Record the master SHA the table
was built from.
+7. **Sidebars are shared between locales.** While only the Chinese tree is
written, a new page is not
+ added to the sidebar (the English site would fail to build); link to it
from an existing page and add
+ the sidebar entry once every tree has the file.
+8. **Never delete a page.** A removed feature keeps its page, rewritten as a
removal notice plus a
+ migration mapping plus the historical usage.
+9. **No `yarn build`.** The only verification is `validate-docs.py` (front
matter, relative links, bare
+ JSX-like tags in `.mdx`, Chinese text in English trees, forbidden
identifiers, table column counts).
+10. **First question, before touching anything: which doc branch gets the
Chinese version** (dev / 4.x /
+ 3.x ...). The cadence is fixed: **write that one Chinese branch -> open a
PR -> the user reviews and
+ approves -> then sync to the other branches and to English.** The sync
scope is confirmed with the
+ user at review time (for example whether a 4.x change also goes to dev or
3.x). Every PR description
+ states the current sync status.
+11. **Subagents only read the Doris repo** and only write their own report
file. The main agent
+ aggregates, verifies facts and writes the docs (or dispatches per-file
batches, see §6).
+
+---
+
+## 1. Preflight
+
+```bash
+cd $DORIS && git fetch upstream master --tags
+git tag -l '4.1.*' # both FROM and TO must exist
+git log --oneline FROM..TO | wc -l
+```
+
+**Ask the user two things before extracting anything** (one question, all at
once):
+
+1. Which doc branch gets the Chinese version: `4.x`
(`versioned_docs/version-4.x` +
+ `i18n/.../version-4.x`), `dev` (`docs/` + `i18n/.../current`) or another
active version
+ (`versions.json` lists them). This sets `$ZH` and decides which release
line version notes refer to.
+2. Which branches and locales to sync to after approval (default: the same
version in English plus dev in
+ both locales; the user may add or remove).
+
+Restate the commit range, the target version and the active version list in
the question so a wrong
+assumption is easy to spot. Put every topic the user says "not yet" to into
`$OUT/forbidden.txt` (one
+identifier or regex per line) for `validate-docs.py`.
+
+## 2. Extract the hard surface (deterministic, no judgment involved)
+
+```bash
+$S/surface-diff.sh $DORIS FROM TO $OUT/surface
+less $OUT/surface/summary.txt
+```
+
+Read the `fe-config` / `session-var` / `be-config` / `ms-config` / `grammar` /
`functions` /
+`schema-tables` / `conf-files` / `deps` sections and list, directly from the
`+`/`-` lines: additions,
+removals, default changes, mutability changes, grammar additions and removals,
function / TVF additions
+and removals, system-table columns, shipped conf lines, dependency versions.
This is the **baseline
+list** — subagent reports can add to it but never override it.
+
+`$OUT/surface/identifiers.txt` holds the config and variable names found in
the diffs; feed it to
+`check-version-claims.py` and `compare-refs.py` with `--from-file`.
+
+## 3. Classify every commit (parallel subagents)
+
+```bash
+$S/split-commits.sh $DORIS FROM TO $OUT/commits 30 # splits by touched
PATHS, never by [subject tag]
+```
+
+Start one `general-purpose` subagent per `batch-NN`, all at once, with the
prompt in
+`references/classify-agent-prompt.md`. Each writes `$OUT/commits/report-NN.md`
and replies with a short
+summary. When all are in:
+
+```bash
+for f in $OUT/commits/report-*.md; do awk '/^## NONE/{exit} /^## /{print}
/Impact|Likely doc area|Flags/{print}' $f; done
+```
+
+gives a HIGH / MEDIUM / LOW index. **Re-check every default in the reports
against `summary.txt`** —
+an intermediate commit can set a value that a later commit reverts.
+
+## 4. Decide and ask
+
+Merge the index and the baseline list into one table and decide per row: `FIX`
an existing page /
+`ADD` to an existing page / `NEW` page / `NOTE` a version note only / `SKIP`
internal change / `ASK` the
+user.
+
+Always `ASK` for: feature commits with no obvious home; large new capability
areas; PRs that say the
+open-source build has no implementation; anything a subagent flagged.
**Collect every ASK into one
+question** instead of interrupting repeatedly. Whatever the user declines goes
into `forbidden.txt`.
+
+Finding a home: `grep -rln <identifier> $ZH`; failing that, the page for the
same topic (configs in
+`admin-manual/config/*-config.md`, session variables in
`sql-manual/basic-element/session-variables.md`
+plus the owning feature page, lakehouse in `lakehouse/catalogs/*`, load in
`data-operate/import/*`,
+upgrade impact in `admin-manual/cluster-management/upgrade.md`). Before
creating a page run
+`find $SITE/docs $SITE/i18n -name '<slug>*'` — dev may already have a better
version; reuse it and fix
+the version note.
+
+## 5. Write the Chinese docs (only the branch chosen in §1, `$ZH`)
+
+Follow `references/doc-conventions.md` (version-note wording, FE/BE config
entry formats, the session
+variable page, removal pages, MDX constraints, the fact-checking table). The
most common mistakes:
+
+- Version numbers: run `check-version-claims.py` first, annotate every release
line, and audit the
+ existing notes on the page while you are there.
+- Claims: read the diff body, not the title; find the gating condition; read
the annotation for
+ mutability; read the call arguments for counting semantics; copy error
strings literally; copy
+ property names from the code constants.
+- Tables: append new rows at the end; never put a blockquote between table
rows.
+- When a rule changes, replace the examples on the page that no longer satisfy
it.
+- Upgrade guide: add an "Upgrading to X.Y.Z" section with three tables —
removed interfaces and
+ syntax / default and behavior changes / deployment and config-file changes —
each row
+ `Change | Affected scope | What to do`.
+- Session variable page: overview table (added / default or semantics changed
/ removed), one section
+ per variable, a removed-variables table.
+
+Then:
+
+```bash
+$S/validate-docs.py $SITE --base upstream/master --forbidden $OUT/forbidden.txt
+```
+
+Write the review report from `references/report-template.md` into
`plan-doc/doris-<ver>-doc-review.md`
+(**not committed to the PR**), create a branch, commit, `gh pr create`. The PR
description must contain:
+the range and method, the list of statements in the existing docs that were
wrong and are now fixed, what
+was deliberately left out, new pages awaiting a sidebar entry, and changes
with no home.
+
+## 6. After approval: sync to the other branches and to English
+
+Enter this section only when the user says the Chinese version is approved;
confirm the sync scope from
+§1. The targets are usually the same version in English plus dev in both
locales (source 4.x), or dev in
+English plus 4.x in both locales (source dev).
+
+1. `cd $DORIS && git fetch upstream master`; note the SHA.
+2. `compare-refs.py $DORIS TO upstream/master --from-file
$OUT/surface/identifiers.txt`, plus feature
+ presence checks (`git grep -l <distinctive string> upstream/master | grep
-v regression-test` —
+ **no narrow pathspecs**, master has been refactored), assembled into a
difference table.
+3. Fill `references/port-agent-prompt.md` into `$OUT/sync-context.md`
(difference table, forbidden
+ topics, master SHA) and start 6–8 file-batch subagents; each ports its
files into all target trees;
+ large files (`upgrade.md`, `iceberg-catalog.mdx`, `fe-config.md` +
`be-config.md`) get a batch of
+ their own.
+4. The main agent does: the other-language / other-branch versions of new
pages (dev versions edited per
+ the difference table), sidebar entries, and pages that exist in one branch
but not another although
+ the feature ships in this version.
+5. A subagent reporting "the SOURCE is wrong" means fixing every tree at once;
"dev already has a better
+ version" means adopting it.
+6. Run `validate-docs.py` over everything; re-run `compare-refs.py` right
before pushing (master moves);
+ update the difference table and the corrections list in the PR description.
+
+## 7. Wrap up
+
+- Append the round's new lessons to `references/pitfalls.md`.
+- The report stays in `plan-doc/`, out of the PR.
+- Note the PR number, branch, master SHA used for the comparison and the
topics the user declined.
diff --git
a/doc-tools/skills/doris-release-docs/references/classify-agent-prompt.md
b/doc-tools/skills/doris-release-docs/references/classify-agent-prompt.md
new file mode 100644
index 00000000000..0e9189dacc1
--- /dev/null
+++ b/doc-tools/skills/doris-release-docs/references/classify-agent-prompt.md
@@ -0,0 +1,85 @@
+# Subagent prompt: classify the doc impact of each commit
+
+How to use: pass the block below as the prompt of a `general-purpose` agent,
filling the `{{...}}`
+placeholders. One agent per `batch-NN` file (about 30 commits); start all
batches at once. The agent
+only reads the Doris repo, writes `{{OUT_DIR}}/report-{{NN}}.md`, and replies
with a summary of at
+most 25 lines.
+
+The main agent MUST do two things with every report and may not take it at
face value:
+1. Re-check every default value / behavior against the final code at
`{{TO_REF}}`. An intermediate
+ commit can set a value that a later commit reverts
(`enable_expr_zonemap_filter` went `true` then
+ back to `false` inside one release).
+2. "HIGH" does not mean "write it" and "LOW" does not mean "skip it"; iron
rules 1–3 decide.
+
+---
+
+You are analyzing Apache Doris commits for the {{VERSION}} release, to
determine what user-facing
+documentation needs to be added / corrected on the Doris docs website.
+
+Repo: `{{DORIS_REPO}}` — **READ ONLY**. Only run read-only git commands (`git
show`, `git log`,
+`git diff`, `git grep`). NEVER checkout, commit, stash, reset, fetch, or edit
any file in that repo.
+
+Your batch of commits: `{{OUT_DIR}}/batch-{{NN}}` (format: `<short-hash>
<subject>` per line).
+
+For EACH commit in your batch:
+1. `git show --stat <hash>` to see files touched.
+2. If it only touches tests (`regression-test/`, `be/test/`, `*/src/test/`),
CI (`.github/`), docker,
+ or build scripts -> mark NONE and move on quickly.
+3. Otherwise read the relevant parts of the diff (`git show <hash> --
<paths>`) and the full commit
+ message (`git log -1 --format=%B <hash>`). The subject tag is NOT a
reliable signal: a
+ `[fix](test)` commit has added a session variable before. Judge by the diff.
+4. Decide whether the change is USER-VISIBLE, i.e. a Doris user reading the
docs would need to know
+ about it. User-visible things include:
+ - New / changed / removed SQL syntax or statements (parser .g4, Command
classes)
+ - New / removed / renamed SQL functions, or changed function semantics /
return type /
+ NULL handling / accepted argument types
+ - New / changed / removed FE config (`Config.java`), BE config
(`be/src/common/config.*`,
+ `be/src/cloud/config.*`), MS config, or session variables
(`SessionVariable.java`) —
+ INCLUDING default-value changes and mutability changes
(`@ConfField(mutable = ...)`)
+ - New / changed table properties, catalog properties, load / job properties
(`PROPERTIES(...)` keys)
+ - Observable behavior changes: different query results, error messages
users hit, privilege
+ requirements, type-support matrices, supported data source / file format
capability
+ - New / removed system tables, information_schema columns, SHOW statement
output columns,
+ HTTP API endpoints, metrics, profile counters
+ - Limits / restrictions added or lifted (e.g. "now supports X on Y table
type")
+ - Deprecations / removals
+ - Shipped config files (`conf/fe.conf`, `conf/be.conf`), start scripts,
module / jar renames,
+ dependency version bumps that docs quote (Paimon, Iceberg, Hive shade,
Arrow ...)
+ - Bug fixes where the OLD documented behavior is now wrong, or where the
doc should state a
+ version-specific difference
+5. Pure internal refactors, logging, memory / perf optimizations with no
observable interface change,
+ and flaky-test fixes -> NONE.
+
+For every non-NONE commit, ALSO record verbatim from the diff (not
paraphrased):
+ - exact identifier names and their default values as they stand at the END
of the commit
+ - exact error-message strings (copy the string literal; note if it is
concatenated across lines)
+ - the exact gating condition of any "on upgrade" / "when enabled" behavior
(e.g. a
+ `variable_version < 400` check means it only fires for 3.x -> 4.x
upgrades)
+ - whether the PR checklist says "Does this need documentation" and whether
it links a doc PR
+ - anything the PR text says is NOT included in the open-source build (e.g.
"OSS no-op")
+
+Output: write a markdown report to `{{OUT_DIR}}/report-{{NN}}.md` with one
section per non-NONE
+commit:
+
+```
+## <hash> <subject>
+- **PR**: #<number(s)>
+- **Impact**: HIGH | MEDIUM | LOW
+- **What changed (user-visible)**: <precise description with exact names /
defaults / strings>
+- **Doc action**: ADD | FIX | NEW-PAGE | NOTE — <what the doc should say>
+- **Likely doc area**: <e.g. lakehouse/catalogs/iceberg-catalog,
admin-manual/config/fe-config, ...>
+- **Flags**: <"feature commit — needs a doc home", "PR says no docs needed",
"OSS no-op", "unsure">
+```
+
+End the report with:
+```
+## NONE (no doc impact)
+<hash> <subject>
+...one line each
+```
+
+Be precise and factual — quote exact identifier names, default values and
error strings from the
+diff. Do not speculate about docs you have not read; your job is only to
characterize the code change.
+
+Finally, reply with a SHORT summary (<= 25 lines): counts of HIGH / MEDIUM /
LOW / NONE, then a
+bullet list of only the HIGH and MEDIUM items (hash + one-line what changed),
then any Flags.
diff --git a/doc-tools/skills/doris-release-docs/references/doc-conventions.md
b/doc-tools/skills/doris-release-docs/references/doc-conventions.md
new file mode 100644
index 00000000000..02d617dfddd
--- /dev/null
+++ b/doc-tools/skills/doris-release-docs/references/doc-conventions.md
@@ -0,0 +1,136 @@
+# doris-website writing conventions for release docs
+
+The existing wording in the repository takes precedence over this file. This
file pins down the
+things that came up repeatedly in the 4.1.4 round and are easy to get wrong.
+
+## 1. Version notes (how iron rule 2 is written)
+
+| Situation | Chinese | English |
+| --- | --- | --- |
+| New feature / config / variable / column | `自 4.1.4 版本起支持。` or, in a table,
`4.1.4 新增` | `Supported since version 4.1.4.` / `Added in 4.1.4` |
+| Default value changed | `默认值：8（4.1.4 之前为 4）` plus one sentence on why |
`Default: 8 (4 before 4.1.4)` |
+| Behavior changed / old doc described behavior that was a bug | `:::caution
版本行为变更（4.1.4）` block stating before -> after and the affected scope |
`:::caution Behavior change (4.1.4)` |
+| Shipped on several release lines | `Doris 4.0 系列自 4.0.8 版本起、4.1 系列自 4.1.4
版本起` | `in version 4.0.8 in the Doris 4.0 series and in version 4.1.4 in the
4.1 series` |
+| Removal | keep the page; top `:::caution` with the removal, the error users
now see, the replacement and a migration mapping table; add "(before 4.1.4)" to
the historical headings | same |
+| Experimental | `4.1.4 新增（实验性）` | `Added in 4.1.4 (experimental)` |
+
+Run `scripts/check-version-claims.py` before writing a version number; check
every active release
+line. The X in "since X" is **the first official release on each line that
contains the change**, not
+the date the PR merged to master.
+
+Audit the notes already on the page too. In the 4.1.4 round
`require_partition_filter` was
+documented as "4.1 series since 4.1.2" but was not in 4.1.2 or 4.1.3;
multimodal EMBED said 4.1.5
+but shipped in 4.1.4; six "since 4.0.8" notes only reached the 4.1 line in
4.1.4.
+
+## 2. Config entry formats
+
+**fe-config.md (English)**
+```
+#### `config_name`
+
+Default: false
+
+Is it possible to dynamically configure: true
+
+Is it a configuration item unique to the Master FE node: true
+
+Added in 4.1.4. <one-sentence purpose>
+
+<value semantics, example, caveats>
+```
+"dynamically configure" comes from `@ConfField(mutable = true)`, "Master only"
from `masterOnly = true`.
+A bare `@ConfField` is NOT dynamically configurable. Chinese: `默认值：` /
`是否可以动态配置：` /
+`是否为 Master FE 节点独有的配置项：`.
+
+**be-config.md (English)**
+```
+#### `config_name`
+
+* Type: int32
+* Description: Added in 4.1.4. <purpose>
+* Default value: 10
+```
+A `DEFINE_m*` prefix (`mInt32`, `mBool`) means the value can be changed at
runtime; say so in the
+description. Chinese: `* 类型：` / `* 描述：` / `* 默认值：`.
+
+Both files are **curated, not exhaustive** (BE documented 205 of 749 configs
at 4.1.4). Document what
+users tune; pure internal knobs (pool sizes, shard counts) may be left out but
must be listed in the
+report so the user can decide.
+
+Put a new entry in its section (`### Metadata and cluster management` / `###
Service` /
+`### Query engine` / `### Load and export` / `### Storage` / `### External
table` /
+`### Compute-storage decoupled mode` ...) next to related entries; do not
append everything to the end.
+
+## 3. Session variables
+
+The 4.x docs had no complete session-variable reference; the 4.1.4 round
created
+`sql-manual/basic-element/session-variables.md` with an "under construction"
notice at the top and
+only the variables that release touched. Later releases append to it in this
format:
+
+```
+### `variable_name`
+
+| Item | Value |
+| --- | --- |
+| Type | Boolean |
+| Default | `true` |
+| Version | Added in 4.1.4 |
+
+<description>
+```
+plus a "What changed in X.Y.Z" overview table (added / default or semantics
changed / removed) and a
+"Removed variables" table. A variable also goes into its owning feature page
(runtime filter variables
+into runtime-filter.md, and so on), with links both ways.
+
+## 4. Removed features, endpoints, syntax
+
+- **Never delete the page** (sidebars are shared across locales; a missing id
breaks the build). Turn
+ it into: removal notice + migration mapping + historical usage.
+- Pages that used it in examples or tutorials switch to the replacement and
note "before 4.1.4 you
+ could also use X".
+- Add a row to the upgrade guide.
+
+## 5. New pages
+
+- `versioned_sidebars/version-4.x-sidebars.json` and `sidebars.ts` are shared
by both locales:
+ **no sidebar entry while the English page does not exist**, or the English
build fails. During the
+ Chinese-first phase create only the Chinese page, link to it from an
existing page, and list it in
+ the report as "awaiting sidebar"; add the entry when syncing English.
+- When the id goes into the sidebar, all four trees (zh 4.x / en 4.x / zh dev
/ en dev) must have the
+ file.
+- One statement per page is the convention, but a small derived statement
(`ADMIN COMPACT TABLET`
+ next to `COMPACT TABLE`) can share the parent page; say so in the report.
+
+## 6. Docusaurus / MDX constraints
+
+- Front matter is JSON (a few key-features / community pages use YAML);
`"language"` is `zh-CN` in
+ Chinese files and `en` in English files.
+- `markdown.format: 'detect'`: `.md` is CommonMark, `.mdx` is MDX. **In
`.mdx`, any `<xxx>` outside
+ a code fence or inline code is parsed as JSX** — placeholders go in
backticks or fenced blocks.
+- Relative links work with or without the extension; anchors are heading slugs
(Chinese headings keep
+ the Chinese: `#支持的运算与-cast-规则`).
+- Never put a blockquote or blank line between table rows — it splits the
table (the MaxCompute
+ property table broke this way); append rows at the end or keep the version
order.
+- Admonitions: `:::caution Title`, `:::info Title`, `:::tip Title`, with a
blank line before and after.
+- No `yarn build`; `scripts/validate-docs.py` is the static check.
+
+## 7. Where facts come from (iron rule 4)
+
+Before writing any claim, verify it in the Doris repo at the **final tag**:
+
+| Claim | How to verify |
+| --- | --- |
+| Default value | `git show <tag>:<file>`, read the field initializer; never
trust the commit message |
+| Dynamically configurable | `@ConfField(mutable = ...)`; BE: the `DEFINE_m*`
prefix |
+| "What happens on upgrade" | find the gating condition (e.g.
`variable_version < 400`) before concluding |
+| Error message | copy the string literal; join fragments concatenated across
lines |
+| Column names / count | the definition (`SchemaTable.java`,
`InsertJob.SCHEMA`, ...) |
+| Privileges | the `checkGlobalPriv` / `checkTblPriv` call sites |
+| Counting semantics | the call arguments (`getAllClusterBackends(false)` is
ALL backends, not the live ones) |
+| Dependency version | `<xxx.version>` in `fe/pom.xml` |
+| Property name | copy from the code constant, never retype
(`mc.enable.namespace.schema` was once written with underscores) |
+
+## 8. Examples must match the rule
+
+When a rule changes (for example the floating-point output format), recompute
or replace the examples
+already on the page; an example that contradicts the rule it sits under is
worse than no example.
diff --git a/doc-tools/skills/doris-release-docs/references/pitfalls.md
b/doc-tools/skills/doris-release-docs/references/pitfalls.md
new file mode 100644
index 00000000000..21479e77f42
--- /dev/null
+++ b/doc-tools/skills/doris-release-docs/references/pitfalls.md
@@ -0,0 +1,78 @@
+# Lessons from the 4.1.4 round (each maps to a step in SKILL.md)
+
+Recorded as "what went wrong -> how it was caught -> how it is prevented now".
Append after every
+release.
+
+## A. Classification
+
+1. **Filtering by the `[tag]` in the subject misses changes.** `[fix](test)
stabilize the flaky
+ shuffle_left_join` added the session variable
`bucket_shuffle_downgrade_ratio`;
+ `[fix](regression) Adjust large TTL cache regression case` added the
upper-bound check on
+ `file_cache_ttl_seconds`. -> `split-commits.sh` splits by touched paths
only.
+2. **A subagent's default value can be an intermediate state.**
`enable_expr_zonemap_filter` was
+ `true` in `009bcf44b3d` and back to `false` in `5e6f47d7cdc`. -> the main
agent re-checks against
+ the final diff from `surface-diff.sh` and `compare-refs.py`.
+3. **"Does this need documentation: No" in the PR does not mean no docs**
(`ADMIN COMPACT TABLET`
+ and peer read both had it unchecked), but "the open-source build only ships
a no-op" (TLS) is a
+ real reason not to write. -> record both signals and let the user decide.
+4. **The same identifier can exist once in FE and once in BE.**
+ `enable_group_commit_streamload_be_forward` existed as an FE config since
4.0.7; the BE config
+ arrived in 4.0.8 / 4.1.4. -> `check-version-claims.py --path` restricts the
search to one file.
+
+## B. Writing (factual errors that were made)
+
+5. **Treating the commit title as the conclusion.** `5ea5dd73e13`
"replication_num /
+ replication_allocation" was documented as "now mutually exclusive"; the
code removes the other
+ legacy property when one is modified. -> read the diff body before writing
a behavior claim.
+6. **Missing the gating condition.** The `enable_nereids_distribute_planner`
refresh sits inside
+ `if (variableVersion < 400)`, i.e. only 3.x -> 4.x upgrades; it was
documented as "every
+ upgrade". -> conventions §7, "what happens on upgrade".
+7. **Mutability inverted twice.**
`enable_forward_group_commit_stream_load_to_follower` is
+ `mutable = true`; `default_get_version_from_ms_timeout_second` is a bare
`@ConfField`. ->
+ `compare-refs.py` now prints mutability.
+8. **Embellishing a count.** `BackendNum` was described as "live, healthy
backends"; the code calls
+ `getAllClusterBackends(false)` = all of them. -> read the argument, don't
guess.
+9. **Rule changed, example kept.** After the shortest-round-trip float format
landed, the old
+ `cast('12345678' as float) -> 1.234568e+07` example stayed under the new
rule and contradicted
+ it. -> conventions §8.
+10. **A mid-table insert with a blockquote split the table** (MaxCompute
properties). ->
+ `validate-docs.py` checks column counts; new rows go at the end.
+11. **A retyped property name** (`mc.enable_namespace_schema` for
`mc.enable.namespace.schema`).
+ -> copy from the code constant.
+12. **Inherited "since 4.0.8" notes without checking the 4.1 line.** Six of
them were 4.1.4 on that
+ line; `require_partition_filter` "since 4.1.2" was 4.1.4; multimodal EMBED
"4.1.5" was 4.1.4. ->
+ every version number goes through `check-version-claims.py`.
+13. **Documented unreleased capabilities.** Whole chapters on Paimon writes
and Variant V2 were
+ written and then withdrawn. -> ask before writing a large new capability
area (SKILL.md §4).
+14. **Wrote new function pages although dev already had better ones.** The dev
`parse_to_variant`
+ page was more complete. -> `find` all four trees before creating a page;
reuse and fix the
+ version note.
+
+## C. Sync
+
+15. **dev is not a copy of 4.x.** master lacked four variables
(`enable_external_scan_task_reuse`,
+ `file_split_size_on_fe/be`, `external_meta_cache_max_weight`), three
defaults differed
+ (`enable_expr_zonemap_filter`, `max_scanners_concurrency`,
`meta_service_rpc_rate_limit_enabled`),
+ Paimon was 1.3.1 vs 1.4.2, `paimon-scanner` was not renamed, Iceberg
`ALTER TABLE SET` and the
+ database-property matrix did not exist, several error strings differed,
`SHOW TABLETS` without
+ `ORDER BY` behaved differently. -> run `compare-refs.py` first and paste
the table into the port
+ prompt.
+16. **master moves.** `external_meta_cache_max_weight` was absent from master
on sync day and
+ landed hours later (#67726). -> record the master SHA in the report / PR;
re-run
+ `compare-refs.py` before pushing.
+17. **Narrow pathspecs miss on master.** Connectors moved to
`fe/fe-connector/`; searching
+ `fe/fe-core/src/main/java/` for `osstables` returned nothing. -> search
the whole tree, then drop
+ test hits.
+18. **dev had already structured the same topic differently.** Schema change
is split into
+ `schema-change-mysql.md` / `schema-change-postgresql.md` on dev. -> "find
the equivalent place
+ and adapt", never paste. Conversely, a page that dev has and 4.x lacks
+ (`schema-change-mysql.md`) is ported down when the feature ships in this
version.
+19. **Sidebars are shared.** Adding an entry in the Chinese-first phase breaks
the English build. ->
+ link from an existing page first; add the entry when English exists.
+
+## D. Process
+
+20. **Chinese first -> user review -> English + other branches** is the
requested cadence; every PR
+ description states the sync status.
+21. The report file (`plan-doc/doris-<ver>-doc-review.md`) stays out of the
PR; its summary goes into
+ the PR description.
diff --git
a/doc-tools/skills/doris-release-docs/references/port-agent-prompt.md
b/doc-tools/skills/doris-release-docs/references/port-agent-prompt.md
new file mode 100644
index 00000000000..b1e0ae9337c
--- /dev/null
+++ b/doc-tools/skills/doris-release-docs/references/port-agent-prompt.md
@@ -0,0 +1,91 @@
+# Subagent prompt: sync the approved Chinese branch to the other branches and
to English
+
+How to use: fill the placeholders and save the block below as
`{{OUT_DIR}}/sync-context.md`. Then
+start subagents in file batches (5–10 files each; large files such as
`upgrade.md`,
+`iceberg-catalog.mdx`, `fe-config.md` + `be-config.md` get a batch of their
own). Each batch prompt
+only needs "Read sync-context.md in full first, then handle these files: ..."
plus batch-specific hints.
+
+The source and target trees come from the answer to SKILL.md §1 — do not
assume the source is 4.x:
+- source = Chinese 4.x -> targets are usually English 4.x, Chinese dev,
English dev
+- source = Chinese dev -> targets are usually English dev, Chinese 4.x,
English 4.x (then it is the
+ SOURCE that describes master, and rule 4 must verify against the 4.x tag
instead)
+The table below is written for source = 4.x; swap the rows for a dev source
and replace "master" in
+rule 4 with the release tag.
+
+**Before starting any subagent the main agent runs** `scripts/compare-refs.py`
and pastes the result
+into "Verified differences". The subagents' own checks supplement it; they do
not replace it.
+
+---
+
+# Doris {{VERSION}} doc sync — shared context
+
+## Goal
+
+The Chinese {{SOURCE_LABEL}} docs have been updated for the Doris {{VERSION}}
release and approved by
+the user. Port each change to the other doc trees.
+
+Repo: `{{WEBSITE_REPO}}` (branch `{{BRANCH}}`).
+
+| Tree | Path prefix | Language |
+| --- | --- | --- |
+| **SOURCE** {{SOURCE_LABEL}} (reviewed and approved) | `{{SOURCE_PREFIX}}` |
Chinese |
+| target {{TARGET1_LABEL}} | `{{TARGET1_PREFIX}}` | English |
+| target {{TARGET2_LABEL}} | `{{TARGET2_PREFIX}}` | Chinese |
+| target {{TARGET3_LABEL}} | `{{TARGET3_PREFIX}}` | English |
+
+(Path prefixes: 4.x English `versioned_docs/version-4.x/`, 4.x Chinese
+`i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/`, dev English `docs/`,
dev Chinese
+`i18n/zh-CN/docusaurus-plugin-content-docs/current/`.)
+
+For each file in your batch, get the change to port with:
+
+```
+git diff {{BASE_REF}} -- {{SOURCE_PREFIX}}<FILE>
+```
+
+Then apply the equivalent change to the target paths.
+
+## Rules
+
+1. **Do not blind-patch.** The target files have diverged: different wording,
section order, extra
+ or missing sections. Read the target, find the equivalent place, adapt. If
the section does not
+ exist in a target, add it where it fits that file's structure — or skip and
report if it
+ genuinely does not apply. The other trees may already document the topic in
a better or different
+ way (dev did for `parse_to_variant` and the schema-change pages): reuse,
don't duplicate.
+2. **English must read like the surrounding English doc.** Match existing
terminology (e.g.
+ "compute-storage decoupled mode"), heading style, admonition style
(`:::caution Behavior change
+ (X.Y.Z)`), table format. Do not translate literally. Keep front matter
valid; `"language": "en"`
+ in English files, `"language": "zh-CN"` in Chinese files.
+3. **Version annotations stay.** Dev docs carry the same "Since version X"
notes as the versioned
+ docs, in the phrasing the target file already uses for similar notes.
+4. **The dev trees document `master`, not {{VERSION}}.** Before porting a
factual claim (default
+ value, "since version", whether a feature exists, an error string, a
dependency version) into
+ `docs/` or `i18n/zh-CN/.../current/`, check it on master in
`{{DORIS_REPO}}` — READ ONLY:
+ `git grep <pattern> {{MASTER_REF}}` / `git show {{MASTER_REF}}:<path>`.
Never checkout / commit /
+ edit / fetch there. Master has been refactored (e.g.
`fe/fe-connector/...`), so do NOT narrow
+ `git grep` with pathspecs — search the whole tree and drop
`regression-test` hits. Verified
+ differences are listed below; verify anything else yourself.
+5. **Never introduce these topics into any tree** — deliberately undocumented
for now:
+ {{EXCLUDED_TOPICS}}
+ If a target already contains such content, leave it; just don't add more.
+6. Do not reformat or restructure anything the change does not touch.
+7. Do not touch sidebar files.
+8. If you find an error in the SOURCE file — wrong default, wrong claim, a
table broken by a
+ mid-table insert, a typo in an identifier — fix the source too, and flag it
prominently in your
+ report. Verify against `{{TO_REF}}` first.
+
+## Verified master-vs-{{VERSION}} differences
+
+(paste the `compare-refs.py {{TO_REF}} {{MASTER_REF}}` table here, plus
feature-presence findings)
+
+| Item | {{VERSION}} | master (dev) | What to do in dev docs |
+| --- | --- | --- | --- |
+| ... | ... | ... | omit / state master value / reword |
+
+Master SHA used for this table: `{{MASTER_SHA}}`. Master moves; if you fetch,
say so.
+
+## Reporting
+
+When done, reply with a short report: for each file, one line per target tree
saying `ported`,
+`already present`, `adapted (how)` or `skipped (why)`. List every claim you
verified on master and
+every SOURCE error you fixed. Flag anything you were unsure about.
diff --git a/doc-tools/skills/doris-release-docs/references/report-template.md
b/doc-tools/skills/doris-release-docs/references/report-template.md
new file mode 100644
index 00000000000..9d13499d4e8
--- /dev/null
+++ b/doc-tools/skills/doris-release-docs/references/report-template.md
@@ -0,0 +1,38 @@
+# Doris {{VERSION}} user-doc audit report ({{SOURCE_LABEL}})
+
+Range: `apache/doris` `{{FROM_REF}}..{{TO_REF}}`, **N** commits (M touching
production code, K
+test/CI only). Scope of this round: {{Chinese 4.x only | Chinese 4.x + English
4.x + dev}}. Master SHA
+used for the dev comparison: `{{MASTER_SHA}}`.
+
+Method: `surface-diff.sh` for the hard user-facing surface ->
`split-commits.sh` into batches -> N
+subagents classifying every commit -> `check-version-claims.py` /
`compare-refs.py` for facts -> doc
+edits -> `validate-docs.py`.
+
+---
+
+## 1. Documentation changes made
+
+### 1.1 Syntax and interfaces added or removed
+| Change | commit | Doc change |
+| --- | --- | --- |
+
+### 1.2 Lakehouse
+### 1.3 Load / streaming jobs
+### 1.4 Configs and session variables
+### 1.5 Other
+### 1.6 Deliberately not written (reason: unreleased / no open-source
implementation / user decision)
+
+---
+
+## 2. Needs your decision
+
+### A. Chinese pages written, awaiting sidebar entry + English
+### B. Changes with no obvious home (suggested location / whether to create a
page)
+### C. Product decisions (publish or not)
+### D. Errors found in existing docs unrelated to this release
+
+---
+
+## 3. Deliberately skipped
+- Internal refactors / logging / optimizations (about N commits)
+- Correctness fixes better suited to the release notes than to user docs
diff --git
a/doc-tools/skills/doris-release-docs/scripts/check-version-claims.py
b/doc-tools/skills/doris-release-docs/scripts/check-version-claims.py
new file mode 100755
index 00000000000..a1a104a3dc3
--- /dev/null
+++ b/doc-tools/skills/doris-release-docs/scripts/check-version-claims.py
@@ -0,0 +1,74 @@
+#!/usr/bin/env python3
+"""Which release tags contain a given identifier? Use this BEFORE writing any
+"自 X.Y.Z 版本起支持" / "Since version X.Y.Z" sentence, and to audit claims
+already in the docs.
+
+usage:
+ check-version-claims.py <doris-repo> --tags 4.0.7 4.0.8 4.1.2 4.1.3
4.1.4-rc04 -- <identifier>...
+ check-version-claims.py <doris-repo> --tags ... --from-file identifiers.txt
+ check-version-claims.py <doris-repo> --tags ... --path
be/src/common/config.h -- <identifier>...
+
+--path restricts the search to one pathspec. Use it when FE and BE share a
+name: `enable_group_commit_streamload_be_forward` existed as an FE config
+since 4.0.7 but the BE config of the same name only arrived in 4.0.8 / 4.1.4,
+so an unrestricted search says "since 4.0.7" and is wrong for the BE entry.
+Error strings are often concatenated across lines in Java/C++ — grep a
+distinctive fragment, not the whole sentence.
+
+An identifier can be a config / variable name, an error string fragment, a
+grammar token, a class name — anything git grep can find. Test directories
+are excluded so a regression-test mention does not count as "supported".
+
+Reading the table: the first tag in each release line that contains the
+identifier is the version to annotate. Doris keeps several lines alive
+(4.0.x and 4.1.x), so a feature backported to both needs BOTH first
+versions, e.g. "Doris 4.0 系列自 4.0.8 起、4.1 系列自 4.1.4 起". In the
+4.1.4 round `require_partition_filter` was documented as "4.1 系列自 4.1.2
+起" — it was not in 4.1.2 or 4.1.3, only 4.1.4.
+"""
+import subprocess, sys
+
+def present(repo, tag, ident, path=None):
+ specs = [path] if path else [':!regression-test', ':!*/test/*',
':!be/test', ':!docker']
+ r = subprocess.run(['git', '-C', repo, 'grep', '-l', '-F', '--', ident,
tag, '--'] + specs,
+ capture_output=True, text=True)
+ return r.returncode == 0 and r.stdout.strip() != ''
+
+def main():
+ args = sys.argv[1:]
+ if not args or '--tags' not in args:
+ print(__doc__); sys.exit(1)
+ repo = args[0]
+ ti = args.index('--tags')
+ tags = []
+ path = None
+ i = ti + 1
+ while i < len(args) and args[i] not in ('--', '--from-file', '--path'):
+ tags.append(args[i]); i += 1
+ if i < len(args) and args[i] == '--path':
+ path = args[i + 1]; i += 2
+ idents = []
+ if i < len(args) and args[i] == '--from-file':
+ idents = [l.split()[-1] for l in open(args[i + 1]) if l.strip() and
not l.startswith('#')]
+ elif i < len(args) and args[i] == '--':
+ idents = args[i + 1:]
+ if not tags or not idents:
+ print(__doc__); sys.exit(1)
+ for t in tags:
+ if subprocess.run(['git', '-C', repo, 'rev-parse', '-q', '--verify', t
+ '^{commit}'],
+ capture_output=True).returncode != 0:
+ print(f'unknown ref: {t}', file=sys.stderr); sys.exit(1)
+ w = max(len(x) for x in idents) + 2
+ print(' ' * w + ' '.join(f'{t:>12s}' for t in tags))
+ for ident in dict.fromkeys(idents):
+ cells = []
+ first = None
+ for t in tags:
+ ok = present(repo, t, ident, path)
+ cells.append(f'{"YES" if ok else "-":>12s}')
+ if ok and first is None:
+ first = t
+ print(f'{ident:{w}s}' + ' '.join(cells) + (f' first: {first}' if
first else ' (nowhere)'))
+
+if __name__ == '__main__':
+ main()
diff --git a/doc-tools/skills/doris-release-docs/scripts/compare-refs.py
b/doc-tools/skills/doris-release-docs/scripts/compare-refs.py
new file mode 100755
index 00000000000..dbca054f3a3
--- /dev/null
+++ b/doc-tools/skills/doris-release-docs/scripts/compare-refs.py
@@ -0,0 +1,108 @@
+#!/usr/bin/env python3
+"""Compare the default value / presence of Doris configs and session variables
+between two refs (typically the release tag and upstream master).
+
+usage:
+ compare-refs.py <doris-repo> <ref-a> <ref-b> <identifier>...
+ compare-refs.py <doris-repo> <ref-a> <ref-b> --from-file identifiers.txt
+
+Identifiers are config / session-variable NAMES as the user sees them
+(e.g. enable_expr_zonemap_filter, autobucket_min_buckets,
+group_commit_max_wal_num_per_table). Each one is looked up in
+ FE Config.java (public static <type> <name> = <default>)
+ SessionVariable.java (@VarAttr(name = "<name>" or CONST) -> field default)
+ be/src/common/config.cpp and be/src/cloud/config.cpp (DEFINE_*(name,
"default"))
+ cloud/src/common/config.h (CONF_*(name, "default"))
+and reported per ref. Lines marked <== DIFF need a dev-doc decision:
+absent on master -> omit from dev docs; different default -> state master's.
+
+Why this exists: in the 4.1.4 round four variables backported to branch-4.1
+did not exist on master at all, and three defaults differed. The dev docs
+describe master, so porting the 4.x text verbatim would have been wrong.
+"""
+import re, subprocess, sys, os
+
+def show(repo, ref, path):
+ r = subprocess.run(['git', '-C', repo, 'show', f'{ref}:{path}'],
capture_output=True, text=True)
+ return r.stdout if r.returncode == 0 else ''
+
+FILES = {
+ 'fe': ['fe/fe-common/src/main/java/org/apache/doris/common/Config.java'],
+ 'sv':
['fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java',
+
'fe/fe-core/src/main/java/org/apache/doris/qe/GlobalVariable.java'],
+ 'be': ['be/src/common/config.cpp', 'be/src/cloud/config.cpp'],
+ 'ms': ['cloud/src/common/config.h'],
+}
+
+def fe_default(txt, name):
+ m = re.search(r'\n\s*public static (?:volatile\s+)?[\w<>\[\]]+\s+' +
re.escape(name) + r'\s*=\s*([^;]+);', txt)
+ if not m:
+ return None
+ # the annotation is the nearest preceding @ConfField; stop at the previous
field declaration
+ before = txt[:m.start()]
+ ann_start = before.rfind('@ConfField')
+ prev_field = before.rfind('public static')
+ if ann_start < 0 or ann_start < prev_field:
+ return f'{m.group(1).strip()} (no @ConfField?)'
+ ann = before[ann_start:]
+ mut = 'mutable=true' if re.search(r'mutable\s*=\s*true', ann) else
'mutable=false'
+ return f'{m.group(1).strip()} ({mut})'
+
+def sv_default(txt, name):
+ # name = "<name>" literal, or name = CONST where CONST = "<name>"
+ const = None
+ mc = re.search(r'String\s+([A-Z0-9_]+)\s*=\s*"' + re.escape(name) + r'"',
txt)
+ if mc:
+ const = mc.group(1)
+ # branch-4.x uses @VariableMgr.VarAttr, master uses @VarAttrDef.VarAttr;
accept any qualifier
+ pat = r'@(?:\w+\.)?VarAttr\(\s*name\s*=\s*(?:"' + re.escape(name) + r'"' +
(r'|' + re.escape(const) if const else '') + r')\b'
+ m = re.search(pat, txt)
+ if not m:
+ return None
+ tail = txt[m.end():m.end() + 3000]
+ f =
re.search(r'\n\s*(?:public|private|protected)\s+(?:static\s+)?[\w<>\[\],\s]+?\s+(\w+)\s*=\s*([^;]+);',
tail)
+ if not f:
+ return '?(field not found)'
+ val = f.group(2).strip()
+ flags = []
+ if re.search(r'varType\s*=\s*VariableAnnotation\.EXPERIMENTAL',
txt[m.start():m.end()+600]): flags.append('EXPERIMENTAL')
+ if re.search(r'needForward\s*=\s*true', txt[m.start():m.end()+600]):
flags.append('needForward')
+ return val + (' [' + ','.join(flags) + ']' if flags else '')
+
+def be_default(txt, name):
+ m = re.search(r'DEFINE_([A-Za-z0-9]+)\(\s*' + re.escape(name) +
r'\s*,\s*("(?:[^"\\]|\\.)*")', txt)
+ return f'{m.group(2)} ({m.group(1)})' if m else None
+
+def ms_default(txt, name):
+ m = re.search(r'CONF_([A-Za-z0-9]+)\(\s*' + re.escape(name) +
r'\s*,\s*("(?:[^"\\]|\\.)*")', txt)
+ return f'{m.group(2)} ({m.group(1)})' if m else None
+
+LOOKUP = {'fe': fe_default, 'sv': sv_default, 'be': be_default, 'ms':
ms_default}
+
+def main():
+ if len(sys.argv) < 5:
+ print(__doc__); sys.exit(1)
+ repo, a, b = sys.argv[1:4]
+ ids = sys.argv[4:]
+ if ids and ids[0] == '--from-file':
+ ids = [l.split()[-1] for l in open(ids[1]) if l.strip() and not
l.startswith('#')]
+ src = {ref: {k: '\n'.join(show(repo, ref, p) for p in ps) for k, ps in
FILES.items()} for ref in (a, b)}
+ print(f'{"identifier":52s} {"kind":4s} {a[:22]:24s} {b[:22]:24s}')
+ diffs = 0
+ for name in dict.fromkeys(ids):
+ found = False
+ for kind, fn in LOOKUP.items():
+ va, vb = fn(src[a][kind], name), fn(src[b][kind], name)
+ if va is None and vb is None:
+ continue
+ found = True
+ sa, sb = (va or 'ABSENT'), (vb or 'ABSENT')
+ flag = ' <== DIFF' if sa != sb else ''
+ if flag: diffs += 1
+ print(f'{name:52s} {kind:4s} {sa[:22]:24s} {sb[:22]:24s}{flag}')
+ if not found:
+ print(f'{name:52s} ?? not found in any known config/variable
file on either ref')
+ print(f'\n{diffs} difference(s)')
+
+if __name__ == '__main__':
+ main()
diff --git a/doc-tools/skills/doris-release-docs/scripts/split-commits.sh
b/doc-tools/skills/doris-release-docs/scripts/split-commits.sh
new file mode 100755
index 00000000000..00b1e125214
--- /dev/null
+++ b/doc-tools/skills/doris-release-docs/scripts/split-commits.sh
@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+# List the commits in a range, split them into "touches production code" vs
+# "tests / CI / docker / build scripts only", and cut the production list into
+# batches for parallel subagent classification.
+#
+# usage: split-commits.sh <doris-repo> <from-ref> <to-ref> <out-dir>
[batch-size=30]
+#
+# Do NOT filter by the [tag] in the subject line: in the 4.1.4 round a
+# "[fix](test)" commit added a session variable and a "[fix](regression)"
+# commit added a table-property validation. Only the touched paths are a
+# reliable signal.
+set -euo pipefail
+REPO=$1; FROM=$2; TO=$3; OUT=$4; BATCH=${5:-30}
+mkdir -p "$OUT"; cd "$REPO"
+git log --pretty=format:'%h %s' "$FROM".."$TO" > "$OUT/commits-all.txt"
+echo >> "$OUT/commits-all.txt"
+: > "$OUT/commits-prod.txt"; : > "$OUT/commits-testonly.txt"
+NONPROD='^(regression-test|docker|\.github|thirdparty|be/test|fe/fe-core/src/test|fe/fe-common/src/test|cloud/test|samples|tools|extension|be/benchmark)/'
+while read -r h rest; do
+ [ -z "$h" ] && continue
+ n=$( { git show --pretty=format: --name-only "$h" | grep -v '^$' | grep -vE
"$NONPROD" || true; } | wc -l | tr -d ' ')
+ if [ "$n" -gt 0 ]; then echo "$h $rest" >> "$OUT/commits-prod.txt"; else
echo "$h $rest" >> "$OUT/commits-testonly.txt"; fi
+done < "$OUT/commits-all.txt"
+( cd "$OUT" && rm -f batch-* && split -l "$BATCH" -d commits-prod.txt batch- )
+printf 'all=%s prod=%s test-only=%s batches=%s (size %s)\n' \
+ "$(grep -c . "$OUT/commits-all.txt")" "$(grep -c . "$OUT/commits-prod.txt")"
\
+ "$(grep -c . "$OUT/commits-testonly.txt")" "$(ls "$OUT"/batch-* | wc -l | tr
-d ' ')" "$BATCH"
+echo "subject-tag histogram of production commits:"
+sed -E 's/^[0-9a-f]+ //; s/^(branch-[0-9.]+: ?|\[branch-[0-9.]+\] ?)//'
"$OUT/commits-prod.txt" | grep -oE '^\[[A-Za-z_ -]+\]' | tr 'A-Z' 'a-z' | sort
| uniq -c | sort -rn | head -12
diff --git a/doc-tools/skills/doris-release-docs/scripts/surface-diff.sh
b/doc-tools/skills/doris-release-docs/scripts/surface-diff.sh
new file mode 100755
index 00000000000..939945ee75e
--- /dev/null
+++ b/doc-tools/skills/doris-release-docs/scripts/surface-diff.sh
@@ -0,0 +1,65 @@
+#!/usr/bin/env bash
+# Extract the user-facing "surface" diff between two Doris refs.
+#
+# usage: surface-diff.sh <doris-repo> <from-ref> <to-ref> <out-dir>
+#
+# Writes one .diff per surface area plus summary.txt (only +/- lines, no
context)
+# so the reader can see every added/removed/changed config, session variable,
+# grammar rule, builtin function, system-table column, HTTP endpoint, metric,
+# shipped conf file line and dependency version WITHOUT trusting commit
messages.
+set -euo pipefail
+REPO=$1; FROM=$2; TO=$3; OUT=$4
+mkdir -p "$OUT"
+cd "$REPO"
+git rev-parse --verify -q "$FROM^{commit}" >/dev/null || { echo "unknown ref:
$FROM" >&2; exit 1; }
+git rev-parse --verify -q "$TO^{commit}" >/dev/null || { echo "unknown ref:
$TO" >&2; exit 1; }
+
+area() { # name, then pathspecs
+ local name=$1; shift
+ git diff "$FROM".."$TO" -- "$@" > "$OUT/$name.diff" 2>/dev/null || true
+ local n; n=$(grep -cE '^[+-]' "$OUT/$name.diff" 2>/dev/null || echo 0)
+ printf '%-16s %6s changed lines (%s)\n' "$name" "$n" "$*" >>
"$OUT/summary.txt"
+}
+: > "$OUT/summary.txt"
+echo "# surface diff $FROM..$TO" >> "$OUT/summary.txt"
+
+area fe-config
'fe/fe-common/src/main/java/org/apache/doris/common/Config.java'
+area session-var
'fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java' \
+
'fe/fe-core/src/main/java/org/apache/doris/qe/GlobalVariable.java'
+area be-config 'be/src/common/config.cpp' 'be/src/common/config.h'
'be/src/cloud/config.cpp' 'be/src/cloud/config.h'
+area ms-config 'cloud/src/common/config.h'
+area grammar 'fe/fe-core/src/main/antlr4/**'
+area functions
'fe/fe-core/src/main/java/org/apache/doris/catalog/Builtin*.java' \
+
'fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/**'
+area schema-tables
'fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java' \
+
'fe/fe-core/src/main/java/org/apache/doris/catalog/InternalSchema.java' \
+
'fe/fe-core/src/main/java/org/apache/doris/catalog/InternalSchemaInitializer.java'
+area show-commands
'fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/Show*.java'
\
+
'fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/Admin*.java'
+area http-be 'be/src/service/http_service.cpp'
'be/src/service/http/action/**' 'be/src/http/action/**'
+area http-fe 'fe/fe-core/src/main/java/org/apache/doris/httpv2/**'
+area metrics
'fe/fe-core/src/main/java/org/apache/doris/metric/MetricRepo.java'
'be/src/util/doris_metrics.*' 'be/src/util/metrics.*'
+area properties
'fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java' \
+
'fe/fe-core/src/main/java/org/apache/doris/datasource/property/**' \
+
'fe/fe-common/src/main/java/org/apache/doris/job/cdc/DataSourceConfigKeys.java'
+area conf-files 'conf/fe.conf' 'conf/be.conf'
'conf/apache_hdfs_broker.conf' 'bin/*.sh'
+area deps 'fe/pom.xml' 'thirdparty/vars.sh' 'be/CMakeLists.txt'
+area build-modules 'build.sh' 'fe/be-java-extensions/pom.xml'
+
+# Quick-read extracts (only +/- lines, drop file headers)
+for f in fe-config session-var be-config ms-config grammar functions
schema-tables conf-files deps; do
+ { echo "===== $f ====="; grep -E '^[+-]' "$OUT/$f.diff" | grep -vE
'^[+-]{3}' ; echo; } >> "$OUT/summary.txt" 2>/dev/null || true
+done
+
+# Config/variable identifiers that appear in the diffs — handy for
check-version-claims.py
+{
+ grep -hoE '^\+\s*public static [a-zA-Z_<>\[\]]+ [a-z0-9_]+ *='
"$OUT/fe-config.diff" | sed -E 's/.* ([a-z0-9_]+) *=/\1/' | sed 's/^/fe-config
/'
+ grep -hoE '^\+.*name = "[a-z0-9_]+"' "$OUT/session-var.diff" | sed -E
's/.*name = "([a-z0-9_]+)".*/\1/' | sed 's/^/session-var /'
+ grep -hoE '^\+.*VarAttr\(name = [A-Z0-9_]+' "$OUT/session-var.diff" | sed -E
's/.*name = ([A-Z0-9_]+).*/\1/' | sed 's/^/session-var-const /'
+ grep -hoE '^\+DE(FINE|CLARE)_[A-Za-z0-9]+\([a-z0-9_]+' "$OUT/be-config.diff"
| sed -E 's/.*\(//' | sed 's/^/be-config /'
+ grep -hoE '^-\s*public static [a-zA-Z_<>\[\]]+ [a-z0-9_]+ *='
"$OUT/fe-config.diff" | sed -E 's/.* ([a-z0-9_]+) *=/\1/' | sed 's/^/REMOVED
fe-config /'
+ grep -hoE '^-DE(FINE|CLARE)_[A-Za-z0-9]+\([a-z0-9_]+' "$OUT/be-config.diff"
| sed -E 's/.*\(//' | sed 's/^/REMOVED be-config /'
+} | sort -u > "$OUT/identifiers.txt" || true
+
+echo "wrote $OUT/summary.txt ($(wc -l < "$OUT/summary.txt") lines) and $(ls
"$OUT"/*.diff | wc -l | tr -d ' ') diff files"
+echo "identifiers: $OUT/identifiers.txt ($(wc -l < "$OUT/identifiers.txt")
entries)"
diff --git a/doc-tools/skills/doris-release-docs/scripts/validate-docs.py
b/doc-tools/skills/doris-release-docs/scripts/validate-docs.py
new file mode 100755
index 00000000000..2d8cc8483f9
--- /dev/null
+++ b/doc-tools/skills/doris-release-docs/scripts/validate-docs.py
@@ -0,0 +1,139 @@
+#!/usr/bin/env python3
+"""Static checks for changed doris-website docs. No yarn build — this is the
+only verification the doc trees get, so run it before every commit.
+
+usage:
+ validate-docs.py <website-repo> [--base <git-ref>] [--forbidden <file>]
+
+Checks every .md/.mdx that differs from --base (default: working tree vs
+index+HEAD via `git status`) under docs/, versioned_docs/ and i18n/:
+ 1. JSON front matter parses; "language" matches the tree (en / zh-CN)
+ 2. every relative markdown link resolves to an existing .md/.mdx
+ 3. .mdx files: no bare <tag> outside code fences / inline code that is not
+ a known component (MDX would treat it as JSX and fail the build)
+ 4. no CJK characters in lines ADDED to English trees
+ 5. added lines contain none of the forbidden identifiers (--forbidden:
+ one pattern per line, e.g. unreleased-feature config names)
+ 6. markdown tables in added lines keep a consistent column count within
+ each table block (a row with a blockquote between rows splits a table)
+Exit code 1 if anything fails.
+"""
+import json, os, re, subprocess, sys
+
+KNOWN_TAGS =
{'Tabs','TabItem','details','summary','br','img','b','i','p','table','tr','td','th','thead','tbody',
+
'a','code','div','span','strong','em','ul','li','ol','hr','sup','sub','h1','h2','h3','h4','h5','h6',
+
'center','font','pre','video','source','iframe','DorisVideo','Video','Tab','Admonition','kbd','u',
+
'Head','Link','style','script','details','figure','figcaption','picture','svg','path','g','rect','circle','line','text'}
+
+def run(cmd, cwd):
+ return subprocess.run(cmd, cwd=cwd, capture_output=True, text=True).stdout
+
+def changed_files(root, base):
+ if base:
+ out = run(['git', 'diff', '--name-only', base], root)
+ files = out.split()
+ else:
+ out = run(['git', 'status', '--porcelain'], root)
+ files = [l[3:].strip() for l in out.splitlines()]
+ return sorted({f for f in files if f.endswith(('.md', '.mdx'))
+ and f.startswith(('docs/', 'versioned_docs/', 'i18n/'))})
+
+def added_lines(root, base, rel):
+ args = ['git', 'diff', '--unified=0'] + ([base] if base else []) + ['--',
rel]
+ out = run(args, root)
+ return [l[1:] for l in out.splitlines() if l.startswith('+') and not
l.startswith('+++')]
+
+def main():
+ args = sys.argv[1:]
+ if not args:
+ print(__doc__); sys.exit(1)
+ root = os.path.abspath(args[0]); base = None; forb = []
+ if '--base' in args: base = args[args.index('--base') + 1]
+ if '--forbidden' in args:
+ forb = [l.strip() for l in open(args[args.index('--forbidden') + 1])
if l.strip() and not l.startswith('#')]
+ files = changed_files(root, base)
+ problems = []
+ for rel in files:
+ p = os.path.join(root, rel)
+ if not os.path.isfile(p):
+ continue
+ t = open(p, encoding='utf-8').read()
+ en_tree = rel.startswith(('docs/', 'versioned_docs/'))
+ # 1 front matter
+ if not t.startswith('---\n'):
+ problems.append((rel, 'no front matter'))
+ else:
+ raw = t[4:t.index('\n---\n', 3) + 1].strip()
+ fm = None
+ if raw.startswith('{'):
+ try:
+ fm = json.loads(raw)
+ except Exception as e:
+ problems.append((rel, f'front matter is not valid JSON:
{e}'))
+ else: # YAML front matter (key-features, community docs): only
sanity-check the shape
+ if not all(re.match(r'^(\s*-|\s*\w[\w.-]*\s*:|\s*$)', l) for l
in raw.split('\n')):
+ problems.append((rel, 'front matter is neither JSON nor
simple YAML'))
+ m = re.search(r'^language:\s*"?([\w-]+)', raw, re.M)
+ fm = {'language': m.group(1)} if m else {}
+ if fm is not None:
+ want = 'en' if en_tree else 'zh-CN'
+ lang = fm.get('language')
+ if lang is not None and lang != want and not (want == 'en' and
lang == 'en-US'):
+ problems.append((rel, f'language={lang}, expected {want}'))
+ # 2 relative links
+ d = os.path.dirname(p)
+ for m in re.finditer(r'\]\((\.\.?/[^)#\s]+)(#[^)\s]*)?\)', t):
+ f = os.path.normpath(os.path.join(d, m.group(1)))
+ if not (os.path.exists(f) or os.path.exists(f + '.md') or
os.path.exists(f + '.mdx')):
+ problems.append((rel, f'broken link {m.group(1)}'))
+ # 3 mdx raw tags
+ if rel.endswith('.mdx'):
+ imported = set(re.findall(r'^import\s+(\w+)', t, re.M))
+ for grp in re.findall(r'^import\s*\{([^}]*)\}', t, re.M):
+ imported.update(x.strip().split(' as ')[-1] for x in
grp.split(',') if x.strip())
+ infence = False
+ for i, l in enumerate(t.split('\n'), 1):
+ if l.strip().startswith('```'):
+ infence = not infence; continue
+ if infence: continue
+ s = re.sub(r'`[^`]*`', '', l)
+ for mm in re.finditer(r'<([A-Za-z][A-Za-z0-9_-]*)', s):
+ if mm.group(1) not in KNOWN_TAGS and mm.group(1) not in
imported:
+ problems.append((rel, f'line {i}: bare <{mm.group(1)}>
outside code (MDX/JSX risk)'))
+ added = added_lines(root, base, rel)
+ # 4 CJK in English trees
+ if en_tree:
+ for l in added:
+ if re.search(r'[\u4e00-\u9fff]', l):
+ problems.append((rel, f'Chinese text in English tree:
{l.strip()[:80]}'))
+ # 5 forbidden identifiers
+ for pat in forb:
+ for l in added:
+ if re.search(pat, l):
+ problems.append((rel, f'forbidden "{pat}":
{l.strip()[:80]}'))
+ # 6 table column consistency (per contiguous table block in the full
file)
+ block = []
+ infence = False
+ for i, l in enumerate(t.split('\n') + [''], 1):
+ if l.strip().startswith('```'):
+ infence = not infence
+ l = ''
+ if not infence and l.lstrip().startswith('|'):
+ cells = re.sub(r'`[^`]*`', '', l).replace('\\|', '')
+ block.append((i, cells.strip().count('|')))
+ else:
+ if len(block) >= 2:
+ counts = {c for _, c in block}
+ if len(counts) > 1:
+ problems.append((rel, f'table at lines
{block[0][0]}-{block[-1][0]} has inconsistent column counts {sorted(counts)}'))
+ block = []
+ print(f'checked {len(files)} changed doc files')
+ if problems:
+ for rel, msg in problems:
+ print(f' {rel}: {msg}')
+ print(f'{len(problems)} problem(s)')
+ sys.exit(1)
+ print('no problems')
+
+if __name__ == '__main__':
+ main()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
