This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 16813ae25c1 [feat](i18n) Select exact sources for manual Korean
batches (#4362)
16813ae25c1 is described below
commit 16813ae25c1dd541fb7074144ac2004f53107823
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Sun Oct 18 21:38:42 2026 +0100
[feat](i18n) Select exact sources for manual Korean batches (#4362)
The manual Korean translation workflow currently accepts one path
prefix. A reviewed batch spanning different folders therefore cannot
select only its intended sources without including unrelated pages.
Add an optional `paths` JSON array of 1–20 exact English Markdown/MDX
source files to manual Korean discovery and the fresh scan before
generation. It is mutually exclusive with the existing `path` prefix.
One-page `validate` still requires `path`; regular generation keeps its
10-target cap and the planner keeps its 20-target cap. Full coverage
counts, scope/version rules, current-page and active-PR exclusion, human
ownership, and exact-master protection remain enforced. Unknown,
duplicate, malformed, and out-of-scope inputs fail before model calls.
The Korean workflow remains manual-only with dry-run as its default,
using the same entire-run lock as Japanese. This change adds no
schedule, write permissions, credentials, or rendering changes.
Validation: Node 20 tests passed (198 passed, zero failures, one
existing Apache test skipped); git diff checks passed. Regression
coverage includes exact selection across folders/formats, stable task
IDs and full counts, current/active exclusion, limits, invalid input,
and rejecting a list in one-page validation before creating scan
artifacts. The existing path-prefix flow and all locale/workflow tests
pass. Build scope correctly skips website compilation for these
non-rendering tools/workflow changes. A real exact-list dry-run and
bounded regular generation will follow on merged master as T08-A
acceptance.
A real Korean planner run with the exact list preserved
required/current/pending counts at 3,336/5/3,331 and selected only the
requested community MDX and mailing-list Markdown. The separate manual
concurrency probe (two Korean and one Japanese dry-run) succeeded
serially, with generation skipped in every run and zero model calls.
Synchronized with merged PR #4361 (`072ace699d`) without conflicts. The
change remains six non-rendering files and still produces
`skip_build=true`; no publication/browser validation is requested.
Post-merge i18n and PoC tests passed (197 tests, zero failures). CI is
rerunning on head `0e8303c3ef0d7c637db6a36e15c6d0ae5002ce74`.
Co-authored-by: morningman <[email protected]>
---
.github/workflows/korean-translation.yml | 5 +++++
scripts/i18n/__tests__/korean-workflow.test.cjs | 19 ++++++++++++++++
scripts/i18n/__tests__/planner.test.cjs | 29 ++++++++++++++++++++++---
scripts/i18n/plan-translations.cjs | 27 ++++++++++++++++++++---
scripts/i18n/workflow-scan.cjs | 5 ++++-
translations/README.md | 26 ++++++++++++++++++++++
6 files changed, 104 insertions(+), 7 deletions(-)
diff --git a/.github/workflows/korean-translation.yml
b/.github/workflows/korean-translation.yml
index 1c07a2e0c55..133a4ea2fcb 100644
--- a/.github/workflows/korean-translation.yml
+++ b/.github/workflows/korean-translation.yml
@@ -18,6 +18,9 @@ on:
path:
description: Optional repository-relative source path prefix
type: string
+ paths:
+ description: Optional JSON array of exact English .md/.mdx files
(exclusive with path)
+ type: string
permissions:
contents: read
@@ -63,6 +66,7 @@ jobs:
I18N_PLUGIN: ${{ inputs.plugin }}
I18N_VERSION: ${{ inputs.version }}
I18N_PATH: ${{ inputs.path }}
+ I18N_PATHS: ${{ inputs.paths }}
I18N_LIMIT: '20'
run: node scripts/i18n/workflow-scan.cjs
"$RUNNER_TEMP/korean-translation"
- uses: actions/upload-artifact@v4
@@ -140,6 +144,7 @@ jobs:
I18N_PLUGIN: ${{ inputs.plugin }}
I18N_VERSION: ${{ inputs.version }}
I18N_PATH: ${{ inputs.path }}
+ I18N_PATHS: ${{ inputs.paths }}
I18N_LIMIT: '20'
run: node scripts/i18n/workflow-scan.cjs
"$RUNNER_TEMP/korean-translation"
- name: Verify the Korean PR label before model calls
diff --git a/scripts/i18n/__tests__/korean-workflow.test.cjs
b/scripts/i18n/__tests__/korean-workflow.test.cjs
index 640f3956b2c..b193deb02e9 100644
--- a/scripts/i18n/__tests__/korean-workflow.test.cjs
+++ b/scripts/i18n/__tests__/korean-workflow.test.cjs
@@ -3,16 +3,31 @@ const assert = require('node:assert/strict');
const fs = require('node:fs');
const path = require('node:path');
const yaml = require('js-yaml');
+const os = require('node:os');
+const {spawnSync} = require('node:child_process');
const workflows = path.resolve(__dirname, '../../../.github/workflows');
const load = name => yaml.load(fs.readFileSync(path.join(workflows, name),
'utf8'));
const korean = load('korean-translation.yml');
const japanese = load('japanese-translation.yml');
+test('one-page validation rejects an exact batch list before creating scan
artifacts', t => {
+ const directory = fs.mkdtempSync(path.join(os.tmpdir(),
'ko-validate-paths-'));
+ t.after(() => fs.rmSync(directory, {recursive:true,force:true}));
+ const output = path.join(directory,'scan');
+ const result = spawnSync(process.execPath,
[path.resolve(workflows,'../../scripts/i18n/workflow-scan.cjs'),output], {
+
encoding:'utf8',env:{...process.env,I18N_LOCALE:'ko',I18N_MODE:'validate',I18N_PATHS:'["community/subscribe-mail-list.md"]'},
+ });
+ assert.equal(result.status,1);
+ assert.match(result.stderr,/Validation mode requires path, not a list/);
+ assert(!fs.existsSync(output));
+});
+
test('Korean generation requires a manual choice and shares the existing
whole-run lock', () => {
assert.deepEqual(Object.keys(korean.on), ['workflow_dispatch']);
assert.equal(korean.on.workflow_dispatch.inputs.mode.default, 'dry-run');
assert.deepEqual(korean.on.workflow_dispatch.inputs.mode.options,
['dry-run', 'validate', 'generate']);
assert.equal(korean.on.workflow_dispatch.inputs.locale, undefined);
+ assert.equal(korean.on.workflow_dispatch.inputs.paths.type, 'string');
assert.deepEqual(korean.permissions, {'contents':'read',
'pull-requests':'read'});
assert.deepEqual(korean.concurrency, japanese.concurrency);
assert.equal(korean.concurrency.group, 'japanese-translation-ja');
@@ -47,6 +62,10 @@ test('Korean discovery and generation exchange Korean
artifacts at the exact sca
const execute = generate.find(step => step.id === 'translation');
assert.equal(execute.env.I18N_MAX_PAGES, "${{ inputs.mode == 'validate' &&
'1' || '10' }}");
assert.equal(execute.env.I18N_PATH, '${{ inputs.path }}');
+ for (const steps of [discover, generate]) {
+ const scan = steps.find(step =>
step.run?.includes('workflow-scan.cjs'));
+ assert.equal(scan.env.I18N_PATHS, '${{ inputs.paths }}');
+ }
assert.equal(execute.env.I18N_CODEX_MODEL,
japanese.jobs.generate.steps.find(step => step.id ===
'translation').env.I18N_CODEX_MODEL);
});
diff --git a/scripts/i18n/__tests__/planner.test.cjs
b/scripts/i18n/__tests__/planner.test.cjs
index 5b3baa377bd..26c0cf630b6 100644
--- a/scripts/i18n/__tests__/planner.test.cjs
+++ b/scripts/i18n/__tests__/planner.test.cjs
@@ -1,7 +1,7 @@
const test = require('node:test');
const assert = require('node:assert/strict');
const {hash} = require('../state.cjs');
-const {plan, formattingOnly, validateActive, validateResolved} =
require('../plan-translations.cjs');
+const {plan, formattingOnly, validateActive, validateResolved,
validateSourcePaths, argumentsFrom} = require('../plan-translations.cjs');
const sha = 'a'.repeat(40);
const baselineSha = 'b'.repeat(40);
@@ -22,14 +22,14 @@ function page(docId, options = {}) {
reviewStatus: 'unverified', change: options.change ?? 'new'};
}
async function run(pages, {old = [], records = [], removed = [], active, files
= {}, historical = {}, reports = [],
- chineseChangedPaths = [], limit = 20, tokenBudget = null} = {}) {
+ chineseChangedPaths = [], limit = 20, tokenBudget = null, filters = {}} =
{}) {
const all = Object.fromEntries(pages.map(item => [item.sourcePath,
source]));
const report = {schemaVersion: 1, sourceCommit: sha, baselineCommit:
baselineSha,
groups: scope.groups.map(group => ({...group, entries:
pages.filter(item => item.pluginId === group.pluginId && item.version ===
group.version),
unregistered: []})), removed, reports};
return plan({report, scope, records: new Map(records.map(item =>
[JSON.stringify([item.pluginId, item.version, item.docId, 'ja']), item])),
baseline: old, active, chineseChangedPaths, readCurrent: name =>
files[name] ?? all[name] ?? null,
- readAt: (ref, name) => historical[`${ref}:${name}`] ?? null, limit,
tokenBudget});
+ readAt: (ref, name) => historical[`${ref}:${name}`] ?? null, limit,
tokenBudget, filters});
}
test('stable full scan includes new pages after baseline, isolates versions,
and never requeues current pages', async () => {
@@ -191,6 +191,29 @@ test('default planning does not exclude a document because
of estimated tokens',
assert(result.selectedEstimatedSourceTokens > 50000);
});
+test('exact source lists select only requested native files while preserving
full counts and limits', async () => {
+ const a = page('a'), b = page('b', {sourcePath:'docs/b.mdx'}), c =
page('c', {pluginId:'community'});
+ const all = await run([a,b,c]);
+ const filtered = await run([a,b,c],
{filters:{paths:[b.sourcePath,c.sourcePath]}});
+ assert.equal(filtered.counts.required, 3);
+ assert.deepEqual(filtered.tasks.map(task=>task.sourcePath),
[b.sourcePath,c.sourcePath]);
+ assert.deepEqual(filtered.selected,
all.tasks.filter(task=>task.sourcePath!==a.sourcePath).map(task=>task.id));
+ assert.equal((await run([a,b,c],
{filters:{paths:[b.sourcePath,c.sourcePath]},limit:1})).selected.length, 1);
+ assert.equal((await run([page('done',{state:'current'})],
{filters:{paths:['docs/done.md']}})).selected.length, 0);
+ const active = {schemaVersion:1,candidates:[{...b,locale:'ja',pr:'#456'}]};
+ const reserved = await run([a,b,c],
{filters:{paths:[b.sourcePath,c.sourcePath]},active});
+ assert.deepEqual(reserved.selected,
[reserved.tasks.find(task=>task.sourcePath===c.sourcePath).id]);
+});
+
+test('exact source lists reject ambiguous, duplicate, malformed and
out-of-scope inputs', async () => {
+ for (const paths of [[], 'docs/a.md', ['docs/a.md','docs/a.md'],
['../docs/a.md'], ['/docs/a.md'],
+ ['i18n/ko/a.md'], ['docs/../a.md'], ['docs/a/'], ['docs/a.md/'],
['docs/a.md\n'], [null], Array.from({length:21},(_,i)=>`docs/${i}.md`)])
+ assert.throws(()=>validateSourcePaths(paths,scope),
/paths|path|scope/i);
+ await assert.rejects(run([page('a')],
{filters:{path:'docs/',paths:['docs/a.md']}}), /mutually exclusive/);
+ await assert.rejects(run([page('a')],
{filters:{paths:['docs/missing.md']}}), /registered translatable source/);
+ assert.deepEqual(argumentsFrom(['--paths','["docs/a.md"]']),
{'--paths':'["docs/a.md"]'});
+});
+
test('active input rejects duplicate, out-of-scope and malformed candidates',
() => {
const candidate = {pluginId: 'default', version: 'current', docId: 'a',
locale: 'ja', sourcePath: 'docs/a.md',
sourceHash: hash(source), pr: '#1'};
diff --git a/scripts/i18n/plan-translations.cjs
b/scripts/i18n/plan-translations.cjs
index 03a3a6ee601..05e20f79307 100644
--- a/scripts/i18n/plan-translations.cjs
+++ b/scripts/i18n/plan-translations.cjs
@@ -85,6 +85,18 @@ function referenceChecks(oldPage, newPage) {
};
}
+function validateSourcePaths(paths, scope) {
+ if (paths === undefined) return;
+ assert(Array.isArray(paths) && paths.length > 0 && paths.length <= 20,
'paths must contain 1 to 20 exact source files');
+ assert.equal(new Set(paths).size, paths.length, 'Duplicate exact source
path');
+ for (const file of paths) {
+ assert(typeof file === 'string' && /\.mdx?$/.test(file) &&
!file.includes('\\') && !/[\x00-\x1f\x7f]/.test(file)
+ && file.split('/').every(part => part && part !== '.' && part !==
'..')
+ && scope.groups.some(group =>
file.startsWith(`${group.sourceRoot}/`)),
+ `Exact source path is outside the managed scope: ${file}`);
+ }
+}
+
async function plan({report, scope, records, baseline = [], active,
readCurrent, readAt,
resolved = {schemaVersion: 1, deletions: []}, chineseChangedPaths = [],
limit = scope.maxBatchPages, tokenBudget = null, filters = {}}) {
@@ -96,6 +108,8 @@ async function plan({report, scope, records, baseline = [],
active, readCurrent,
assert(Number.isInteger(limit) && limit >= 0 && limit <=
scope.maxBatchPages, 'Limit exceeds managed batch size');
assert(tokenBudget === null || Number.isInteger(tokenBudget) &&
tokenBudget >= 0, 'Invalid token budget');
if (filters.locale && filters.locale !== scope.locale) throw new
Error(`Unsupported locale: ${filters.locale}`);
+ validateSourcePaths(filters.paths, scope);
+ assert(!(filters.path && filters.paths), 'path and paths are mutually
exclusive');
const key = keyFor(scope.locale);
const candidates = validateActive(active, scope);
const resolvedDeletions = validateResolved(resolved);
@@ -106,6 +120,8 @@ async function plan({report, scope, records, baseline = [],
active, readCurrent,
const pages = report.groups.filter(group => managed(scope,
group)).flatMap(group =>
group.entries.filter(page => page.state !== 'draft' && page.state !==
'retain_english')
.map(page => ({...page, groupId: group.id})));
+ const sourcePaths = new Set(pages.map(page => page.sourcePath));
+ for (const file of filters.paths ?? []) assert(sourcePaths.has(file), `Not
a registered translatable source file: ${file}`);
const current = new Set(pages.map(key));
assert.equal(current.size, pages.length, 'Duplicate current identity');
const removed = [...old.values()].filter(page => !current.has(key(page))
&& page.disposition !== 'draft'
@@ -199,6 +215,7 @@ async function plan({report, scope, records, baseline = [],
active, readCurrent,
assert.equal(allCounts.required, allCounts.mergedCurrent +
allCounts.pending);
const selectedTasks = tasks.filter(task => (!filters.plugin ||
task.pluginId === filters.plugin)
&& (!filters.version || task.version === filters.version)
+ && (!filters.paths || filters.paths.includes(task.sourcePath))
&& (!filters.path || task.sourcePath.startsWith(filters.path) ||
task.targetPath?.startsWith(filters.path)));
const selected = [];
let tokens = 0;
@@ -218,7 +235,7 @@ async function plan({report, scope, records, baseline = [],
active, readCurrent,
function argumentsFrom(argv) {
const options = {}, known = new Set(['--source-ref', '--output',
'--audit-output', '--active-candidates', '--locale', '--plugin',
- '--version', '--path', '--limit', '--token-budget']);
+ '--version', '--path', '--paths', '--limit', '--token-budget']);
for (let i = 0; i < argv.length; i += 2) {
assert(known.has(argv[i]) && argv[i + 1] && !options[argv[i]],
`Unknown or incomplete option: ${argv[i]}`);
options[argv[i]] = argv[i + 1];
@@ -234,6 +251,9 @@ async function main() {
const config = loadConfig(root);
assert(scope.locale === locale, `Scope is for ${scope.locale}, not the
managed ${locale}`);
assert(config.locales[locale]?.sourceLocale === 'en', 'Planner requires an
English primary source');
+ const sourcePaths = options['--paths'] === undefined ? undefined :
JSON.parse(options['--paths']);
+ validateSourcePaths(sourcePaths, scope);
+ assert(!(options['--path'] && sourcePaths), 'path and paths are mutually
exclusive');
const report = await audit({root, sourceRef: options['--source-ref'] ??
'HEAD'});
if (options['--audit-output']) fs.writeFileSync(options['--audit-output'],
JSON.stringify(report, null, 2) + '\n');
const chineseRoots = ['i18n/zh-CN/docusaurus-plugin-content-docs',
'i18n/zh-CN/docusaurus-plugin-content-docs-community'];
@@ -261,7 +281,8 @@ async function main() {
chineseChangedPaths, readCurrent, readAt,
limit: options['--limit'] === undefined ? scope.maxBatchPages :
Number(options['--limit']),
tokenBudget: options['--token-budget'] === undefined ? null :
Number(options['--token-budget']),
- filters: {locale: options['--locale'], plugin: options['--plugin'],
version: options['--version'], path: options['--path']}});
+ filters: {locale: options['--locale'], plugin: options['--plugin'],
version: options['--version'],
+ path: options['--path'], paths: sourcePaths}});
const output = JSON.stringify(result, null, 2) + '\n';
if (options['--output']) fs.writeFileSync(options['--output'], output);
else process.stdout.write(output);
@@ -269,4 +290,4 @@ async function main() {
}
if (require.main === module) main().catch(error => {console.error(error);
process.exitCode = 1;});
-module.exports = {plan, validateActive, validateResolved, formattingOnly,
argumentsFrom};
+module.exports = {plan, validateActive, validateResolved, validateSourcePaths,
formattingOnly, argumentsFrom};
diff --git a/scripts/i18n/workflow-scan.cjs b/scripts/i18n/workflow-scan.cjs
index eea629a629d..32adc3d5a2a 100644
--- a/scripts/i18n/workflow-scan.cjs
+++ b/scripts/i18n/workflow-scan.cjs
@@ -57,6 +57,8 @@ function summarize(plan, reconciliation, eligible, mode,
language) {
function main() {
const output = process.argv[2];
assert(output && process.argv.length === 3, 'Usage: workflow-scan.cjs
<output-directory>');
+ assert(process.env.I18N_MODE !== 'validate' || !process.env.I18N_PATHS,
+ 'Validation mode requires path, not a list of paths');
fs.mkdirSync(output, {recursive: true});
const reconciliation = JSON.parse(fs.readFileSync(path.join(output,
'open-pr-reconciliation.json'), 'utf8'));
const planFile = path.join(output, 'plan.json');
@@ -64,7 +66,8 @@ function main() {
const args = [path.join(__dirname, 'plan-translations.cjs'),
'--source-ref', 'HEAD',
'--active-candidates', path.join(output, 'active-candidates.json'),
'--output', planFile,
'--audit-output', auditFile, '--limit', process.env.I18N_LIMIT ||
'20'];
- for (const [key, option] of [['I18N_PLUGIN', '--plugin'], ['I18N_VERSION',
'--version'], ['I18N_PATH', '--path']]) {
+ for (const [key, option] of [['I18N_PLUGIN', '--plugin'], ['I18N_VERSION',
'--version'],
+ ['I18N_PATH', '--path'], ['I18N_PATHS', '--paths']]) {
if (process.env[key]) args.push(option, process.env[key]);
}
execFileSync(process.execPath, args, {cwd: root, stdio: 'inherit'});
diff --git a/translations/README.md b/translations/README.md
index c27d3018da7..6e106b5b081 100644
--- a/translations/README.md
+++ b/translations/README.md
@@ -603,6 +603,32 @@ No real model call or GitHub write occurs.
## Validation
+### Selecting an exact Korean manual batch
+
+The Korean workflow remains manual-only and defaults to `dry-run`. Its optional
+`paths` input accepts a JSON array of 1–20 exact English source files, so a
bounded
+batch can select documents from different folders without including every page
+under a shared prefix:
+
+```bash
+gh workflow run korean-translation.yml --repo apache/doris-website --ref
master \
+ -f mode=dry-run \
+ -f
'paths=["community/subscribe-mail-list.md","community/how-to-contribute/contribute-to-doris.mdx"]'
+```
+
+Use `paths` or the existing `path` prefix, not both. The same list is used in
the
+fresh scan immediately before generation. `validate` still requires one exact
+`path`; use `generate` for a regular batch. The planner considers at most 20
+targets and generation still produces at most 10. Exact selection preserves
+full coverage counts, current-page and active-PR exclusion, human ownership,
+scope/version checks, and exact-master protection. Unknown, duplicate,
+malformed or out-of-scope source paths fail before model calls. This input does
+not enable scheduled generation.
+
+The planner CLI exposes the same filter as `--paths '<JSON array>'`.
+
+### Validation commands
+
```bash
node --test scripts/i18n/__tests__/*.test.cjs scripts/i18n/poc/*.test.cjs
node scripts/i18n/verify-fixture.cjs
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
