This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 56b6478a7b5 [fix](i18n) Protect Korean metadata and staged translation
baselines (#4356)
56b6478a7b5 is described below
commit 56b6478a7b5d563becce7e112b46926757cf9d2b
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Sun Oct 18 17:01:46 2026 +0100
[fix](i18n) Protect Korean metadata and staged translation baselines (#4356)
A Korean document could retain `language: ja` or drop visible front
matter without receiving validation feedback. Manual staged application
could also overwrite a newer machine translation when its target and
state were updated consistently after staging. This PR reports metadata
defects during generation and admission, and binds staged application to
the target, effective state file, and shard-layout marker captured
before generation.
Manual OpenAI staging manifests now use schema version 2; regenerate
older stages before applying them. Page-state and Codex workflow
artifact schemas remain version 1. The existing three-attempt Codex
policy still retains final content issues as explicit review warnings.
Korean fixtures cover native identities and dynamic plans, active PR
exclusions, source changes and moves, MDX, retry feedback, Dev/4.x
reuse, legacy state and page shards, human ownership, concurrent writes,
and exact staged application. Operational instructions explain explicit
locale selection and script-driven planning without fixed batch
registration.
Validation: Node 20 ran 190 tests: 189 passed, no failures, and one
existing Apache integration test was skipped. Both the metadata gap and
consistent target/state replacement were reproduced before their fixes.
Fixtures use injected model processes and synthetic staging manifests;
no real model calls or credentials were used. A read-only native Korean
scan at this committed implementation inspected 38 open PRs: 3,336
required pages, 4 current, 3,332 pending, 0 active candidates or
exceptions, and 10 dynamically selected tasks; generation remained
disabled.
## Versions
Shared translation infrastructure applies to the configured Dev, 4.x,
and community translation scopes. No document content or routes change.
## Languages
- [x] Japanese
- [x] Korean
## Docs Checklist
- [x] Checked by AI
- [x] Test Cases Built
- [x] Updated required version and language counterparts, or explained
why not
- [x] If only one language changed, confirmed whether source/translation
counterparts need sync
Japanese regression fixtures pass with the shared validation and staging
changes. English/Chinese source documents and existing translation
records are unchanged. Production Korean registration, scheduled
generation, and bulk translation are outside this PR.
Co-authored-by: morningman <[email protected]>
---
.github/workflows/build-check.yml | 1 +
scripts/i18n/__tests__/korean-toolchain.test.cjs | 292 +++++++++++++++++++++
scripts/i18n/__tests__/translation-engine.test.cjs | 6 +-
scripts/i18n/check-translations.cjs | 3 +-
scripts/i18n/document-checks.cjs | 22 +-
scripts/i18n/translate-docs.cjs | 29 +-
scripts/i18n/workflow-generate.cjs | 5 +-
translations/README.md | 62 ++++-
8 files changed, 405 insertions(+), 15 deletions(-)
diff --git a/.github/workflows/build-check.yml
b/.github/workflows/build-check.yml
index 3528ea8df34..8b85a056ce0 100644
--- a/.github/workflows/build-check.yml
+++ b/.github/workflows/build-check.yml
@@ -107,6 +107,7 @@ jobs:
node --test scripts/i18n/__tests__/inventory.test.cjs
node --test scripts/i18n/__tests__/planner.test.cjs
node --test
scripts/i18n/__tests__/translation-engine.test.cjs
+ node --test scripts/i18n/__tests__/korean-toolchain.test.cjs
node --test scripts/i18n/__tests__/workflow.test.cjs
node --test scripts/i18n/__tests__/codex-adapter.test.cjs
node --test
scripts/i18n/__tests__/translate-document.test.cjs
diff --git a/scripts/i18n/__tests__/korean-toolchain.test.cjs
b/scripts/i18n/__tests__/korean-toolchain.test.cjs
new file mode 100644
index 00000000000..4c9702e85a2
--- /dev/null
+++ b/scripts/i18n/__tests__/korean-toolchain.test.cjs
@@ -0,0 +1,292 @@
+const test = require('node:test');
+const assert = require('node:assert/strict');
+const path = require('node:path');
+const {inspectDocument} = require('../document-checks.cjs');
+const {loadConfig} = require('../state.cjs');
+const settings = loadConfig(path.resolve(__dirname,'../../..')).locales.ko;
+const glossary = require('../../../translations/glossary/ko.json');
+const source = '---\ntitle: Guide\ndescription: Doris guide\nsidebar_label:
Guide\nlanguage: en\nkeywords: [Doris, guide]\nslug: /guide\n---\n#
Guide\n\nDoris guide.\n';
+const translated = '---\ntitle: 가이드\ndescription: Doris 사용 안내\nsidebar_label:
가이드\nlanguage: ko\nkeywords: [Doris, 가이드]\nslug: /guide\n---\n# 가이드\n\nDoris 사용
안내입니다.\n';
+const inspect = translation =>
inspectDocument({source,translation,extension:'.md',glossary,targetLocale:'ko',targetScriptPattern:settings.scriptPattern});
+
+test('Korean prose cannot silently retain a Japanese or English front-matter
language', async () => {
+ assert.deepEqual((await inspect(translated)).issues,[]);
+ for (const language of ['ja','en']) {
+ const result = await inspect(translated.replace('language:
ko',`language: ${language}`));
+ assert(result.issues.some(issue => issue.check === 'frontmatter' &&
issue.message.includes('language')));
+ }
+});
+
+test('complete Korean documents retain visible metadata fields and keyword
coverage', async () => {
+ for (const changed of [translated.replace('title: 가이드\n',''),
translated.replace('description: Doris 사용 안내\n',''),
+ translated.replace('keywords: [Doris, 가이드]','keywords: [Doris]')]) {
+ assert((await inspect(changed)).issues.some(issue => issue.check ===
'frontmatter'));
+ }
+});
+
+const fs = require('node:fs');
+const os = require('node:os');
+const {execFileSync} = require('node:child_process');
+const {hash,identity,stateFileFor,loadRecords} = require('../state.cjs');
+const {plan: makePlan} = require('../plan-translations.cjs');
+const {prepare,applyCandidates,applyStaged,readState,targetFor} =
require('../translate-docs.cjs');
+const {pairFor,copyPair} = require('../workflow-generate.cjs');
+const {translateWholeDocument} = require('../translate-document.cjs');
+const {codexAdapter} = require('../codex-adapter.cjs');
+const {checkLinks} = require('../document-checks.cjs');
+const {structure} = require('../structure.cjs');
+const scope = require('../../../translations/scope/ko.json');
+const repo = path.resolve(__dirname,'../../..');
+const mdxSource = '---\ntitle: Guide\nlanguage: en\nslug: /guide\n---\nimport
Card from "@site/src/Card";\n\n# Guide\n\nDoris guide.
[Next](./next.md)\n\n<Card title="Details" mode={1} />\n\n```sql\nSELECT
1;\n```\n';
+const mdxTarget = mdxSource.replace('title: Guide','title:
가이드').replace('language: en','language: ko')
+ .replace('# Guide','# 가이드').replace('Doris guide.','Doris 사용
안내입니다.').replace('[Next]','[다음]')
+ .replace('title="Details"','title="세부 정보"');
+
+function taskFor(docId,
{version='current',pluginId='default',sourcePath,content=mdxSource,state='untranslated'}
= {}) {
+ const group = scope.groups.find(group => group.pluginId === pluginId &&
group.version === version);
+ sourcePath ??= `${group.sourceRoot}/${docId}.mdx`;
+ const relative = sourcePath.slice(group.sourceRoot.length+1);
+ const plugin = pluginId==='default'?'':`-${pluginId}`;
+ const directory = version==='current'?'current':`version-${version}`;
+ return
{pluginId,version,docId,sourcePath,locale:'ko',sourceHash:hash(content),state,reviewStatus:'unverified',
+
targetHash:null,targetPath:`i18n/ko/docusaurus-plugin-content-docs${plugin}/${directory}/${relative}`,
+
permalink:`/ko/${pluginId==='default'?`docs/${version==='current'?'dev':version}`:'community'}/${docId}/`};
+}
+
+async function planFor(pages,{baseline=[],records=new
Map(),active,reports=[],chineseChangedPaths=[],files={},historical={}}={}) {
+ return
makePlan({scope,report:{schemaVersion:1,sourceCommit:'a'.repeat(40),baselineCommit:'b'.repeat(40),reports,
+
groups:scope.groups.map(group=>({...group,unregistered:[],entries:pages.filter(page=>page.pluginId===group.pluginId&&page.version===group.version)}))},
+ baseline,records,active,chineseChangedPaths,limit:20,
+
readCurrent:file=>files[file]??(pages.some(page=>page.sourcePath===file)?mdxSource:null),readAt:(ref,file)=>historical[`${ref}:${file}`]??null});
+}
+
+test('Korean dynamic plans isolate native identities, source updates, moves
and active PRs', async () => {
+ const
pages=[taskFor('new'),taskFor('same'),taskFor('same',{version:'4.x'}),taskFor('same',{pluginId:'community'}),
+
taskFor('done',{state:'current'}),taskFor('edited',{state:'stale_technical'}),
+
taskFor('moved',{sourcePath:'docs/new/moved.mdx',state:'state_conflict'}),taskFor('human',{state:'target_modified'})];
+ const baseline=[{...pages[4],disposition:'translate'},
+ {...pages[5],sourceHash:hash('old source'),disposition:'translate'},
+
{...pages[6],sourcePath:'docs/old/moved.mdx',disposition:'translate',targetHash:hash('이전
번역')},
+ {...taskFor('gone'),sourceHash:hash('removed
source'),disposition:'translate'}];
+ const
active={schemaVersion:1,candidates:[{...pages[0],sourceHash:hash('older
candidate'),pr:'#fixture-open'}]};
+ const
options={baseline,active,files:{'i18n/ko/docusaurus-plugin-content-docs/current/new/moved.mdx':'이전
번역'},
+
chineseChangedPaths:['i18n/zh-CN/docusaurus-plugin-content-docs/current/done.mdx'],
+
reports:[{kind:'unsupported_source_route',pluginId:'default',version:'current',sourceLocale:'zh-CN',sourcePath:'i18n/zh-CN/only.mdx'}]};
+ const result=await planFor(pages,options);
+ assert.deepEqual(result,await planFor(pages,options));
+ assert.equal(result.counts.mergedCurrent,1);
+ assert.equal(result.tasks.find(task=>task.docId==='edited').kind,'update');
+ assert.equal(result.tasks.find(task=>task.docId==='moved').kind,'move');
+ assert.equal(result.tasks.find(task=>task.docId==='gone').kind,'delete');
+
assert.equal(result.tasks.find(task=>task.docId==='human').requiresHumanReview,true);
+
assert(!result.selected.includes(result.tasks.find(task=>task.docId==='new').id));
+ assert(result.exceptions.some(item=>item.kind==='stale_active_candidate'));
+ assert(result.exceptions.some(item=>item.kind==='zh_only_change'));
+
assert(result.exceptions.some(item=>item.kind==='unsupported_source_route'));
+ assert.equal(new
Set(result.tasks.filter(task=>task.docId==='same').map(task=>task.id)).size,3);
+
assert(result.tasks.every(task=>task.locale==='ko'&&task.targetPath.startsWith('i18n/ko/')));
+ const afterClose=await
planFor(pages,{...options,active:{schemaVersion:1,candidates:[]}});
+
assert(afterClose.selected.includes(afterClose.tasks.find(task=>task.docId==='new').id));
+ assert.equal(pairFor({...pages[0],pluginId:'other'},pages,scope),null);
+});
+
+function fixture(t) {
+ const root=fs.mkdtempSync(path.join(os.tmpdir(),'ko-toolchain-'));
+ t.after(()=>fs.rmSync(root,{recursive:true,force:true}));
+ const write=(file,text)=>{const
full=path.join(root,file);fs.mkdirSync(path.dirname(full),{recursive:true});fs.writeFileSync(full,text);};
+ write('.gitignore','.i18n-build/\n.codex/\n');
+
write('translations/config.json',fs.readFileSync(path.join(repo,'translations/config.json'),'utf8'));
+ write('translations/scope/ko.json',JSON.stringify(scope));
+ write('translations/scope/ja.json',JSON.stringify({...scope,locale:'ja'}));
+
write('translations/state/ko/docs.json','{"schemaVersion":1,"pages":[]}\n');
+
write('translations/state/ko/shards.schema.json','{"schemaVersion":1,"format":"page-record-v1"}\n');
+
write('translations/state/ja/docs.json','{"schemaVersion":1,"pages":[],"sentinel":"unchanged"}\n');
+
write('scripts/check_md_links_single.py',fs.readFileSync(path.join(repo,'scripts/check_md_links_single.py'),'utf8'));
+ fs.mkdirSync(path.join(root,'static'));
+ write('docs/guide.mdx',mdxSource);write('docs/next.md','# Next\n');
+
write('versioned_docs/version-4.x/guide.mdx',mdxSource);write('versioned_docs/version-4.x/next.md','#
Next\n');
+ const
git=(...args)=>execFileSync('git',args,{cwd:root,encoding:'utf8',stdio:['ignore','pipe','pipe']}).trim();
+
git('init','-q');git('add','.');git('-c','user.name=Fixture','-c','[email protected]','commit','-qm','source');
+ const sourceCommit=git('rev-parse','HEAD');
+ const
primary={...taskFor('guide'),id:'T06-DEV-000000000000001',kind:'translate_new'};
+ const
partner={...taskFor('guide',{version:'4.x'}),id:'T06-4X-000000000000002',kind:'translate_new'};
+ const
plan={schemaVersion:1,locale:'ko',sourceCommit,tasks:[primary,partner],selected:[primary.id,partner.id]};
+ write('.codex/auth.json','{}'); // Fake snapshot for an injected process;
no real credentials/model.
+ return {root,write,git,plan,primary,partner};
+}
+
+function model(root,answer) {
+ let calls=0;const prompts=[];
+ const
adapter=codexAdapter({language:settings.languageName,locale:'ko',codexHome:path.join(root,'.codex'),
+ spawnImpl:(_bin,args,options)=>{
+
assert(args.includes('read-only')&&args.includes('--ignore-user-config'));
+ assert(!Object.hasOwn(options.env,'OPENAI_API_KEY'));
+ prompts.push(options.input);calls++;
+ const translation=typeof answer==='function'?answer(calls):answer;
+
fs.writeFileSync(args[args.indexOf('--output-last-message')+1],JSON.stringify({translation}));
+ return
{status:0,stdout:JSON.stringify({type:'turn.completed',usage:{input_tokens:10,output_tokens:10}})+'\n'};
+ }});
+ return {adapter,prompts,get calls(){return calls;}};
+}
+
+const generate = (data,state,adapter) =>
prepare({root:data.root,plan:data.plan,taskId:data.primary.id,scope,state,adapter,
+ glossary,target:targetFor(data.root,'ko')});
+
+test('Korean Codex fixtures retry metadata and SQL feedback, then apply
per-page shards and a safe Dev/4.x copy', async t => {
+ const data=fixture(t), state=readState(data.root,'ko');
+ const
mock=model(data.root,attempt=>attempt===1?mdxTarget.replace('language:
ko','language: ja')
+ :attempt===2?mdxTarget.replace('SELECT 1','SELECT 2'):mdxTarget);
+ const candidate=await generate(data,state,mock.adapter);
+ assert.equal(mock.calls,3);assert.equal(mock.adapter.usedTokens,60);
+ assert(mock.prompts.every(prompt=>prompt.includes('natural Korean')));
+ assert.equal(candidate.record.generation.acceptedWithWarnings,false);
+
assert(candidate.record.generation.history[0].issues.some(issue=>issue.check==='frontmatter'));
+
assert(candidate.record.generation.history[1].issues.some(issue=>issue.check==='protected-content'));
+ const
partner=pairFor(data.primary,data.plan.tasks,scope);assert.equal(partner.id,data.partner.id);
+
assert.equal(pairFor(data.primary,[{...data.partner,sourceHash:hash('different')}],scope),null);
+
assert.equal(pairFor(data.primary,[{...data.partner,sourcePath:'versioned_docs/version-4.x/other/guide.mdx'}],scope),null);
+ const copied=copyPair(candidate,partner,scope,state,data.root);
+
checkLinks({root:data.root,targetPath:copied.record.targetPath,target:copied.target});
+ const
ja=fs.readFileSync(path.join(data.root,'translations/state/ja/docs.json'),'utf8');
+ applyCandidates(state,[candidate,copied]);
+ const records=loadRecords(data.root,'ko');assert.equal(records.size,2);
+ for(const record of records.values()) {
+
assert.equal(record.locale,'ko');assert.equal(record.reviewStatus,'machine');assert.equal(record.ownership,'machine');
+ assert.equal(record.sourceCommit,data.plan.sourceCommit);
+ assert(fs.existsSync(path.join(data.root,stateFileFor(record))));
+ }
+
assert.equal(fs.readFileSync(path.join(data.root,'translations/state/ja/docs.json'),'utf8'),ja);
+
assert.equal(fs.readFileSync(path.join(data.root,'translations/state/ko/docs.json'),'utf8'),'{"schemaVersion":1,"pages":[]}\n');
+
assert.equal(fs.readFileSync(path.join(data.root,'docs/guide.mdx'),'utf8'),mdxSource);
+
assert.equal(fs.readFileSync(path.join(data.root,'versioned_docs/version-4.x/guide.mdx'),'utf8'),mdxSource);
+});
+
+test('Korean generation rejects source races, human ownership and cross-locale
tasks', async t => {
+ const data=fixture(t), mock=model(data.root,mdxTarget);
+ data.write('docs/guide.mdx',mdxSource+'\nChanged\n');
+ await
assert.rejects(generate(data,readState(data.root,'ko'),mock.adapter),/Source
changed/);
+ data.write('docs/guide.mdx',mdxSource);
+ const
human=readState(data.root,'ko');human.data.pages.push({...data.primary,ownership:'human'});
+ await assert.rejects(generate(data,human,mock.adapter),/Human maintained/);
+ const
wrong={...data,plan:{...data.plan,tasks:[{...data.primary,locale:'ja'}]}};
+ await
assert.rejects(generate(wrong,readState(data.root,'ko'),mock.adapter),/outside
the managed/);
+ assert.equal(mock.calls,0);
+});
+
+test('Korean candidate application refuses target edits and concurrent shard
creation', async t => {
+ const data=fixture(t), state=readState(data.root,'ko');
+ const candidate=await
generate(data,state,model(data.root,mdxTarget).adapter);
+ data.write(data.primary.targetPath,'human edit');
+ assert.throws(()=>applyCandidates(state,[candidate]),/Target changed/);
+
assert.equal(fs.readFileSync(path.join(data.root,data.primary.targetPath),'utf8'),'human
edit');
+ fs.rmSync(path.join(data.root,data.primary.targetPath));
+
data.write(stateFileFor(candidate.record),JSON.stringify({schemaVersion:1,page:{...candidate.record,ownership:'human'}}));
+ assert.throws(()=>applyCandidates(state,[candidate]),/State changed/);
+ assert(!fs.existsSync(path.join(data.root,data.primary.targetPath)));
+});
+
+test('a third wrong-language Korean fixture remains an explicitly warned
review candidate', async () => {
+ const result=await translateWholeDocument({source:'# Guide\n\nDoris
guide.\n',extension:'.md',glossary,
+
targetLocale:'ko',targetScriptPattern:settings.scriptPattern,adapter:{translateDocument:async()=>({translation:'#
ガイド\n\nDoris のガイド。\n'})}});
+
assert.equal(result.requests,3);assert.equal(result.generation.acceptedWithWarnings,true);
+ assert(result.generation.warnings.some(issue=>issue.check==='language'));
+ assert.equal(result.generation.history.length,3);
+});
+
+test('Korean staged application checks the exact bytes once, uses Ko shards
and rejects a Ja selector', async t => {
+ const data=fixture(t), state=readState(data.root,'ko');
+ const words={Guide:'가이드','Doris guide.':'Doris 사용
안내입니다.',Next:'다음',Details:'세부 정보'};
+ const
adapter={model:'fixture',translate:async({units})=>({translations:units.map(unit=>({id:unit.id,text:words[unit.text]??unit.text}))})};
+ const candidate=await generate(data,state,adapter);
+ assert.equal(candidate.record.generation,undefined,'Legacy unit-based
provider must not claim a Codex retry history');
+ const directory='.i18n-build/reviewed';
+ const planFile=path.join(data.root,'.i18n-build/plan.json');
+ data.write('.i18n-build/plan.json',JSON.stringify(data.plan));
+ const candidatePath=`${directory}/${data.primary.id}/candidate.mdx`;
+ data.write(candidatePath,candidate.target);
+ // Synthetic staging protocol, not a live provider call or live
verification.
+
data.write(`${directory}/manifest.json`,JSON.stringify({schemaVersion:2,provider:'openai',liveCall:true,liveVerified:false,
+
applied:false,sourceCommit:data.plan.sourceCommit,glossaryVersion:glossary.version,candidates:[{
+
id:data.primary.id,sourceHash:data.primary.sourceHash,sourcePath:data.primary.sourcePath,
+
targetPath:data.primary.targetPath,targetHash:candidate.record.targetHash,targetChanged:true,
+ baseline:candidate.baseline,record:candidate.record}]}));
+ const previous=process.env.I18N_LOCALE;
+ try {
+ process.env.I18N_LOCALE='ja';
+ await
assert.rejects(applyStaged(data.root,planFile,directory),/Invalid translation
plan/);
+ process.env.I18N_LOCALE='ko';
+ data.write(candidatePath,candidate.target+'\n나중에 편집\n');
+ await assert.rejects(applyStaged(data.root,planFile,directory),/edited
after validation/);
+ data.write(candidatePath,candidate.target);
+ const result=await applyStaged(data.root,planFile,directory);
+
assert.equal(result.applied,true);assert.equal(result.liveVerified,false);
+
assert.equal(fs.readFileSync(path.join(data.root,data.primary.targetPath),'utf8'),candidate.target);
+
assert.equal(loadRecords(data.root,'ko').get(identity(candidate.record)).targetHash,hash(candidate.target));
+ assert.equal(loadRecords(data.root,'ja').size,0);
+
assert.equal(fs.readFileSync(path.join(data.root,'translations/state/ko/docs.json'),'utf8'),'{"schemaVersion":1,"pages":[]}\n');
+ await
assert.rejects(applyStaged(data.root,planFile,directory),/Invalid staged live
candidates/);
+ } finally {if(previous===undefined)delete process.env.I18N_LOCALE;else
process.env.I18N_LOCALE=previous;}
+});
+
+test('Korean staged candidates cannot replace a newer consistent machine
target and state', async t => {
+ const data=fixture(t), state=readState(data.root,'ko');
+ const words={Guide:'가이드','Doris guide.':'Doris 사용
안내입니다.',Next:'다음',Details:'세부 정보'};
+ const candidate=await
generate(data,state,{model:'fixture',translate:async({units})=>({
+
translations:units.map(unit=>({id:unit.id,text:words[unit.text]??unit.text}))})});
+ const directory='.i18n-build/older-candidate';
+ data.write('.i18n-build/plan.json',JSON.stringify(data.plan));
+
data.write(`${directory}/${data.primary.id}/candidate.mdx`,candidate.target);
+
data.write(`${directory}/manifest.json`,JSON.stringify({schemaVersion:2,provider:'openai',liveCall:true,
+
liveVerified:false,applied:false,sourceCommit:data.plan.sourceCommit,glossaryVersion:glossary.version,
+
candidates:[{id:data.primary.id,sourceHash:data.primary.sourceHash,sourcePath:data.primary.sourcePath,
+
targetPath:data.primary.targetPath,targetHash:candidate.record.targetHash,
+
targetChanged:true,baseline:candidate.baseline,record:candidate.record}]}));
+ const newer=candidate.target.replace('Doris 사용 안내입니다.','Doris의 새로운 사용
안내입니다.');
+ data.write(data.primary.targetPath,newer);
+
data.write(stateFileFor(candidate.record),JSON.stringify({schemaVersion:1,page:{...candidate.record,targetHash:hash(newer)}}));
+ const previous=process.env.I18N_LOCALE;
+ try {
+ process.env.I18N_LOCALE='ko';
+ await
assert.rejects(applyStaged(data.root,path.join(data.root,'.i18n-build/plan.json'),directory),/baseline
changed since staging/);
+
assert.equal(fs.readFileSync(path.join(data.root,data.primary.targetPath),'utf8'),newer);
+
assert.equal(loadRecords(data.root,'ko').get(identity(candidate.record)).targetHash,hash(newer));
+ } finally {if(previous===undefined)delete process.env.I18N_LOCALE;else
process.env.I18N_LOCALE=previous;}
+});
+
+test('Korean formatting-only source changes require review and keep the
recorded hashes', async () => {
+ const before=mdxSource, after=mdxSource+'\n';
+ const page=taskFor('formatted',{content:after,state:'stale_unknown'});
+ const
old={...page,sourceHash:hash(before),sourceCommit:'b'.repeat(40),targetHash:hash('이전
번역'),disposition:'translate'};
+ const result=await
planFor([page],{baseline:[old],files:{[page.sourcePath]:after},historical:{[`${old.sourceCommit}:${page.sourcePath}`]:before}});
+ assert.equal(result.tasks[0].kind,'formatting_review');
+ assert.equal(result.tasks[0].requiresHumanReview,true);
+ assert.deepEqual(result.selected,[]);
+ assert.equal(old.sourceHash,hash(before));
+});
+
+test('Korean legacy state remains readable and an existing page shard wins
without a marker', async t => {
+ const data=fixture(t);
+ fs.rmSync(path.join(data.root,'translations/state/ko/shards.schema.json'));
+
data.git('add','-u');data.git('-c','user.name=Fixture','-c','[email protected]','commit','-qm','legacy
layout');
+ data.plan.sourceCommit=data.git('rev-parse','HEAD');
+ const state=readState(data.root,'ko');
+ const candidate=await
generate(data,state,model(data.root,mdxTarget).adapter);
+ applyCandidates(state,[candidate]);
+
assert(!fs.existsSync(path.join(data.root,stateFileFor(candidate.record))));
+ assert.equal(loadRecords(data.root,'ko').size,1);
+
data.write(stateFileFor(candidate.record),JSON.stringify({schemaVersion:1,page:{...candidate.record,ownership:'human'}}));
+
assert.equal(loadRecords(data.root,'ko').get(identity(candidate.record)).ownership,'human');
+ await
assert.rejects(generate(data,readState(data.root,'ko'),model(data.root,mdxTarget).adapter),/Human
maintained/);
+});
+
+test('Korean generation rejects versions outside scope, path drift and symlink
targets without a model call', async t => {
+ const data=fixture(t), mock=model(data.root,mdxTarget);
+ for (const changed of [{version:'3.x'},
{targetPath:'i18n/ja/docusaurus-plugin-content-docs/current/guide.mdx'},
+ {sourcePath:'docs/../guide.mdx'}]) {
+ const plan={...data.plan,tasks:[{...data.primary,...changed}]};
+ await
assert.rejects(generate({...data,plan},readState(data.root,'ko'),mock.adapter),/outside|managed|target|Invalid/i);
+ }
+ fs.symlinkSync(os.tmpdir(),path.join(data.root,'i18n'));
+ await
assert.rejects(generate(data,readState(data.root,'ko'),mock.adapter),/Symlink/);
+ assert.equal(mock.calls,0);
+});
diff --git a/scripts/i18n/__tests__/translation-engine.test.cjs
b/scripts/i18n/__tests__/translation-engine.test.cjs
index 7d98b66201d..e775076ad58 100644
--- a/scripts/i18n/__tests__/translation-engine.test.cjs
+++ b/scripts/i18n/__tests__/translation-engine.test.cjs
@@ -174,10 +174,12 @@ test('staged live candidate applies the exact reviewed
bytes and state once', as
sourceCommit, sourceHash: hash(source), sourceTechnicalHash:
(await structure(source, '.md')).technicalHash,
targetHash: hash(generated.target), reviewStatus: 'machine',
ownership: 'machine',
translatedAt: '2026-10-13T00:00:00.000Z', glossaryVersion:
glossary.version};
- fs.writeFileSync(path.join(stage, 'manifest.json'),
JSON.stringify({schemaVersion: 1, provider: 'openai',
+ fs.writeFileSync(path.join(stage, 'manifest.json'),
JSON.stringify({schemaVersion: 2, provider: 'openai',
liveCall: true, liveVerified: false, applied: false, sourceCommit,
glossaryVersion: glossary.version,
candidates: [{id, sourceHash: hash(source), sourcePath:
planned.sourcePath,
- targetPath: planned.targetPath, targetHash:
hash(generated.target), targetChanged: true, record}]}));
+ targetPath: planned.targetPath, targetHash:
hash(generated.target), targetChanged: true,
+ baseline: {targetHash: null, statePath:
'translations/state/ja/docs.json',
+ stateHash: hash('{"schemaVersion":1,"pages":[]}\n'),
schemaHash: null}, record}]}));
fs.writeFileSync(path.join(root, 'docs/a.md'), '# Changed\n');
await assert.rejects(applyStaged(root, planFile,
'.i18n-build/staged'), /Source changed since planning/);
assert.equal(fs.existsSync(path.join(root, planned.targetPath)),
false);
diff --git a/scripts/i18n/check-translations.cjs
b/scripts/i18n/check-translations.cjs
index f88fd593977..5ff70333cbd 100644
--- a/scripts/i18n/check-translations.cjs
+++ b/scripts/i18n/check-translations.cjs
@@ -6,7 +6,7 @@ const {identity, hash, loadRecords, loadRecordsAt,
pipelineLocale} = require('./
const {structure, alignHeadings} = require('./structure.cjs');
const {collect} = require('./inventory.cjs');
const {protectionSignature} = require('./protection.cjs');
-const {signature, isDocumentGeneration, acceptedFallback} =
require('./document-checks.cjs');
+const {signature, isDocumentGeneration, acceptedFallback, checkFrontMatter} =
require('./document-checks.cjs');
async function checkTranslations({root, locale = pipelineLocale(root), pages,
reports = [], files, records, previousRecords = new Map()}) {
const changed = new Set(files);
@@ -55,6 +55,7 @@ async function checkTranslations({root, locale =
pipelineLocale(root), pages, re
const ext = path.extname(page.sourcePath);
const parsed = await structure(source, ext);
const fallback = acceptedFallback(record, source, target);
+ if (!fallback) checkFrontMatter(source, target, locale);
let aligned = target;
try {aligned = await alignHeadings(parsed, target, ext);}
catch (error) {if (!fallback) throw error;}
diff --git a/scripts/i18n/document-checks.cjs b/scripts/i18n/document-checks.cjs
index 49cfc6e748b..249a4a748bf 100644
--- a/scripts/i18n/document-checks.cjs
+++ b/scripts/i18n/document-checks.cjs
@@ -1,6 +1,8 @@
// Shared by generation retries and the PR preflight. Link resolution is the
// same Python implementation used by check-md-links.yml.
+const assert = require('node:assert/strict');
const fs = require('node:fs');
+const matter = require('gray-matter');
const os = require('node:os');
const path = require('node:path');
const {spawnSync} = require('node:child_process');
@@ -71,6 +73,23 @@ async function checkSyntax(target, extension) {
}
}
+function checkFrontMatter(source, target, locale) {
+ const before = matter(source).data;
+ const after = matter(target).data;
+ for (const key of ['title','description','sidebar_label','keywords']) {
+ if (!Object.hasOwn(before,key)) continue;
+ assert(Object.hasOwn(after,key), `Restore front matter field: ${key}`);
+ if (Array.isArray(before[key])) {
+ assert(Array.isArray(after[key]) && after[key].length ===
before[key].length
+ && after[key].every(value => typeof value === 'string' &&
value.trim()), `Restore complete front matter field: ${key}`);
+ } else if (typeof before[key] === 'string') {
+ assert(typeof after[key] === 'string' && (!before[key].trim() ||
after[key].trim()), `Restore front matter field: ${key}`);
+ } else assert.deepEqual(after[key],before[key],`Restore front matter
field: ${key}`);
+ }
+ if (Object.hasOwn(before,'language') || Object.hasOwn(after,'language'))
+ assert.equal(after.language,locale,`Set front matter language to
${locale}`);
+}
+
async function inspectDocument({source, translation, extension, glossary,
targetLocale, targetScriptPattern,
root, targetPath, preserveFormatting = false}) {
let target =
translation.replace(/^```(?:markdown|mdx?)\s*\n([\s\S]*)\n```\s*$/, '$1');
@@ -88,6 +107,7 @@ async function inspectDocument({source, translation,
extension, glossary, target
if (!preserveFormatting) target = aligned;
});
await check('syntax', () => checkSyntax(target, extension));
+ await check('frontmatter', () => checkFrontMatter(source, target,
targetLocale));
await check('protected-content', async () => {
const translated = await structure(aligned, extension);
if (signature(translated) !== signature(parsed))
@@ -112,4 +132,4 @@ async function inspectDocument({source, translation,
extension, glossary, target
return {target, issues};
}
-module.exports = {policy, signature, isDocumentGeneration, acceptedFallback,
checkLinks, checkSyntax, inspectDocument};
+module.exports = {policy, signature, isDocumentGeneration, acceptedFallback,
checkLinks, checkSyntax, checkFrontMatter, inspectDocument};
diff --git a/scripts/i18n/translate-docs.cjs b/scripts/i18n/translate-docs.cjs
index c4136741c26..026f8555d4d 100644
--- a/scripts/i18n/translate-docs.cjs
+++ b/scripts/i18n/translate-docs.cjs
@@ -9,6 +9,7 @@ const {structure} = require('./structure.cjs');
const {protectionSignature} = require('./protection.cjs');
const {checkedTask, safeFile, openAIAdapter, mockAdapter, translateDocument} =
require('./translation-engine.cjs');
const {translateManagedDocument} = require('./translate-document.cjs');
+const {checkFrontMatter} = require('./document-checks.cjs');
function optionsFrom(argv) {
const values = {}, tasks = [];
@@ -76,6 +77,17 @@ function statePathForRecord(state, record) {
return state.sharded || state.shards.has(shard) ? shard :
relative(state.root, state.file);
}
+// Capture the inputs from the generation snapshot, before awaiting a model.
+// A later apply must not overwrite a newer, internally consistent translation.
+function stagedBaseline(state, task, previousTarget) {
+ const shard = stateFileFor(task);
+ const statePath = state.shards.has(shard) ? shard : relative(state.root,
state.file);
+ const raw = state.shards.has(shard) ? state.shards.get(shard) : state.raw;
+ return {targetHash: previousTarget === null ? null : hash(previousTarget),
statePath,
+ stateHash: raw === null ? null : hash(raw),
+ schemaHash: state.schemaRaw === null ? null : hash(state.schemaRaw)};
+}
+
function writeAtomic(file, value) {
const temp = path.join(path.dirname(file),
`.translation-${randomUUID()}.tmp`);
try {
@@ -96,6 +108,7 @@ async function prepare({root, plan, taskId, scope, state,
adapter, glossary, tar
&& previousTarget !== null && hash(previousTarget) ===
existing.targetHash,
'Existing target or identity changed; manual review required');
}
+ const baseline = stagedBaseline(state, task, previousTarget);
const wholeDocument = typeof adapter.translateDocument === 'function';
let previousSource = null, baselineReason;
if (existing) {
@@ -119,6 +132,7 @@ async function prepare({root, plan, taskId, scope, state,
adapter, glossary, tar
previousTarget: canReuse ? previousTarget : null,
targetLocale: target.locale, targetScriptPattern: target.scriptPattern,
maxDocumentChars, maxChunkChars});
+ if (!wholeDocument) checkFrontMatter(source, generated.target,
target.locale);
if (!wholeDocument && adapter.model !== 'fixture') {
assert(generated.targetChanged,
`Live candidate contains no ${target.language} text; review
manually`);
@@ -131,7 +145,7 @@ async function prepare({root, plan, taskId, scope, state,
adapter, glossary, tar
reviewStatus: 'machine', ownership: 'machine', translatedAt: new
Date().toISOString(),
glossaryVersion: glossary.version,
...(generated.generation ? {generation: generated.generation} : {})};
- return {task, targetFile, previousTarget, target: generated.target, record,
+ return {task, targetFile, previousTarget, baseline, target:
generated.target, record,
units: generated.units, reusedUnits: generated.reusedUnits, requests:
generated.requests,
targetChanged: generated.targetChanged};
}
@@ -197,7 +211,7 @@ async function applyStaged(root, planFile, relativeStage) {
const scope = JSON.parse(fs.readFileSync(scopeFile(root, locale), 'utf8'));
const stage = stageDirectory(root, relativeStage);
const manifest = JSON.parse(fs.readFileSync(path.join(stage,
'manifest.json'), 'utf8'));
- assert(manifest.schemaVersion === 1 && manifest.provider === 'openai' &&
manifest.liveCall === true
+ assert(manifest.schemaVersion === 2 && manifest.provider === 'openai' &&
manifest.liveCall === true
&& manifest.applied === false && manifest.sourceCommit ===
plan.sourceCommit
&& Array.isArray(manifest.candidates) && manifest.candidates.length > 0
&& manifest.candidates.length <= scope.maxBatchPages, 'Invalid staged
live candidates');
@@ -225,11 +239,14 @@ async function applyStaged(root, planFile, relativeStage)
{
'Staged candidate was edited after validation');
const parsed = await structure(source, path.extname(task.sourcePath));
const translated = await structure(target,
path.extname(task.targetPath));
+ checkFrontMatter(source, target, locale);
assert.equal(protectionSignature(translated),
protectionSignature(parsed), 'Staged protected content changed');
assert.equal(item.record.sourceTechnicalHash, parsed.technicalHash,
'Staged source technical hash changed');
const existing = state.data.pages.find(page => identity(page) ===
identity(task));
const targetFile = safeFile(root, task.targetPath);
const previousTarget = fs.existsSync(targetFile) ?
fs.readFileSync(targetFile, 'utf8') : null;
+ assert.deepEqual(item.baseline, stagedBaseline(state, task,
previousTarget),
+ 'Translation baseline changed since staging; regenerate and review
the candidate');
assert(!previousTarget || existing, 'Untracked target needs human
review');
if (existing) assert(existing.sourcePath === task.sourcePath &&
existing.targetPath === task.targetPath
&& previousTarget !== null && hash(previousTarget) ===
existing.targetHash,
@@ -282,14 +299,14 @@ async function main() {
const stageRelative = options['--stage-dir'] ??
'.i18n-build/translation-candidates';
const stage = stageDirectory(root, stageRelative);
fs.mkdirSync(stage, {recursive: true});
- const manifest = {schemaVersion: 1, sourceCommit: plan.sourceCommit,
provider,
+ const manifest = {schemaVersion: 2, sourceCommit: plan.sourceCommit,
provider,
model: adapter.model, glossaryVersion: glossary.version,
liveCall: provider === 'openai', liveVerified: false,
applied: false, usedTokens: adapter.usedTokens,
- candidates: candidates.map(({task, record, units, reusedUnits,
requests, targetChanged}) => ({
+ candidates: candidates.map(({task, record, baseline, units,
reusedUnits, requests, targetChanged}) => ({
id: task.id, kind: task.kind, sourcePath: task.sourcePath,
targetPath: task.targetPath,
sourceHash: record.sourceHash, targetHash: record.targetHash,
units, reusedUnits, requests,
- targetChanged, record,
+ targetChanged, baseline, record,
}))};
for (const candidate of candidates) {
const folder = path.join(stage, candidate.task.id);
@@ -301,4 +318,4 @@ async function main() {
}
if (require.main === module) main().catch(error =>
{console.error(error.message); process.exitCode = 1;});
-module.exports = {optionsFrom, prepare, applyCandidates, applyStaged,
readState, statePathForRecord, targetFor, scopeFile, glossaryFile};
+module.exports = {optionsFrom, prepare, applyCandidates, applyStaged,
readState, stagedBaseline, statePathForRecord, targetFor, scopeFile,
glossaryFile};
diff --git a/scripts/i18n/workflow-generate.cjs
b/scripts/i18n/workflow-generate.cjs
index 0b7109b4ce1..9a6b4bde2f9 100644
--- a/scripts/i18n/workflow-generate.cjs
+++ b/scripts/i18n/workflow-generate.cjs
@@ -1,5 +1,5 @@
// Trusted-master automation only. The workflow supplies the requested mode and
-// model credential; this module writes only validated managed ja files/state.
+// model credential; this module writes only validated files/state for the
selected managed locale.
const assert = require('node:assert/strict');
const fs = require('node:fs');
const path = require('node:path');
@@ -7,7 +7,7 @@ const {execFileSync} = require('node:child_process');
const {hash, identity, languageName, pipelineLocale} = require('./state.cjs');
const {checkedTask, managedPath, safeFile} =
require('./translation-engine.cjs');
const {codexAdapter} = require('./codex-adapter.cjs');
-const {prepare, applyCandidates, readState, statePathForRecord, targetFor} =
require('./translate-docs.cjs');
+const {prepare, applyCandidates, readState, stagedBaseline,
statePathForRecord, targetFor} = require('./translate-docs.cjs');
const {markerPrefixFor} = require('./automation-naming.cjs');
const root = path.resolve(__dirname, '../..');
@@ -51,6 +51,7 @@ function copyPair(primary, partner, scope, state, rootDir =
root) {
docId: partner.docId, sourcePath: partner.sourcePath, targetPath:
partner.targetPath,
sourceHash: partner.sourceHash};
return {...primary, task: partner, targetFile, previousTarget, record,
+ baseline: stagedBaseline(state, partner, previousTarget),
requests: 0, reusedFrom: primary.task.id};
}
diff --git a/translations/README.md b/translations/README.md
index 069c11bd858..c710e71a3fb 100644
--- a/translations/README.md
+++ b/translations/README.md
@@ -47,6 +47,62 @@ Translation commands must select their locale explicitly now
that both `ja`
and `ko` are configured, for example `I18N_LOCALE=ko`. The site build flags
select rendered languages; they do not select the translation pipeline locale.
+## Korean toolchain and script-driven execution
+
+Korean uses its own `scope/ko.json`, `glossary/ko.json`, inventory and page
state.
+Select the pipeline explicitly with `I18N_LOCALE=ko`; rendering flags do not
+select a translation pipeline. A loop can recount and plan from the current
+committed checkout without fixed batch registration:
+
+```bash
+I18N_LOCALE=ko node scripts/i18n/audit-coverage.cjs --source-ref HEAD --output
/tmp/ko-coverage.json
+I18N_LOCALE=ko node scripts/i18n/plan-translations.cjs --source-ref HEAD
--limit 10 \
+ --active-candidates /tmp/ko-active-candidates.json --output /tmp/ko-plan.json
+```
+
+The active-candidates file must come from the current open PR scan. A
standalone
+plan without it does not establish cross-PR deduplication. Replan after source,
+target/state, HEAD or active PR changes; use the selected native task IDs
rather
+than historical inventory batch IDs. Reports on a local feature-branch commit
+can include local candidates; merged coverage is recounted on merged master.
+The loop is implemented separately in its own worktree.
+
+The trusted-master Codex workflow translates whole documents or eligible
changed
+blocks, with at most three attempts and cumulative feedback. A third-attempt
+candidate can retain content warnings: inspect
`generation.acceptedWithWarnings`,
+`generation.warnings` and `generation.history` before publication. This is a
+review candidate, not a language/technical quality pass. Review status remains
+machine until actual human proofreading.
+
+The independent `translate-docs.cjs --provider openai` entry uses protected
text
+units and its own validation; it is not the Codex workflow. It stages live
bytes
+for explicit application. The locale selector also applies to that step:
+
+```bash
+I18N_LOCALE=ko node scripts/i18n/translate-docs.cjs \
+ --apply-staged .i18n-build/<reviewed-stage> --plan /tmp/ko-plan.json
+```
+
+Application binds current HEAD, source bytes, reviewed target bytes, state and
+ownership. Manual staging manifests use schema version 2 and freeze the
original
+target, effective state file and shard-layout marker hashes before generation.
+Application refuses a newer matching target/state pair as well as inconsistent
+edits. Regenerate older version-1 stages before applying them; page state and
+Codex workflow artifact schemas remain version 1.
+
+Korean starts with the page-record marker and writes independent
+shards. Legacy records remain readable and existing shards retain precedence.
+Human-owned targets, edited target/state bytes, path drift and symlinks require
+review. Dev/4.x reuse requires the same doc ID, relative filename and full
source
+hash; an existing partner is updated against its own baseline, not overwritten
+by a copy. Each reused target still receives its own link/state admission
checks.
+
+Generation and admission retain visible front matter and keyword entries, and
+check any declared `language` against the target locale. These checks leave the
+existing source/technical hash algorithms unchanged. Offline Korean fixtures
+exercise these boundaries without real credentials or model calls; actual
+single-page generation is a separate validation step.
+
## State schema, version 1
`state/ja/docs.json` is the legacy shared ledger. New Japanese records can live
@@ -173,7 +229,7 @@ commit. The 2026-10-12 inventory is only a comparison
baseline. New registered
pages enter the plan even when they have never appeared in that inventory.
```bash
-node scripts/i18n/plan-translations.cjs --source-ref HEAD \
+I18N_LOCALE=ja node scripts/i18n/plan-translations.cjs --source-ref HEAD \
--limit 20 --output /tmp/japanese-plan.json
```
@@ -309,7 +365,7 @@ node scripts/i18n/plan-translations.cjs --source-ref HEAD \
--output /tmp/japanese-plan.json
export OPENAI_API_KEY='<secret>'
export I18N_MODEL='<explicit-model-id>'
-node scripts/i18n/translate-docs.cjs --plan /tmp/japanese-plan.json \
+I18N_LOCALE=ja node scripts/i18n/translate-docs.cjs --plan
/tmp/japanese-plan.json \
--task '<selected-task-id>' --stage-dir .i18n-build/translation-candidates
```
@@ -319,7 +375,7 @@ Review the candidate and manifest, then apply those **same
bytes** without
another model call:
```bash
-node scripts/i18n/translate-docs.cjs \
+I18N_LOCALE=ja node scripts/i18n/translate-docs.cjs \
--apply-staged .i18n-build/translation-candidates \
--plan /tmp/japanese-plan.json
```
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
