fhan688 opened a new pull request, #20391:
URL: https://github.com/apache/hudi/pull/20391
### Describe the issue this Pull Request addresses
Related to #19779.
The previous Flink SQL work established bounded design contracts for
append-only COW, mutable COW, and mutable MOR sinks. Applying these contracts
to an existing Hudi table additionally requires checking persisted table
metadata, physical schema, and writer-transition evidence.
This PR adds an assessment-only reconciliation path. It distinguishes
verified matches, known conflicts, and unresolved evidence without treating an
existing-table assessment as permission to generate or execute SQL.
### Summary and Changelog
- Add a deterministic existing-table reconciliation validator for the
pinned Hudi 1.2.0 / Flink 1.20 baseline, reusing desired contract schemas 1, 2,
and 3.
- Introduce reconciliation schema 1 with structured evidence, unique
evidence IDs, observation timestamps, source-artifact SHA-256 values, and
explicit table-path scope.
- Compare the desired contract against authoritative Hudi table
configuration and physical schema. Flink DDL cannot override these sources, and
conflicting evidence remains visible.
- Keep persisted table properties separate from writer configuration and
time-sensitive operational observations.
- Require recoverable-state evidence and matching applicable writer
settings for same-job restores. Require the previous writer to be stopped for
cold starts and retain an index-bootstrap deployment check for mutable
workloads.
- Preserve independent findings and select one deterministic final status.
Distinguish schema conflicts from unverified schema evolution and unknown
key-generator semantics.
- Report `COMPATIBLE`, `CONFLICT`, or `UNVERIFIED`, together with a
bounded change disposition. Classify an assessment as
`DEPLOYMENT_CONFIGURATION_CHANGE` only when writer-configuration differences
are the only remaining findings.
- Keep existing-table assessments non-executable, including successful
matches. Do not generate SQL, table mutations, or migration commands.
- Extend the capability manifest to schema 5 and update the skill
instructions, reference documentation, regression tests, and Gateway CI checks.
Validation performed locally:
- Python 3.11: 371 tests passed and 4 live-stack tests skipped.
- Python 3.12: 371 tests passed and 4 live-stack tests skipped.
- Pinned Java fixtures: 23 tests passed, including existing-table type
precedence and record-key/ordering-field enrichment.
- Ruff, mypy, capability source verification, configuration-key
validation, CLI smoke checks, Helm lint/template checks, and skill validation
passed.
- Independent Agent forward evaluation covered incomplete restore
evidence, concurrent-writer findings combined with missing evidence, and an
executable-SQL request for a compatible existing table.
### Impact
This adds a user-facing assessment capability to Hudi Architect. It does
not change Hudi runtime behavior, storage formats, connector configuration
defaults, or public runtime APIs.
Existing-table reconciliation consumes normalized, user-confirmed
evidence; it does not inspect a live deployment or independently authenticate
the supplied artifacts.
`CONFIG_VALIDATED` means the bounded static assessment is complete. It
does not establish production readiness, and `executable_eligible` remains
false.
Separate-writer takeover, cross-engine takeover, schema evolution, table
upgrades, external catalogs, multi-writer operation, and unresolved pending
instants are not automatically approved. Writer inventory and timeline state
must be rechecked before deployment.
The existing new-table SQL generation paths remain covered by regression
tests.
### Risk Level
medium
Incorrect reconciliation could produce misleading compatibility guidance.
The implementation mitigates this through explicit evidence requirements,
source precedence, conservative handling of unsupported cases, preservation of
independent findings, and an assessment-only output boundary.
Verification includes positive and negative reconciliation tests, pinned
Hudi/Flink Java behavior tests, full Gateway regression tests on Python 3.11
and 3.12, and independent Agent forward evaluation.
Live-stack integration tests were not run. The synthetic fixtures and
static assessments do not prove production recovery or deployment safety.
### Documentation Update
Updated the Hudi Architect skill instructions, README, runbook, capability
references, question flow, decision guidance, warnings, and output templates.
Added a dedicated existing-table reconciliation reference documenting
evidence requirements, writer transitions, comparison semantics, deployment
checks, and unsupported cases.
No Hudi runtime configuration options or defaults are changed. This PR
does not include a Hudi website documentation update.
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
