fhan688 opened a new pull request, #20410:
URL: https://github.com/apache/hudi/pull/20410
### Describe the issue this Pull Request addresses
This is a focused follow-up to #20391 and the Hudi Architect work tracked
in #19779.
The existing-table reconciliation validator requires every evidence
envelope to declare the target table path. It also accepts an independent
`hudi_table_identity.table_path` fact so that the identity of the observed
table can be verified.
When the evidence envelope is scoped to the desired table but its inner
identity fact names a different path, the current implementation classifies the
mismatch as a durable table-property conflict. This produces `BLOCKED`,
`CONFLICT`, and `MIGRATION_REQUIRED`.
However, `hudi_table_identity.table_path` is provenance evidence rather
than a persisted Hudi table property. A disagreement between the evidence scope
and the identity fact proves that the evidence is contradictory, but it does
not prove that the target table requires migration.
### Summary and Changelog
This PR tightens the existing-table reconciliation semantics for
contradictory table identity evidence.
Changes include:
- Classify an in-scope identity path mismatch as
`FLINK_EXISTING_EVIDENCE_CONFLICT`.
- Report the corresponding comparison as `UNVERIFIED` instead of
`CONFLICT`.
- Produce `REVIEW_REQUIRED`, `UNVERIFIED` reconciliation, and `UNVERIFIED`
change disposition for this case.
- Retain the identity evidence `source_ids` in the comparison output for
traceability.
- Add regression coverage for the complete status, reconciliation result,
change disposition, finding code, compared paths, and provenance.
- Document that contradictory identity evidence must not be used to infer
a table migration.
- Extend the `FLINK_EXISTING_EVIDENCE_CONFLICT` documentation with this
trigger.
No code was copied.
### Impact
This change only affects existing-table assessments where:
1. The outer evidence envelope is scoped to the desired table.
2. The inner `hudi_table_identity.table_path` fact identifies a different
path.
For this contradictory evidence case, the result changes as follows:
- Previous result: `BLOCKED / CONFLICT / MIGRATION_REQUIRED`
- New result: `REVIEW_REQUIRED / UNVERIFIED / UNVERIFIED`
Matching identity evidence and evidence envelopes scoped to another table
retain their existing behavior.
There are no changes to public runtime APIs, configuration keys, generated
Flink SQL, table metadata, or execution eligibility. The validator remains
assessment-only and does not perform table mutations or migration actions.
The reconciliation schema remains unchanged because this PR reuses the
existing finding code, comparison result, status, and change-disposition
vocabulary.
There is no performance impact.
### Risk Level
Low.
The change is limited to the classification of one contradictory evidence
case and moves the result in a more conservative direction. It prevents
untrusted identity evidence from producing a deterministic migration conclusion.
Verification completed:
- Full Gateway test suite on Python 3.11: `374 passed, 4 skipped`.
- Full Gateway test suite on Python 3.12: `374 passed, 4 skipped`.
- Ruff checks passed for the CI-configured source and validator paths.
- mypy passed for 22 source files.
- Hudi Architect configuration-key and pinned capability validation passed.
- Flink 1.20.1/Hudi 1.2.0 SQL fixtures: `23 passed`.
- Hudi Architect skill validation passed.
### Documentation Update
Updated the Hudi Architect existing-table reconciliation reference to
clarify that:
- The outer evidence path determines whether the evidence is in scope.
- The inner identity path independently verifies the observed table
identity.
- A disagreement between them is an evidence conflict and must not imply
that the target table requires migration.
Updated the `FLINK_EXISTING_EVIDENCE_CONFLICT` warning reference with the
new trigger.
No Hudi website update is required because this PR does not add a new
feature, configuration option, or public runtime API.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
