LeonxLJX opened a new pull request, #19832: URL: https://github.com/apache/hudi/pull/19832
## Summary Persist the index type a table was created with into `hoodie.properties` under a new, namespaced key `hoodie.table.index.type` (HUDI-37, mirrored from uber/hudi#409). `HoodieTableConfig.create()` now copies the write-time index configuration — `hoodie.index.class` if set, otherwise `hoodie.index.type` — into the new `INDEX_TYPE` table config before the properties file is first written. ## Why this shape (root cause) Hudi persists several one-time, creation-time configs (table type, timeline layout, timezone...) but the index choice is not among them. The creation-time index is therefore only known to the writer that created the table, and no reader or operator can cheaply validate the index used by later writes against the one the table was built with. That is exactly the hazard vinoth described in the issue: a dataset written with BloomIndex all along can silently switch to a cold HBaseIndex and start producing duplicates because the new index has no state. This PR deliberately **records rather than enforces**: failing a job whose index differs from the recorded one changes runtime behaviour for existing pipelines and deserves its own change with a migration/upgrade discussion. ## Scope decisions 1. `hoodie.index.class` takes precedence over `hoodie.index.type`, mirroring the resolution order in `HoodieIndexConfig`. 2. Only persisted when explicitly present in the creation config. Tables created with engine defaults are byte-identical to before — nothing is guessed from an engine-specific default, and `testCreate` keeps its `props.size() == 7` assertion. 3. Written only in the `create()` path; the `update()` / `modify()` paths are untouched, so a later per-job index override can never rewrite the creation-time record. ## Alternatives considered - **Persist only the resolved index class.** Rejected: loses the symbolic `hoodie.index.type` the operator actually configured; keeping the human-facing type string is friendlier for inspection and for future `HoodieTableConfig` consumers. - **Reuse the write-time keys verbatim inside `hoodie.properties`.** Rejected: `hoodie.properties` uses a dedicated `hoodie.table.*` namespace for table metadata; mixing job-level keys in makes it ambiguous which values are creation-time truth versus per-job overrides. - **Enforce immutability (throw on mismatch).** Deferred — behavioural change, see follow-ups. ## Tests - `testPersistsIndexTypeFromWriteConfig` — BLOOM from the write config is recorded and readable back through both the property and the new `getIndexType()` accessor. - `testIndexClassTakesPrecedenceOverType` — a custom `hoodie.index.class` wins over `hoodie.index.type`. - `testNoIndexTypeRecordedWhenNotConfigured` — nothing is written when the creation config does not specify an index. - `testDefinedTableConfigs` count updated 46 -> 47. ## Suggested follow-ups for maintainers 1. Consider whether the common-path engine default (BLOOM) should also be recorded so *default* tables become introspectable too — this trades one benign extra property for full observability, and would make a later enforcement feature complete. 2. If enforcement is wanted, the natural seam is a write-path pre-flight in `HoodieTableMetaClient`/`HoodieWriteConfig` comparing the job's `HoodieIndexConfig` against `getIndexType()`. Fixes #14425 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
