nsivabalan commented on PR #19205: URL: https://github.com/apache/hudi/pull/19205#issuecomment-5221101457
**Correcting one thing I told you earlier in this thread**, and rewriting the PR description to be authoritative — the design settled over several rounds and the text had drifted behind it. ### Downgrade of a selective mode: now allowed, not rejected Earlier today I said a v10 → v9 downgrade would **throw** for the three selective modes, on the reasoning that there was no honest boolean to write back. That was over-engineered, and @nsivabalan pointed out why: **every selective mode already persists `hoodie.populate.meta.fields=false`**, because the boolean is always derived from the mode rather than taken from the caller (`TableBuilder`, `HoodieWriteConfig.Builder`, hudi-cli). So there is nothing to decide at downgrade time. Drop the mode, restate the `false` that is already on disk, done — no special case: | Table mode | v9 sees | Round trip | |---|---|---| | `ALL` | `populate=true` | lossless | | `NONE` | `populate=false` | lossless | | any selective mode | `populate=false` | mode is lost | That is the correct reading for v9: it has no code that reads `_hoodie_commit_time` selectively, so presenting the table as having no meta columns is the only honest thing it can say. Files keep their populated columns; only how the table advertises itself changes. The direction is safe — the table **under-claims**. Over-claiming is the bug, and that is exactly what the boolean write-back exists to prevent (@voonhous's original catch on the `true` default). What remains lossy is that a re-upgrade cannot restore the mode: it derives from the boolean, which now reads `false`, so the table comes back as `NONE`, and hudi-cli cannot widen it back. **Recreating the table is the only way to recover it.** The downgrade warns, and `reUpgradeAfterDowngradingASelectiveModeResolvesToNone` pins the round trip rather than leaving it implicit. This also resolves @voonhous's one-way concern on the downgrade handler more cleanly than the throw did: the state is reachable, documented, warned about, and tested — rather than being made unreachable by a hard failure the operator cannot work around. ### Also corrected: the incremental-query error message It said "recreate it with `hoodie.populate.meta.fields=true` or `hoodie.meta.fields.mode=COMMIT_TIME_ONLY`". True, but it reads as though the mode could be flipped on the existing table. It cannot — a `NONE` table reaching `COMMIT_TIME_ONLY` is a *widening*, which hudi-cli refuses too. The message now says that explicitly and points at recreation. ### PR description rewritten Given how many times this has turned over, the description is now the single source of truth, with four numbered sections stating the behavior precisely: 1. **Resolving a single config** — the precedence rule plus the full truth table for both properties, including that `mode` + `populate=true` is *not* an error. 2. **The mode is a table property** — the inherit-or-reject rule the write client enforces for every engine. 3. **Which mode changes are legal** — the narrowing lattice, why widening is refused even with `--force`, and the fact that `COMMIT_TIME_ONLY` / `FILE_NAME_ONLY` are mutually wider so neither can become the other. 4. **Upgrade and downgrade** — the table above, plus why upgrade keeps the boolean where `EightToNineUpgradeHandler` deletes what it replaces. It also now lists the three follow-ups in landing order: **#19206** (hudi-cli, already rebased onto this head and updated to enforce section 3), **#19378** (call-site migration), and the pre-existing CoW incremental `count()` bug, which needs a JIRA and is out of scope here. Tests: `TestTenToNineDowngradeHandler` 14/14, upgrade/downgrade 84/84, `TestMetaFieldsModeE2E` 27/27, `TestBaseHoodieWriteClient` 110/110. checkstyle and scalastyle clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
