hudi-agent commented on code in PR #19799: URL: https://github.com/apache/hudi/pull/19799#discussion_r3946494543
########## website/docs/cli.md: ########## @@ -785,6 +762,442 @@ table change-table-type COW ╚════════════════════════════════════════════════╧══════════════════════════════════════╧══════════════════════════════════════╝ ``` +### Changing the Meta Fields Mode + +`hoodie.meta.fields.mode` decides which of Hudi's meta columns are physically written into base files. It is a +storage-layout decision baked in at write time, so a write can never change it on an existing table. +`table set-meta-fields-mode` is the sanctioned way to change it. + +```java +hudi:trips->table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY +``` + +`--target-mode` accepts `ALL`, `NONE`, `COMMIT_TIME_ONLY`, `FILE_NAME_ONLY` or `COMMIT_TIME_AND_FILE_NAME`. The value +is resolved case-insensitively and trimmed, so `commit_time_only` is accepted too. Setting the mode the table is +already in is a no-op and reports as much. + +On a table that already has commits, two guards apply, because this command changes the table property without +rewriting a single existing file: + +- **Widening is refused outright**, and `--force` does not override it. Widening means the target mode populates a + meta column the current mode does not. Since earlier files are not rewritten, the table would advertise a column + that is null for every row written so far, and incremental queries and file-name lookups silently skip exactly + those rows. To widen, recreate the table. The comparison the CLI makes is `MetaFieldsMode#isWiderThan`, the same + helper the write path uses. Note the write path itself is stricter rather than equivalent: it rejects any writer + mode that differs from the table's, and consults `isWiderThan` only to choose the error wording. +- **Narrowing needs `--force`** (default `false`). It leaves mixed-mode files: old commits keep the old layout, new + commits use the new one, and incremental and file-pruning semantics differ between the two sets. Passing `--force` + logs a warning recording the transition and the commit count. + +Neither guard applies to a table with no commits, where the mode can be set freely. + +### Inspecting the Timeline + +`commits show` lists completed commits, and only commit, deltacommit, replacecommit and clustering actions. The +timeline commands show every instant regardless of action and state, which is what you want when diagnosing a stuck +table: a compaction sitting in `REQUESTED` never appears in `commits show`, and neither does a rollback, in any +state. + +```java +hudi:trips->timeline show active --limit 10 +hudi:trips->timeline show incomplete +``` + +Both print `Instant`, `Action`, `State`, and the `Requested` / `Inflight` / `Completed` file modification times. +`timeline show incomplete` restricts the listing to instants that are not yet completed. + +| Option | Default | Applies to | Description | +| --- | --- | --- | --- | +| `--limit` | `10` | both | Number of rows to display. | +| `--sortBy` | unset | both | Field to sort by. | +| `--desc` | `false` | both | Reverse the ordering. | +| `--headeronly` | `false` | both | Print the header only. | +| `--show-rollback-info` | `false` | both | Annotate the Action column in both directions: `Rolls back ...` on a rollback instant, and `Rolled back by ...` on any instant a rollback targeted. | +| `--show-time-seconds` | `false` | both | Include seconds in the instant file modification times. | +| `--with-metadata-table` | `false` | `timeline show active` only | Show the metadata table timeline alongside the data table, adding `MT Action`, `MT State` and the three matching MT time columns. | + +The metadata table has its own timeline, and the two can disagree when a metadata commit fails. To read it directly: + +```java +hudi:trips->metadata timeline show active --limit 10 +hudi:trips->metadata timeline show incomplete +``` + +These accept `--limit`, `--sortBy`, `--desc`, `--headeronly` and `--show-time-seconds`. They have no +`--show-rollback-info` and no `--with-metadata-table`, since they are already scoped to the metadata table. + +### Diffing a File or Partition + +`diff file` and `diff partition` replay the timeline and show every commit that touched a given file group or +partition, which is the quickest way to answer "what has been writing to this file". Both report the standard commit +columns plus the write statistics for the matching entries only. + +```java +hudi:trips->diff file --fileId 5f8a1e0b-1b4b-4a3f-9b1a-2c7d6e5f4a3b-0 --limit 10 +hudi:trips->diff partition --partitionPath 2026/08/26 --includeArchivedTimeline true +``` + +`diff file` takes `--fileId` and `diff partition` takes `--partitionPath` as a partition path relative to the table +base path. `diff partition` is only meaningful on a partitioned table. Both then share these options: + +| Option | Default | Description | +| --- | --- | --- | +| `--includeArchivedTimeline` | `false` | Also scan archived instants, not just the active timeline. | +| `--startTs` | unset, meaning now minus 10 days | Start of the instant range. Only applied when `--includeArchivedTimeline` is `true`. | +| `--endTs` | unset, meaning now minus 1 day | End of the instant range. Only applied when `--includeArchivedTimeline` is `true`. | +| `--limit` | `-1`, meaning no limit | Number of rows to display. | +| `--sortBy` | unset | Field to sort by. | +| `--desc` | `false` | Reverse the ordering. | +| `--headeronly` | `false` | Print the header only. | + +Note the interaction between the three range options, which is easy to get wrong. `--startTs` and `--endTs` are used +to select archived instants only. With the default `--includeArchivedTimeline false` the whole active timeline is +scanned and both bounds are ignored, so passing a narrow range does not restrict the output. Set +`--includeArchivedTimeline true` for the bounds to take effect, and note that the archived range is then merged with +the full active timeline rather than replacing it. + +### Repairing a Table + +`repair show empty commit metadata` scans completed instants on the active timeline and reports the ones whose +metadata file is empty, which is what a commit interrupted between file creation and metadata write leaves behind. + +```java +hudi:trips->repair show empty commit metadata +``` + +Note that this command writes its findings to the CLI log at `WARN` level rather than returning a table, so with the +default logging configuration you will see the `Empty Commit: ...` lines in the console log rather than in a rendered +result. It only reports; it does not modify the timeline. + +`rename partition` rewrites the data under one partition value into another, as a Spark job, and deletes the old +partition on success. It only works on a table with a single partition field: the rewrite looks the partition field +up by name in the dataframe schema, and on a multi-field table that name is the comma-joined list, so the job fails +before writing anything. + +```java +hudi:trips->set --conf SPARK_HOME=<SPARK_HOME> +hudi:trips->rename partition --oldPartition 2026/08/26 --newPartition 2026-08-26 --sparkMaster local[2] +``` + +`repair deprecated partition` is the special case of that rename for tables written before Hudi settled on its +placeholder for the null partition value: it rewrites data from the deprecated `default` partition into +`__HIVE_DEFAULT_PARTITION__`. + +```java +hudi:trips->repair deprecated partition --sparkMaster local[2] +``` + +Both take `--sparkProperties` (a Spark properties file path, empty by default), `--sparkMaster` (unset by default, +which means `yarn` -- see the note under the Command reference legend) and +`--sparkMemory` (`4G` by default). Both read the old partition, rewrite those records under the new partition value, +and then issue a `delete_partition` write against the old one, so the change goes through the timeline rather than +behind it. Both are a no-op when the old partition holds no records. + +They differ in one respect worth knowing: `rename partition` additionally removes the old partition directory from +storage after the delete write, logging a warning if that removal fails, whereas `repair deprecated partition` leaves +the emptied `default` directory in place. Either way these rewrite data, so take a savepoint first if the table +matters. + +### Auditing Storage Locks + +When a table uses a storage-based lock provider, the lock provider can record every lock transition to a set of JSONL +files, so that a suspected concurrency violation can be reconstructed after the fact. The audit is off by default and +is controlled by a config file next to the locks themselves, at +`<basePath>/.hoodie/.locks/audit_enabled.json`; the audit records land in `<basePath>/.hoodie/.locks/audit/`. + +```java +hudi:trips->locks audit enable +Lock audit enabled successfully. +Audit config written to: /user/hive/warehouse/table1/.hoodie/.locks/audit_enabled.json +Audit files will be stored at: /user/hive/warehouse/table1/.hoodie/.locks/audit +``` + +`locks audit status` reports whether auditing is on, and where both the config and the records live. A table that has +never had auditing enabled reports `DISABLED` with the config file marked `(not found)`. + +```java +hudi:trips->locks audit status +Lock Audit Status: ENABLED +Table: /user/hive/warehouse/table1 +Config file: /user/hive/warehouse/table1/.hoodie/.locks/audit_enabled.json +Audit files location: /user/hive/warehouse/table1/.hoodie/.locks/audit +``` + +`locks audit validate` is the reason to collect the records. It parses every `.jsonl` file in the audit folder into +transaction windows and checks them against each other. Overlapping windows are reported as errors, since two writers +holding the lock at once is exactly the violation the lock provider exists to prevent. A transaction that never +released its lock is reported as a warning, which usually means a driver OOM or a non-graceful shutdown rather than a +correctness problem. The verdict is `PASSED`, `WARNING` when only warnings were found, or `FAILED` when any error was. + +```java +hudi:trips->locks audit validate +Validation Result: PASSED +Audit Files: 12 total, 12 parsed successfully, 0 failed to parse +Transactions Validated: 12 +Issues Found: 0 +Details: All audit lock transactions validated successfully +``` + +With no audit folder or no audit files the command reports `PASSED` with zero transactions validated, so a `PASSED` +verdict on its own does not prove that auditing was ever on. Check `locks audit status` first. + +`locks audit cleanup` prunes old records. `--ageDays` defaults to `7` and `--dryRun` defaults to `false`, so run it +with `--dryRun true` first to see what it would remove. + +```java +hudi:trips->locks audit cleanup --dryRun true --ageDays 30 +``` + +`locks audit disable` turns auditing off. It keeps the existing records by default; pass `--keepAuditFiles false` to +delete them at the same time, which internally runs the same cleanup with no age threshold. + +```java +hudi:trips->locks audit disable --keepAuditFiles true +``` + +All five commands require a table to be connected, and report `No Hudi table loaded. Please connect to a table first.` +otherwise. + +## Command reference + +Every command `hudi-cli` exposes, grouped by area, with its options and their defaults. An option marked +`(required)` has no default and must be supplied; a value in backticks after an option is its default; an option +listed with neither is optional and unset by default, which for most commands means the empty string reaches the +command. The sections above cover the commonly used ones in more depth. + +One default is worth stating up front because it is not the empty string it looks like: an unset or blank +`--sparkMaster` resolves to `yarn`. The CLI treats a blank value as absent and falls back to its own default, so +every `--sparkMaster` shown below without a value runs against YARN unless you pass something. That differs from +the compaction and marker commands, which show `local` explicitly. Review Comment: 🤖 Small contradiction with the reference below: `marker delete` has `defaultValue = ""` for `--sparkMaster` (`MarkersCommand.java:41`), so it falls back to `yarn` like the others — and the entry at line 1101 correctly shows it without a value. Could this just say "the compaction commands and `trigger archival`, which show `local` explicitly"? Same sentence in `versioned_docs/version-1.2.0/cli.md:942`. <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
