nsivabalan opened a new pull request, #20154: URL: https://github.com/apache/hudi/pull/20154
> **Stacked on #20151.** This branch includes that PR's commit because both checks plug into the > framework it introduces. Review only the last commit > (`feat(utilities): add metadata table sync and compaction lag checks ...`); the diff collapses to > ~1,580 lines once #20151 merges and this is rebased. The companion `record-index` check is in > #20153; the skill here documents all three because they are run together. ### Describe the issue this Pull Request addresses Part of #19262 under the Agentic Lakehouse umbrella #19256; plan in https://github.com/apache/hudi/issues/19262#issuecomment-5750570015. Two metadata-table failure modes are quiet and common. The metadata table can fall out of step with the data table — a writer running without it, a rollback that only half-applied — and every reader that trusts its listing then sees the wrong files. And metadata-table compaction can stall, most often because a long-pending data-table instant pins the compaction instant, after which every metadata read merges an ever-growing pile of log files. Neither raises an error. The full `HoodieMetadataTableValidator` exists for the first case but is a heavyweight Spark job; nothing exists for the second. ### Summary and Changelog Adds two checks to `HoodieTableHealthChecker`, both engine-agnostic (no Spark session), plus a companion skill. **`mdt-sync`** — is the metadata table caught up, and does its listing agree with storage? - *Sync lag:* completed data-table writes newer than the metadata table's synced instant (`getSyncedInstantTime()`). Any lag is reported as unhealthy: the metadata-table delta commit completes before the data-table instant does, so "behind" is never a read-ordering artifact. - *Listing consistency, on a sample:* for up to 20 partitions — the 20 most recently written plus the 2 oldest, since drift originates in recent writes, cleans, and rollbacks — compares the set of latest base-file and log-file names from a `FileSystemBackedTableMetadata` view against a `HoodieBackedTableMetadata` view over the same timeline snapshot. Any difference is unhealthy, naming the partition and the first few differing files. - *Findings* point at the right next step: rerun if a writer is active; look for failed metadata delta commits (`metadata timeline show incomplete` in `hudi-cli`); and for a confirmed listing drift, run the full `HoodieMetadataTableValidator` before rebuilding the metadata table, which the finding describes as disruptive. Lag alone never suggests a rebuild. **`mdt-compaction`** — is metadata-table compaction lagging, and for how long? - Completed metadata-table delta commits since its last compaction against `hoodie.metadata.compact.max.delta.commits` with 2.0x slack; echoes the last compaction instant, the lag duration in minutes, pending metadata compactions, and the earliest pending data-table instant. - The findings state the actual scheduling rule from `HoodieBackedTableMetadataWriter .compactIfNecessary`: the compaction instant is chosen just below the earliest pending data-table instant that already has a completed metadata instant, and compaction is skipped when an instant with that time already exists — so a long-pending data-table instant pins it, and that instant is the first thing to check. Never suggests deleting or rebuilding the metadata table for this condition; it recovers once compaction runs. Both follow the fail-fast contract from #20151 (`hoodie.metadata.enable`, and for compaction `hoodie.metadata.compact.max.delta.commits`, must be supplied or `--apply-all-defaults` chosen), skip with a stated reason when the table has no metadata table, and echo `observed.*` / `effective.*` values. Every metadata reader and file-system view is closed per run. **Skill** — `hudi-agent-gateway/skills/hudi-mdt-health/`, same layout as `hudi-table-health`: runs `--checks mdt-sync,mdt-compaction,record-index --output JSON`, interprets each status, and recommends without acting. Guardrails: never suggests rebuilding the metadata table unless `mdt-sync` reported a listing inconsistency confirmed by the validator; never suggests disabling the metadata table to silence a verdict. Registration is two one-line edits (`allChecks()`, `--checks` help) plus runbook sections. No code copied from elsewhere. **Tests:** 29 — `TestMetadataTableSyncCheck` and `TestMetadataCompactionLagCheck` (no Spark; skip paths, sampling, lag counting with explicit completion times, never-compacted and bootstrap-instant cases, pending data-table instant named) and `TestMetadataTableSyncCheckFunctional` (`HoodieSparkClientTestBase`: real writes, consistent listing healthy, a base file and a log file deleted from storage each reported, a data-table commit missing from the metadata table reported as lag). ### Impact Additive. No public API or storage changes; read-only. No new `hoodie.*` configs. Known limitation, documented: the listing sample is drawn from the metadata table's partition list, so a partition present on storage but wholly absent from the metadata table is not caught here; the finding's pointer to the full validator covers it. ### Risk Level none ### Documentation Update `### mdt-sync` and `### mdt-compaction` sections added to `HoodieTableHealthChecker.md`. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
