nsivabalan opened a new pull request, #20151: URL: https://github.com/apache/hudi/pull/20151
### Describe the issue this Pull Request addresses Part of #19262 (Hudi-aware agent skills and tools), under the Agentic Lakehouse umbrella #19256. See https://github.com/apache/hudi/issues/19262#issuecomment-5750570015 for the overall plan this is the first PR of. Operators managing Hudi tables have no single way to answer "is this table progressing as expected?" The individual signals exist across the timeline, but composing them means knowing exactly which APIs to call and what the writer's configuration implies. There is no tool that takes a table and a set of writer properties and says whether its table services are keeping up. This is also a prerequisite for the agent-facing work in #19262/#19263: everything the agent gateway can see today goes through Trino SQL over the catalog, and SQL cannot see timeline state, file-slice layout, or metadata-table consistency. An agent asked "is this table healthy?" currently has nothing factual to stand on. ### Summary and Changelog Adds `HoodieTableHealthChecker`, a read-only tool that reports whether a table's services are keeping up with ingestion. Intended for a scheduled run — daily is typical — with the exit code wired into alerting. This first PR establishes the framework, the four table-service checks (all timeline-only), and a Claude Code skill that interprets the report. Subsequent PRs add metadata-table checks (filesystem-vs-metadata sync, metadata compaction lag, record-level-index sizing) on the same framework, and a separate layout analyzer. **Framework** (`org.apache.hudi.utilities.health`) - `HealthCheck` — interface for one diagnosis; `getName()` makes each check selectable from the command line. - `HealthStatus` — `SKIPPED < HEALTHY < UNHEALTHY`, ordered so a table-level verdict is a `reduce(max)` over the individual outcomes. - `HealthCheckResult` — verdict, one-line summary, findings, and the effective configuration the check reasoned with. - `HealthCheckContext` — the table, the writer properties, and whether defaults may be applied; holds the fail-fast helpers so each check needs three lines of boilerplate. **Checks** - `CompactionCadenceCheck` — computes how many delta commits the configured trigger strategy allows before compaction should be scheduled, and compares against how many accumulated since the last completed compaction. Applies a 2.0x slack factor, because scheduling and execution are separate and async compaction routinely trails by a cycle. Skipped on Copy-on-Write. When unhealthy, distinguishes "nothing scheduled" from "scheduled but never completed". - `CleanerCadenceCheck` — write commits since the last completed clean against the configured trigger, with 2.0x slack and a floor of 5 commits (the trigger defaults to one commit, and without a floor ordinary async lag would alert). Distinguishes "no clean scheduled" from "clean stuck". - `SavepointArchivalBlockCheck` — archival stops at the earliest savepoint, so a forgotten savepoint pins the active timeline indefinitely. Reports savepoints sitting behind the configured archival retention, and separately flags any savepoint older than 7 days as likely forgotten. - `ArchivalCadenceCheck` — active timeline against configured retention with 1.1x slack, plus an absolute ceiling of 5000 completed instants past which timeline loading is itself a problem regardless of configuration. Points at the savepoint check rather than guessing at a cause. **Driver** — `HoodieTableHealthChecker` with `--base-path`, `--checks`, `--props` / `--hoodie-conf`, `--apply-all-defaults`, `--output TABLE|JSON`, `--verbose`. Exit codes: `0` healthy, `1` unhealthy, `2` error. **Docs** — `HoodieTableHealthChecker.md` runbook: invocation, per-check explanation with governing properties, JSON schema, a cron example, and caveats. **Skill** — `hudi-agent-gateway/skills/hudi-table-health/`, following the layout of `hudi-architect` (#19380): `SKILL.md` drives collecting the base path and writer properties, running the tool with `--output JSON`, and turning each check's status and findings into a diagnosis and a next step; `references/checks.md` documents the JSON envelope and per-check properties and remedies. The skill diagnoses only — it never runs a table service or deletes a savepoint — and never passes `--apply-all-defaults` without the user choosing it. The JSON envelope is designed for this: per check, `status`, a one-line `summary`, `findings[]` written as self-contained actionable sentences, and `effectiveConfigs{}` where `observed.*` keys are measurements, `effective.*` are derived thresholds, and bare `hoodie.*` keys are the supplied properties. No code was copied from elsewhere. #### Writer properties, and why there is no silent default Nearly every question here is relative to how the table is written: whether fifty delta commits since the last compaction is fine depends on the configured trigger; whether a thousand instants in the active timeline is fine depends on the configured retention. Hudi has a default for each, so the tool *could* always produce an answer — but for a table written with non-default settings that answer is wrong in the worst direction, reporting healthy tables as unhealthy until operators stop trusting the tool. So a check whose governing property was not supplied reports `SKIPPED` and names the property, rather than guessing. `--apply-all-defaults` is an explicit opt-in to evaluating against Hudi defaults. Every check echoes the configuration it used, so a verdict can be audited against the values that produced it. A run where everything skipped exits `0` — a skipped check is silent about health, not an assertion of it. ### Impact Purely additive. New class and new package in `hudi-utilities`; nothing existing changes behavior. - No public API changes. No `@PublicAPIClass` / `@PublicAPIMethod` touched. - No storage format changes. The tool is strictly read-only — it never schedules or runs a table service, and never writes to the table. - **No new `hoodie.*` configs.** Thresholds ship as documented constants rather than configuration, to avoid adding tuning surface for a diagnostic tool. - The four checks in this PR are timeline-only, so the tool runs as a plain `java -cp` process; no Spark session is required. ### Risk Level none Read-only tool on a new code path, reachable only by explicit invocation. No existing call site reaches any of this. ### Documentation Update `HoodieTableHealthChecker.md` is added alongside the tool. No new configs, so no config documentation changes are needed. Happy to add a website page if reviewers would like one. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
