nsivabalan opened a new pull request, #20197: URL: https://github.com/apache/hudi/pull/20197
### Describe the issue this Pull Request addresses Part of #19263 (specialized sub-agents for optimization and analysis), under the Agentic Lakehouse umbrella #19256. Builds on the event-log analyzer in #20196. Hudi exposes a lot of tuning surface — 939 distinct `hoodie.*` keys, of which roughly 237 are performance or sizing levers. Knowing which of them matters for a particular table, at its current size, with its current access pattern, is expertise most users reasonably do not have. Worse, about 93 of those levers only take effect under some other config's value, so a change can be made in good faith and do nothing at all. This adds the knowledge layer for an optimization advisor: what to look for in a *working* pipeline, what it costs, what to change, and — equally important — when to leave it alone. ### Summary and Changelog A skill plus two catalogs under `hudi-agent-gateway/skills/hudi-optimizer/`. No code; this is knowledge, structured so an agent can apply it against evidence. **Three required inputs.** Writer properties, a Spark event log, and read access to the table. The central discipline is that **an observation is only trustworthy when at least two of the three agree**. An event log alone says "the write stage is slow". With the table it says "and the median base file is 8 MB". With the properties it says "and your small-file limit sits below your batch size, so small-file handling never engages" — which is the first version a user can act on. Every catalog row records which inputs corroborate it, and the skill states its confidence accordingly: with one input it describes, with two it recommends, with three it recommends with verification. **`references/observation-catalog.md`** — 30 rows of "this works, but could be better". Each row names the measurable signals as concrete fields in the analyzer, health-checker or layout-analyzer JSON; which inputs corroborate it; the change with exact config keys and direction; a risk tier; expected payoff; and how to verify it worked. The backbone is derived expectations — comparing what the configuration *implies* against what was measured. "Your target file size is 120 MB and your median base file is 8 MB" needs no baseline and no history. Several rows are counter-intuitive and say so: reducing index parallelism, lowering Spark memory fractions, and one row whose correct outcome is to change nothing at all, because a cleaner that appears idle is usually correct. **`references/capability-catalog.md`** — 7 rows of "adopt this feature", which is where an agent beats documentation. The docs can say a record-level index gives faster lookups; they cannot say whether it suits *this* table at *this* size with *this* update pattern. Each row carries preconditions, measured signals both for and against, estimated payoff, adoption cost, and verification. The record-level index row is the worked example. Its premise is mechanical: `HoodieSimpleIndex.fetchRecordLocations` parallelises over base files, so simple-index tagging cost scales with table size rather than batch size — which is visible in an event log as a tagging stage whose task count tracks the file count instead of the record count. It carries eight signals for and seven against, including the most important counter-signal: with time-ordered keys and updates concentrated in recent partitions, bloom key-range pruning may already be near-optimal and the index would add metadata-table overhead for no gain. The agent must be able to say "don't". It also sizes the index before recommending it, using the same `estimateFileGroupCount` the writer uses at initialisation, so the recommendation carries its own cost estimate. And it notes that `hoodie.index.type` defaults to `SIMPLE` on Spark via a per-engine builder override — so the user is on that index by default rather than by choice, and the wording reflects that. **Risk tiers, and a refusal.** `safe` (reversible config), `operational` (affects table services, reversible but needs a cycle to observe), `durable` (changes what is on storage). **The agent does not recommend durable changes as optimizations** — switching index type, table type, partitioning or key generator belongs to a design conversation with an architecture decision record, not to a tuning pass. The refusal is structural rather than advisory: the observation catalog contains no durable rows by construction, the three durable capability rows carry an explicit refusal block where the recommendation would otherwise be, and the skill has a worked example of saying the refusal out loud so it is not quietly skipped. **`SKILL.md`** — collects the inputs, invokes `HoodieTableHealthChecker` and `HoodieTableLayoutAnalyzer` rather than asking the user to run them separately, and reasons from their underlying measurements rather than their summary verdicts. It carries the event-log analyzer's caveats into its own reasoning: a phase share names where work was *forced*, not where it logically belongs, and with streaming writes to the metadata table enabled a large share of the DAG is unattributable, so the skill says so rather than reasoning over a partial picture. It recommends only, and never applies a change. Recipes come from Spark deployments, since that is where the evidence is. The core patterns — index choice, file sizing, table-service cadence, metadata-table cost — apply to Flink as well, and Flink coverage will deepen over time. Every `hoodie.*` key cited was verified against the source tree; several keys that exist only in folklore were dropped rather than guessed at. The JSON field paths were verified the same way, against the actual tool sources. ### Impact Purely additive documentation. No Java, no configs, no behaviour change. Nothing imports or executes any of it. The skill is read-only by construction: it recommends, and never mutates a table. ### Risk Level none ### Documentation Update Self-contained. Thresholds in the catalogs are starting points derived from a small set of measured runs; they are documented as such rather than presented as tuned constants, and are the most likely thing to want revision once more workloads are measured. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
