nsivabalan opened a new pull request, #19380:
URL: https://github.com/apache/hudi/pull/19380
### Describe the issue this Pull Request addresses
Part of #19263 (specialized sub-agents), under the #19256 umbrella.
**Milestone 1, raised
for community feedback** — it targets pipeline modelling, with multi-writer,
table-services
operationalization, and Flink as tracked extensions. See *Scope*.
**The problem: onboarding onto Hudi is harder than it should be.**
Hudi is a good deal more than a table format — it ships ingestion utilities,
self-managing
table services, indexing, catalog sync, and a CLI, across three engines
(Spark, Flink, Java).
Each of those engines offers more than one way to ingest, with different
deployment models
behind them: HoodieStreamer in continuous or run-once mode, Spark
DataSource, Spark SQL,
Structured Streaming, Flink SQL and the Flink datastream API. Table services
can run inline,
async in the writing job, or as separate jobs. That breadth is a genuine
strength — it is what
lets one system serve a 100GB dimension table and a petabyte fact table,
streaming and batch —
but the feedback we consistently get from the community is that getting
*started* feels
complex, and that the number of configs is overwhelming when you are still
working out which
of those paths you are even on.
So the distance between *"I have a Kafka topic of trip updates"* and *"here
is a correct table
configuration"* gets bridged today by one of three things:
1. **Assembling the answer yourself from scattered sources.** Combing
through Slack history
and GitHub issues for someone with a similar-sounding workload, reading
through dozens of
blog posts of varying vintage, and cross-referencing a config reference
that is organized
by *config* rather than by *decision*. Nothing in that reference tells
you that
`hoodie.compaction.target.io` is denominated in megabytes, or that
partitioning cannot be
changed after table creation. Worse, a Slack thread or blog post that was
correct for
0.14 may quietly not be for 1.2, and nothing signals which.
2. **Copying someone else's config bundle.** Fast, and produces a table
sized for someone
else's workload. This is the most common path and the most quietly
damaging one.
3. **Asking an experienced Hudi practitioner.** Reliable, and doesn't scale.
A large share
of our community support traffic is exactly this request.
**This PR aims at the onboarding experience, by automating option 3 so
nobody has to fall
back on 1 or 2.** Instead of combing through Slack, GitHub issues, and a
hundred blog posts
to work out how to design a pipeline with Hudi, every user gets their own
Hudi practitioner
to talk to — and walks out with a config bundle, an ADR, and a runnable
submit command in a
matter of minutes.
The platform breadth above is exactly what an interview is good at
absorbing. A user does not
need to know that HoodieStreamer continuous mode gets async compaction
in-process while Spark
DataSource does not; they need to answer "where does the data come from?"
and "how is the
pipeline expressed?" and have the engine, writer, deployment model, and
service posture fall
out of those answers. The optionality stays — it just stops being the user's
problem to
navigate cold.
The valuable thing a solutions architect brings isn't knowing config names —
it's knowing
*which questions to ask*. They don't open the config reference. They ask:
how often does
data land? Do records change after insert? What do consumers filter on? Can
that column's
value change for a record? Then they derive the Hudi decisions from those
answers, and tell
you which decisions you can't take back.
That interview is a formalizable artifact: the questions are finite, their
gating is
deterministic, the derivations are table lookups, and the traps are
enumerable. This skill
encodes it, and it stays in the repo next to the configs it reasons about —
so unlike a blog
post, it can be kept honest against the code (see the key validator below).
It also directly answers a question raised on #19264. @vaibhavk1992 asked
whether the
agent would have Hudi-specific knowledge baked in rather than behaving like
a generic
text-to-SQL layer, and the reply was that we plan to land skills and tools
that make it
really know Hudi internals. This is one of those, aimed at design time.
### Summary and Changelog
Adds a **design-time architect agent**: an interactive skill that interviews
a user about
their workload in plain language and produces a Hudi table design — an
Architecture
Decision Record, a grouped `hoodie.*` config bundle, and a runnable submit
command. It
asks *workload* questions ("how often does data land?", "what do consumers
filter on?"),
never *Hudi* questions ("which index type?"), and derives the Hudi decisions
from the
answers.
**What a user gets.** A ~15-minute conversation instead of days of research
across Slack,
GitHub issues, blog posts, and the config reference — ending in three
artifacts they can put
straight through normal design review: an ADR with tradeoff tables and
measurable revisit
conditions, a config bundle where every property encodes a decision, and a
submit command
whose flags are split into load-bearing (derived from the design) versus
environment
placeholders (paths, memory, Scala/Spark versions the flow deliberately
never asks about).
It is also, deliberately, an *interview* rather than a questionnaire — it
explains the
tradeoff before each durable decision, recommends one with reasoning, and
lets the user
override. Someone who has never designed a Hudi table gets a working one;
someone who has
gets a second opinion and a written record of why.
**Why this is a separate agent rather than part of the gateway loop.** The
landed loop is
grounded and read-only — it answers from tool results over tables that
exist, and its
system prompt tells it to never invent table or column names. Table design
is the opposite
shape: generative, interrogative, and about a table that does not exist yet,
so there is
nothing to query and nothing to ground against. That is the narrower-toolset
argument in
#19263 applied to design time. #19263 names optimization, analysis, and
visualization as
its examples; this is a fourth specialist alongside them.
Worth noting one asymmetry, since it came up on #19264: the observation that
CoW vs MoR
shouldn't matter because queries have the same semantics is true for
**reads**, which is
all the current loop does. At **design** time it's one of the most
consequential choices
on the table, with a whole tradeoff surface behind it and a table rewrite as
the cost of
changing your mind. Design-time and read-time agents need different
knowledge.
**Three properties that distinguish this from asking a foundation model
directly:**
- **It asks workload questions, never Hudi questions.** A user who has never
heard of an
index can still answer "can this record move to a different date after
insert?" — and
that answer is what selects index scope.
- **It refuses to guess at config.** Every `hoodie.*` key it emits comes
from the checked-in
reference tables, and `validate_config_keys.py` verifies all 83 of them
against
`ConfigProperty` definitions in the source tree. A model configuring Hudi
from memory
produces plausible properties that don't exist — and unlike a wrong query,
a wrong table
property is durable.
- **It surfaces irreversibility as a first-class output.** Table type,
partitioning, record
key, bucket count, and record-index file-group count are one-way. Every
ADR carries a
durability table naming each applicable one and what unlocking costs.
**Changelog — files added, all under
`hudi-agent-gateway/skills/hudi-architect/`:**
| File | What it holds |
|---|---|
| `SKILL.md` | Flow control, four-tier gating, question-delivery contract,
guardrails |
| `references/question-flow.md` | Round-by-round questions with conditional
gating per tier |
| `references/decision-tables.md` | Derivation rules per decision domain
(engine, writer, table type, index, partitioning, retention, services,
meta-fields, key) |
| `references/warnings.md` | 14 named traps, each with trigger condition,
message template, and firing point |
| `references/config-templates.md` | `hoodie.*` templates per decision, plus
sample bundles for three workload archetypes |
| `references/adr-template.md` | ADR output structure |
| `README.md` | What it is, how to invoke, what to stress-test during review
|
| `RUNBOOK.md` | Operator guide for data engineering / ETL teams: setup,
pre-session workload checklist, how to read the output, limitations |
| `validate_config_keys.py` + allowlist | Validates every `hoodie.*` key
referenced by the skill against the source tree; exit 1 on unknown keys;
allowlist entries each carry a documented reason |
**Tiering.** The interview scales to intent, gated by one opening question:
exploring
(concept explanation), prototyping (minimum questions for a genuinely
runnable table),
productionizing (full workload interview), production-at-scale (adds index
sizing math and
a derived-facts checkpoint). A prototyping user gets disclosed defaults they
consent to
rather than a bundle full of placeholders.
**Playtesting.** Four workload archetypes were driven end-to-end while
writing this, and
the fixes are folded in. Three were config defects that would otherwise have
shipped:
- `hoodie.compaction.target.io` is denominated in **MB, not bytes** (default
`512000` =
500GB). Earlier drafts emitted byte counts, off by ~10^6. It is also an IO
*ceiling* that
throttles compaction, not a sizing target.
- `hoodie.metadata.record.index.*` are **deprecated aliases for the global
properties**.
Partitioned RLI uses `hoodie.metadata.record.level.index.*` and is **per
partition**.
Templates were setting global knobs under a partitioned heading.
- An initialized record index is **not resizable**. Adding an RLI later is
free; the
file-group count is durable at initialization, like bucket count.
That last class of bug is the reason `validate_config_keys.py` exists:
correctness here has
to be mechanically checkable, not reviewed by eye.
### Scope
**Milestone 1 targets pipeline modelling** — taking a workload description
and deriving the
durable shape of the table and the ingestion pipeline around it: table type,
partitioning,
record key and ordering, index, writer selection, retention, and the
table-service posture
that follows from those. That's the slice where a wrong answer is most
expensive, because
most of it is fixed at table creation.
**This patch is raised to get community feedback on the direction and the
interview itself
while the extensions are being built.** Deliberately out of M1 scope, each
already sketched
in the flow so it can be filled in rather than retrofitted:
| Extension | Where M1 stops today |
|---|---|
| **Multi-writer** | The skill asks whether anything else writes the table
and, if so, emits the OCC skeleton plus a blocking pre-launch item in the ADR.
It does **not** pick a lock provider or model conflict behavior. Provider
choice, OCC vs NBCC, and conflict resolution are the extension. |
| **Table-services operationalization** | Cleaner, archival, and compaction
posture are derived from cadence, but only as *design-time* settings. Tuning
against observed behavior — compaction backlog, clustering, layout optimization
— is deliberately pushed to an operations agent and named as such in the
guardrails. This is the natural join with the optimization sub-agent in #19263.
|
| **Flink** | Asked as an engine option and routed, but the Flink derivation
path (bucket-index dominance, different reader configs, checkpoint interaction)
is not yet built out. Spark is what has been driven end-to-end. |
| **Catalog / metastore sync** | `hoodie.datasource.hive_sync.*` is out of
scope; the ADR flags it as a pre-launch item when a catalog-based engine reads
the table. |
| **Non-Kafka source configs** | Kafka gets full source-class and
schema-provider derivation. DFS/JDBC/Pulsar/Kinesis are routed but leave source
properties to the user. |
Also deferred: benchmarking, record-level TTL, z-order/layout guidance,
multi-table
transactions, `CONSISTENT_HASHING` bucket sizing, and versions other than
1.2.0. All of
these are stated as limitations in `RUNBOOK.md` §8 rather than being
silently missing — the
skill defers honestly instead of guessing, and anything it declines to
decide lands in the
ADR's open-questions section as a pre-launch action item.
**One structural gap worth naming rather than leaving to be discovered:
there is no rule
engine.** M1 is knowledge and flow only, so config correctness rests on the
checked-in
reference tables plus the key validator, not on model inference. A
hallucinated `hoodie.*`
becomes a durable table property, not a re-runnable wrong answer, so the
derivation logic —
workload profile schema, decision functions, property generator, fixture
tests — is owed as
a follow-up in this module's idiom before this drives anything unattended.
Landing the
knowledge first makes that work reviewable against something concrete, and
the feedback from
this patch is what should shape it.
### What feedback would be most useful
The interview is the part that benefits most from review by people who have
designed real
Hudi tables, since its quality is entirely in which questions fire and in
what order:
- **Run a table you already operate through the flow** and compare the
design it produces
against what you actually built. Divergence in either direction is the
most valuable
signal here — either the skill is wrong, or it found something.
- **Questions that shouldn't be asked**, or that are asked in Hudi terms
where a workload
phrasing exists. The design rule is that no question should be asked whose
answer wouldn't
change the recommendation.
- **Traps that are missing** from `references/warnings.md`, or that fire at
the wrong moment.
A warning whose only firing point is a checkpoint a tier skips is a
warning that never
fires — one round of that class of bug is already fixed in this patch, and
there may be more.
- **Derivations you disagree with** in `references/decision-tables.md`,
especially around
index selection and the retention safe-maximums per commit cadence.
- **Whether the extension ordering above matches what users actually hit
first** — multi-writer
and table-services operationalization are my read of the priority; Flink
ahead of both is a
reasonable argument.
### Impact
**No impact on existing behavior.** Purely additive: 10 new files, all
documentation and one
standalone validation script. No existing file is modified, no Hudi API,
config, or storage
format is touched, and nothing is wired into the gateway's runtime — the
skill is loaded by
an external agent host (Claude Code, or any MCP-speaking client), not by
`hudi_agent_gateway`. `setuptools` package discovery is `where = ["src"]`,
so `skills/` is
not swept into the wheel.
**User-facing gain:** a supported path from workload requirements to a
reviewed Hudi table
design, without needing to read the config reference first.
### Risk Level
**low**
Nothing executes in any Hudi code path. The only executable file is
`validate_config_keys.py`, a developer-run script that reads markdown and
Java/Scala
sources and exits non-zero on unknown config keys; it never writes.
The residual risk is **advice quality** — a user could act on a design that
is wrong for
their workload. Mitigations:
- Every `hoodie.*` key is machine-verified against `ConfigProperty`
definitions (83 keys,
currently clean). Run: `python3
hudi-agent-gateway/skills/hudi-architect/validate_config_keys.py`
- The skill is explicitly barred from inventing config combinations not
present in
`references/`; unresolved decisions must be surfaced as open questions
instead.
- It cannot deploy, modify tables, or apply configuration — design-time
only, by guardrail.
- Every output separates confirmed facts from inferred facts from
assumptions, so a reviewer
can tell what was checked from what was guessed.
- One-way decisions are flagged inline during the conversation and again in
the ADR's
durability table.
- Four workload archetypes driven end-to-end; the resulting corrections are
in this PR.
Known gaps in that verification: the exploration tier and the Flink path
have not been
driven end-to-end yet, and the multi-writer question path
(production-at-scale only) hasn't
been exercised live.
### Documentation Update
None required for this PR — no new configs, no changed defaults, no public
API or storage
change. The skill is self-documenting (`README.md` for reviewers,
`RUNBOOK.md` for
operators).
Website documentation would be worth adding once the direction is agreed and
the follow-up
rule engine lands; happy to take that as a separate contribution if
reviewers want it
surfaced on hudi.apache.org rather than discovered in-repo.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
