ad1happy2go opened a new pull request, #19422:
URL: https://github.com/apache/hudi/pull/19422
> **Stacked on #19411** (Spark Thrift Server engine): this branch contains
that PR's commit plus one commit of its own. **Review only the last commit**
(`feat(gateway): schema hints`); the diff will collapse to it once #19411
merges and this branch is rebased.
### Describe the issue this Pull Request addresses
Part of the agentic-lakehouse umbrella #19256. Small local models routinely
skip `describe_table` and guess column names (`bedrooms` for `beds`, invented
table names), then fail or hallucinate -- the gateway's tool-based schema
discovery depends on exactly the multi-hop planning that small models are worst
at. Production text-to-SQL systems converge on a hybrid: compact schema context
up front plus execution-error feedback that carries the real names. This PR
adds both, auto-derived from the engine (never hand-written), behind one knob.
### Summary and Changelog
`GATEWAY_SCHEMA_HINTS = off | errors | prompt | both` (default `both`);
everything degrades gracefully and `off` restores today's pure tool-based
discovery.
**`tools/schema_cache.py` -- the snapshot**
- Single-flight TTL cache (default 300s) over a new per-connector
`fetch_schema()`: one `information_schema` query on Trino; `SHOW TABLES` +
`DESCRIBE TABLE` per table on Spark (stops at the `# Partition Information`
section). Size-capped (`GATEWAY_SCHEMA_MAX_TABLES`/`_MAX_COLUMNS`), Hudi
`_hoodie_*` meta columns filtered so hints never steer the model toward them.
- Strictly fail-open: a refresh failure keeps the previous snapshot and
never breaks the query path; failures are debounced by the same TTL so an
unreachable engine is not hammered by probes.
**`errors` mode -- execution-guided self-correction**
- A column/table-not-found error now returns an actionable hint built from
the cached schema: fuzzy match on the offending identifier (leaf of qualified
names) plus the real names to copy from: `Did you mean `beds`? Columns in
`listings`: listing_id, address, price, beds, ...`. Handles Spark
(`UNRESOLVED_COLUMN`, `TABLE_OR_VIEW_NOT_FOUND`) and Trino (`COLUMN_NOT_FOUND`,
`TABLE_NOT_FOUND`) message shapes; non-name errors keep the existing generic
hint; bounded at 700 chars.
**`prompt` mode -- schema in the system prompt**
- A compact snapshot (`listings(listing_id string, address string, ...)`) is
rendered into the system prompt per turn via a callable `create_react_agent`
prompt, so the model writes SQL against real names without a discovery hop; the
message-trimming hook is preserved (`llm_input_messages`-aware). The snapshot
is warmed at gateway startup and follows the TTL thereafter, so schema changes
propagate without agent rebuilds.
**Plumbing**
- `LakehouseConnector.fetch_schema()` added with a `return {}` default --
third-party connectors are unaffected and simply have hints disabled until they
implement it. `build_schema_cache()` + `schema_cache` parameter threaded
through `build_registry`/`AgentCache`; helm chart gains `schemaHints`; README
section + config rows.
**Tests** -- 28 new offline tests: cache
TTL/fail-open/caps/meta-filter/peek-background-refresh, did-you-mean hints for
both engines' error shapes (incl. qualified identifiers, no-close-match
fallback, bounded output, non-name errors silent), mode gating, end-to-end
through both engines' tool handlers, prompt rendering and trimming interplay,
and both connector fetchers.
No code copied from other projects.
### Impact
New optional configs (`GATEWAY_SCHEMA_HINTS`,
`GATEWAY_SCHEMA_CACHE_TTL_SECONDS`, `GATEWAY_SCHEMA_MAX_TABLES/_MAX_COLUMNS`)
documented in the module README; new helm value `schemaHints`. Default `both`
changes model-facing behavior only (better hints, a schema block in the system
prompt); the tool API and HTTP surfaces are unchanged. Costs one schema fetch
per TTL window plus the prompt tokens for the capped snapshot;
`GATEWAY_SCHEMA_HINTS=errors` keeps the prompt lean, `off` restores current
behavior exactly.
### Risk Level
low -- additive and fail-open by construction (schema fetch failures degrade
hints, never queries). 172 offline tests pass; ruff and mypy clean; helm lint
clean. Verified live against Apache Kyuubi serving Hudi 1.3 tables with
llama3.1: the previously failing question ("active 3-bedroom homes under $800k
in Downtown") went from hallucinated column names and a leaked tool-call JSON
to correct names (`status`/`beds`/`neighborhood`) on the first tool call, and a
forced `bedrooms` query returned the did-you-mean hint carrying the real column
list.
### Documentation Update
Module README updated in this PR ("Schema hints" section + config table
rows). Website docs tracked with the umbrella #19256.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
