nsivabalan opened a new issue, #20606:
URL: https://github.com/apache/hudi/issues/20606
### Feature Description
**What the feature achieves:**
Make `hudi-agent-gateway` return Spark Connect query results in JSON without
losing type information, so agents and MCP clients see the same values they
would get from Spark. This is item M2.2 of the 2.0 plan on #20234.
**Why this feature is needed:**
With `GATEWAY_SQL_BACKEND=spark-connect`, `SparkConnectClient._execute_sync`
returns `[list(row) for row in df.collect()]`, and the tool layer encodes the
payload with `json.dumps(payload, default=str)`. That loses information for
several Spark types:
- **Structs:** a PySpark `Row` is a tuple, so a struct column becomes a JSON
array. The field names disappear, and so do the names inside arrays of structs.
- **Timestamps:** `datetime` values fall through to `str()`. The output has
no UTC offset, so a reader can't tell which time zone the value is in.
`TIMESTAMP` and `TIMESTAMP_NTZ` come out looking the same.
- **Decimals:** `Decimal` values become JSON strings through `str()`.
Precision survives, but the type changes, and nothing tells the client the
column was numeric.
- **Other types to check:** `DATE`, `BINARY`, maps with non-string keys, and
the `_hoodie_*` meta columns.
An agent that reads these results can misread time zones, lose track of
nested fields, or treat money columns as text.
### User Experience
**How users will use this feature:**
- Configuration changes needed: none expected.
- API changes: tool results from `query_lakehouse` (and the MCP equivalent)
on the Spark Connect backend gain faithful encodings. Structs become JSON
objects keyed by field name. Timestamps become ISO-8601 strings, with an offset
for `TIMESTAMP` and without one for `TIMESTAMP_NTZ`. Decimals use one
documented, consistent encoding.
- Usage examples: `SELECT named_struct('a', 1, 'b', 'x') AS s,
current_timestamp() AS ts, CAST(1.10 AS DECIMAL(10,2)) AS d` returns `{"s":
{"a": 1, "b": "x"}, "ts": "...+00:00", "d": ...}` instead of `[1, "x"]`, a
naive timestamp and a string.
**Suggested scope:**
1. A type-aware encoder for collected rows, applied before the existing
result-size limit and truncation, so the limit still applies to the final
payload.
2. Decide the decimal encoding (exact string vs. JSON number) and the
timestamp zone (session time zone vs. UTC), and document both.
3. Compare with the Trino backend's output. Make the two backends consistent
where that's practical, or document where they differ.
4. Unit tests covering nested structs, arrays of structs, maps, `TIMESTAMP`
vs `TIMESTAMP_NTZ`, `DATE`, high-precision `DECIMAL`, `BINARY` and NULLs. Add a
live check in `tests/integration/test_spark_connect_live.py`.
### Hudi RFC Requirements
**RFC PR link:** (if applicable)
**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? Only the JSON shape of gateway
tool results on the Spark Connect backend.
- Does this change storage format? No
- Justification: a contained change in `hudi-agent-gateway`.
---
Parent: #20234
@cshuo could you take this one up as well? It's the remaining Phase 2 item
(M2.2) from the 2.0 plan, and we'd like it to land within 2.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]