nsivabalan opened a new issue, #20606:
URL: https://github.com/apache/hudi/issues/20606

   ### Feature Description
   
   **What the feature achieves:**
   Make `hudi-agent-gateway` return Spark Connect query results in JSON without 
losing type information, so agents and MCP clients see the same values they 
would get from Spark. This is item M2.2 of the 2.0 plan on #20234.
   
   **Why this feature is needed:**
   With `GATEWAY_SQL_BACKEND=spark-connect`, `SparkConnectClient._execute_sync` 
returns `[list(row) for row in df.collect()]`, and the tool layer encodes the 
payload with `json.dumps(payload, default=str)`. That loses information for 
several Spark types:
   
   - **Structs:** a PySpark `Row` is a tuple, so a struct column becomes a JSON 
array. The field names disappear, and so do the names inside arrays of structs.
   - **Timestamps:** `datetime` values fall through to `str()`. The output has 
no UTC offset, so a reader can't tell which time zone the value is in. 
`TIMESTAMP` and `TIMESTAMP_NTZ` come out looking the same.
   - **Decimals:** `Decimal` values become JSON strings through `str()`. 
Precision survives, but the type changes, and nothing tells the client the 
column was numeric.
   - **Other types to check:** `DATE`, `BINARY`, maps with non-string keys, and 
the `_hoodie_*` meta columns.
   
   An agent that reads these results can misread time zones, lose track of 
nested fields, or treat money columns as text.
   
   ### User Experience
   
   **How users will use this feature:**
   - Configuration changes needed: none expected.
   - API changes: tool results from `query_lakehouse` (and the MCP equivalent) 
on the Spark Connect backend gain faithful encodings. Structs become JSON 
objects keyed by field name. Timestamps become ISO-8601 strings, with an offset 
for `TIMESTAMP` and without one for `TIMESTAMP_NTZ`. Decimals use one 
documented, consistent encoding.
   - Usage examples: `SELECT named_struct('a', 1, 'b', 'x') AS s, 
current_timestamp() AS ts, CAST(1.10 AS DECIMAL(10,2)) AS d` returns `{"s": 
{"a": 1, "b": "x"}, "ts": "...+00:00", "d": ...}` instead of `[1, "x"]`, a 
naive timestamp and a string.
   
   **Suggested scope:**
   1. A type-aware encoder for collected rows, applied before the existing 
result-size limit and truncation, so the limit still applies to the final 
payload.
   2. Decide the decimal encoding (exact string vs. JSON number) and the 
timestamp zone (session time zone vs. UTC), and document both.
   3. Compare with the Trino backend's output. Make the two backends consistent 
where that's practical, or document where they differ.
   4. Unit tests covering nested structs, arrays of structs, maps, `TIMESTAMP` 
vs `TIMESTAMP_NTZ`, `DATE`, high-precision `DECIMAL`, `BINARY` and NULLs. Add a 
live check in `tests/integration/test_spark_connect_live.py`.
   
   ### Hudi RFC Requirements
   
   **RFC PR link:** (if applicable)
   
   **Why RFC is/isn't needed:**
   - Does this change public interfaces/APIs? Only the JSON shape of gateway 
tool results on the Spark Connect backend.
   - Does this change storage format? No
   - Justification: a contained change in `hudi-agent-gateway`.
   
   ---
   
   Parent: #20234
   
   @cshuo could you take this one up as well? It's the remaining Phase 2 item 
(M2.2) from the 2.0 plan, and we'd like it to land within 2.0.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to