morningman opened a new pull request, #67530:
URL: https://github.com/apache/doris/pull/67530
### What problem does this PR solve?
Related Jira: DORIS-28389
Problem Summary:
Arrow has no equivalent for LARGEINT, IPV4, IPV6, JSON or VARIANT. Each of
them travels over Flight SQL as some other Arrow type and is then
indistinguishable from a column that is natively of that type -- a LARGEINT and
a STRING both arrive as `utf8`, an IPV4 and an INT both arrive as `int32`. The
`doris_type` field metadata is the only thing that tells them apart.
That metadata was attached in `get_arrow_schema_from_block` and
`get_arrow_schema_from_expr_ctxs`, which only see the top level columns.
`convert_to_arrow_type` recurses into ARRAY / MAP / STRUCT building Arrow
**types**, and Arrow keeps metadata on the `Field` rather than on the
`DataType`, so every nested element lost it: `ListType(item_type)` synthesizes
a bare `"item"` field, `MapType(key_type, val_type)` synthesizes bare
`"key"`/`"value"` fields, and the STRUCT branch built its fields without going
through the metadata helper.
Reading this through a Python ADBC client:
```sql
SELECT CAST(495 AS LARGEINT) AS scalar_value,
named_struct('count', CAST(495 AS LARGEINT)) AS struct_value,
array(CAST(495 AS LARGEINT)) AS array_value,
map('k', CAST(495 AS LARGEINT)) AS map_value;
```
**Before** -- only the top level column is identifiable:
```
scalar_value: string metadata={'doris_type': 'LARGEINT'}
struct_value: struct<count: string>
count: string metadata={}
array_value: list<item: string>
item: string metadata={}
map_value: map<string, string>
entries: struct<key: string not null, value: string>
key: string metadata={}
value: string metadata={}
```
**After**:
```
scalar_value: string metadata={'doris_type': 'LARGEINT'}
struct_value: struct<count: string>
count: string metadata={'doris_type': 'LARGEINT'}
array_value: list<item: string>
item: string metadata={'doris_type': 'LARGEINT'}
map_value: map<string, string>
entries: struct<key: string not null, value: string>
key: string metadata={}
value: string metadata={'doris_type': 'LARGEINT'}
```
A nested IPV4 was the worse case of the same defect: it arrives as its 32
bits read as a signed `int32` (`192.168.1.1` as `-1062731519`) with nothing
left to say it was ever an address.
### What is changed?
`convert_to_arrow_type` now builds the child `Field`s through
`create_arrow_field_with_metadata` at every level, for ARRAY, MAP and STRUCT.
The names and the nullability are exactly the ones Arrow's own constructors
produced -- `"item"` nullable, `"key"` non-nullable, `"value"` nullable -- so
**only the metadata is new**, and the record batch builders, which are made
from this same schema (`FromBlockToRecordBatchConverter` reads
`_schema->field(idx)->type()`), are unaffected. `DataType::Equals` ignores
metadata by default, so the batch still type-matches the schema.
The metadata lookup is also completed: JSONB and VARIANT are serialized as
`utf8` too and were carrying no `doris_type` at all, not even at the top level.
The values (`JSON`, `VARIANT`) match what the FE reports for the same column
under `ARROW:FLIGHT:SQL:TYPE_NAME` in `FlightSqlSchemaHelper`.
No value or Arrow type changes -- this is metadata only.
### Release note
Fix Arrow Flight SQL losing the Doris logical type (`doris_type` field
metadata) of LARGEINT / IPV4 / IPV6 nested inside STRUCT, ARRAY and MAP, which
made a nested LARGEINT indistinguishable from a STRING for ADBC clients. JSON
and VARIANT columns now carry the same metadata as well.
### Check List (For Author)
- Test
- [x] Unit Test
New `be/test/format/arrow/arrow_row_batch_test.cpp`, 7 cases:
nested LARGEINT in STRUCT / ARRAY / MAP value / MAP key; every level of
`array<struct<largeint>>` and `map<string, array<array<largeint>>>`;
nested IPV4 / IPV6 / JSON / VARIANT; a negative control asserting STRING
and
INT carry no metadata nested or not; an assertion that the nested types
still
compare equal to `arrow::list(utf8())` / `arrow::map(utf8(), utf8())` with
`check_metadata=false` and differ only with `check_metadata=true`; and one
that runs `convert_to_arrow_batch` over the new schema and checks
`ValidateFull()`, schema equality including metadata, and the values.
Verified the assertions are not vacuous: with the `arrow_row_batch.cpp`
change
reverted, 5 of the 7 fail; the negative control and the batch-building
guard
correctly stay green.
- [x] Manual test (add detailed scripts or steps below)
Built BE locally and ran the Jira's repro through
`adbc_driver_flightsql` (Python), plus a real table with
`largeint`, `array<largeint>`, `struct<count:largeint, name:string>`,
`map<string,largeint>`, `ipv4` and `json` columns. Nested fields now carry
`doris_type` at every depth, a STRING sibling inside the same STRUCT
correctly
carries none, and the values are unchanged (LARGEINT extremes round trip
losslessly as text).
- Behavior changed:
- [x] Yes.
The Arrow schema returned over Flight SQL now carries `doris_type` field
metadata on nested fields, and on top level JSON / VARIANT columns. Arrow
types, field names, nullability and values are unchanged, so a client that
ignores metadata sees no difference.
- Does this need documentation?
- [x] No.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01FP36MDAsUyKSSDQXQohQsK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
