yihua opened a new pull request, #832:
URL: https://github.com/apache/hudi-rs/pull/832
## Description
The DataFusion table provider now declares top-level string and binary
columns as `Utf8View` / `BinaryView`, following
`datafusion.execution.parquet.schema_force_view_types` (on by default,
forwarded from the session by the table factory or passed as a table option),
so the copy-on-write parquet scan decodes strings straight into view arrays.
The merge-on-read scan casts its string and binary columns to the declared view
types and leaves every other column as the reader produced it. DataFusion
consumers now get view arrays for these columns unless they set the option to
`false`; the Python binding keeps `Utf8` by default, since it cannot see a
Python session's own setting.
TPC-H with DataFusion, local, median of 3 alternating rounds, before ->
after:
- SF1 copy-on-write: total 4354 -> 4090 ms (-6%); Q19 212 -> 155 ms, Q10 265
-> 238 ms, Q12 280 -> 252 ms, Q13 145 -> 129 ms.
- SF0.01 merge-on-read: total 723 -> 734 ms, unchanged within noise.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
Tests cover the schema following the option by default, when off,
case-insensitively, through `SET` plus `CREATE EXTERNAL TABLE`, and rejecting
an invalid value; merge-on-read reads (file group reader versions 1 and 2)
returning the declared view types with the same values as `Utf8`; the batch
cast touching only view-typed columns, keeping the batch's nullability, and
refusing a misaligned batch; and the Python default. The shared test helpers
accept both string representations, so existing tests run unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
