eldenmoon opened a new pull request, #67424:
URL: https://github.com/apache/doris/pull/67424
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Variant V2 Parquet export rendered the existing
metadata/value encoding through JSON and UTF-8 before writing Parquet. This
change selects Arrow's Parquet Variant extension only in the Parquet
transformer, lets `DataTypeVariantV2SerDe` append the existing metadata/value
bytes with exact buffer reservation, and keeps the common Arrow converter
generic for extension storage arrays. On the fixed 10 GB workload, the wall
median decreases from 77.7376 to 53.0676 seconds and BE CPU decreases from
245.04 to 213.88 core-seconds, improvements of 31.73% and 12.72% respectively.
DuckDB recognizes and decodes the output as VARIANT; Doris legacy Parquet
schema inference and Stream Load do not yet consume this logical type end to
end.
### Release note
Variant V2 Parquet export writes the Parquet Variant logical type directly.
Doris reader and load integration is not included in this PR.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- ASAN BE UT: `DataTypeSerDeArrowTest.*`, 13/13 passed.
- Write/read canonical equality covers object, array, string,
positive and negative integers, floating point, booleans, Variant null, and SQL
null.
- [x] Manual test
- Release build: `env BUILD_TYPE=RELEASE GLIBC_COMPATIBILITY=OFF
./build.sh --be --hive-udf` passed.
- DuckDB 1.5.2 recognized all 2,906,455 exported rows as VARIANT,
reported no null values, and decoded sampled objects.
- One warmup and three measured trials on the fixed 10 GB workload.
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [ ] No.
- [x] Yes. Variant V2 Parquet export uses the Parquet Variant logical
type.
- Does this need documentation?
- [x] No. This remains experimental pending Doris reader/load
integration.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
