Gabriel39 opened a new pull request, #68381:
URL: https://github.com/apache/doris/pull/68381
### What problem does this PR solve?
Related PR: #67784. This is the second split, based on the primitives merged
in #68297.
Arrow batch conversion mixes protocol serialization with table-specific UUID
and Variant handling, while writers construct schemas separately. Introduce
explicit Doris, Python, Arrow Flight, Parquet, Hive, Iceberg and Paimon
convertors, instantiated with their own schema parameters and timezone. Move
table schema construction/decoding into the convertors and route nested SerDe
writes through the selected format.
Separate Parquet, Hive and Iceberg writers and migrate all callers. Preserve
current Parquet timestamp encoding, UUID/Variant layouts and external type
mappings. Keep Python's numeric conversion timezone consistent with its
declared Arrow schema, and reject incompatible nested target schemas before
casts or child access. No FE, Thrift, ORC timestamp, or binary computation
changes are included.
### Release note
Fix Python UDF timestamp conversion to preserve wall-clock values when the
Arrow protocol declares a fixed-offset timezone.
### Check List (For Author)
- Test
- [x] Unit Test: schema ownership/isolation, slices, invalid schemas,
nested/null values, UUID/Variant bytes, fixed-offset Python batches, and
Parquet timestamp representation.
- [x] Regression test: add Python UDF/UDTF/UDAF timezone assertions and
extract the corresponding existing Python snapshot corrections from #67784.
- Behavior changed:
- [x] Yes: align Python UDF conversion with its Arrow timezone
declaration; report invalid nested schema bindings as errors.
- Does this need documentation?
- [x] No.
Validation: ASAN BE build and 396 selected tests passed; the final
incremental build and 285 focused tests also passed. clang-format 16 and
Groovy/embedded Python syntax checks passed. Python SQL and external-catalog
end-to-end regressions were not run locally and remain for CI.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
