Gabriel39 opened a new pull request, #67784:
URL: https://github.com/apache/doris/pull/67784
### What problem does this PR solve?
Issue Number: None
Related PRs: #66891, #65446
Problem Summary:
This PR is based on #66891 and completes the timestamp contracts needed by
Iceberg and Paimon on `branch-4.1`.
It fixes the following issues in the Arrow writer refactor:
- Allow timezone-free Arrow timestamps only for Doris `DATETIMEV2`;
`TIMESTAMPTZ` continues to require instant semantics.
- Validate that nested Array, Map, and Struct bindings preserve their Arrow
field contracts while recursively accepting the declared timestamp
representation.
- Include the complete Arrow type definitions required by the Iceberg and
Paimon converters.
- Map Doris `DATETIMEV2(p)` to Paimon `TIMESTAMP(p)` and Doris
`TIMESTAMPTZ(p)` to Paimon `TIMESTAMP_LTZ(p)` without losing precision.
It also ports the timestamp correctness fixes from #65446 to the direct
scanner architecture used by `branch-4.1`:
- Preserve Parquet INT96 wall-clock values by default and add the explicit
`hive.parquet.time-zone` compatibility property for HMS catalogs and file TVFs.
- Keep Hudi's legacy session-timezone behavior and prevent Iceberg/Paimon
tables discovered through HMS from inheriting the Hive compatibility timezone.
- Preserve Paimon `TIMESTAMP` versus `TIMESTAMP_LTZ` semantics, including
nested filter-only projections backed by unannotated INT96.
- Round ORC timestamp nanoseconds consistently and keep timestamp
statistics/SARG pruning conservative.
- Write Parquet INT64 logical timestamps by default while retaining explicit
`enable_int96_timestamps=true` compatibility.
- Carry the required fixed-offset timezone detection and normalization
helpers into `branch-4.1`.
### Release note
Parquet Export and Outfile now write INT64 logical timestamps by default.
Set `enable_int96_timestamps=true` for legacy readers that require INT96.
Scanner v2 preserves raw INT96 wall-clock values by default; set
`hive.parquet.time-zone` for Hive files normalized by a known writer timezone.
Hudi keeps its existing session-timezone behavior, while Iceberg and Paimon
retain their table-format timestamp semantics. ORC timestamp nanoseconds are
rounded consistently to microseconds.
### Check List (For Author)
- Test
- [x] Unit Test
- [x] Compile validation
- [x] FE Checkstyle
- [ ] Regression test
Validation:
- FE focused tests: 197 passed, 0 failures/errors.
- Hudi JNI scanner tests: 2 passed, 0 failures/errors.
- All 28 changed C++ production/test translation units compiled with the
branch's Arrow 24 dependency; the #66891 Arrow schema translation unit was also
verified with Arrow 24.
- Clang-format 16 dry-run passed for all 38 affected C/C++ source and header
files.
- FE Checkstyle passed with 0 violations.
- Full local ASAN UT binary execution was not available because the existing
local CMake cache was configured against Arrow 17; the changed translation
units were recompiled with Arrow 24, and `run buildall` is requested below for
the standard full CI environment.
- Behavior changed:
- [x] Yes
- [ ] No
- Does this need documentation?
- [ ] Yes
- [x] No
This is a branch backport of already documented behavior from #65446 plus
internal Arrow/Paimon type-contract fixes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
