morningman opened a new pull request, #68918:
URL: https://github.com/apache/doris/pull/68918
### What problem does this PR solve?
Issue Number: None
Related PR: #68532, #68786
Problem Summary:
**In short.** This ports the tests of branch-4.2 #68532 that master does not
have yet. Master re-implemented #68532 in #68786; a hunk-by-hunk comparison
found all of its product behavior on master, but about twenty of its assertions
had no master equivalent. This PR adds them, fixes three Paimon tests that set
a misspelled option, and changes no product code.
**Background.** #68532 (merged into branch-4.1 and synced to branch-4.2)
maps external binary columns to VARBINARY and instant timestamps to TIMESTAMPTZ
across Hive, Iceberg, Paimon, JDBC, Hudi, MaxCompute and the Trino connector.
On master the external catalogs live in `fe/fe-connector/*`, so #68786
re-implemented the change on the connector plugins rather than cherry-picking
it, and moved most of the #68532 tests along. The ones listed below were not
moved.
**The problem, and what it cost.** Without these tests nothing on master
fails if one of these behaviors regresses:
| Area | Behavior with no test on master |
| --- | --- |
| Iceberg partitions | 0x-hex BINARY/FIXED/UUID commit values are decoded,
and wrong lengths or malformed hex are rejected; transform ranges carry
`+00:00` bounds and the proleptic year zero; zoned and unzoned partition
timestamps round-trip in every session zone, including both instants of a DST
fold; commit values honor their explicit offsets |
| Iceberg writes | zoned write defaults keep the instant and nested binary
defaults stay bytes with the old flags off; binary/UUID commit values become
typed partitions; a NULL static overwrite on a binary/fixed/uuid key replaces
only the NULL partition (full and hybrid specs) and reaches the sink as a NULL
marker distinct from bytes; MODIFY to TIMESTAMPTZ on a zoned field is a no-op
while DATETIMEV2 is rejected; Iceberg does not use Hive's INT96 time zone |
| fe-core | TIMESTAMPTZ range partitions keep UTC bounds and a non-empty
NULL-min range in UTC, Asia/Shanghai and America/New_York sessions; the legacy
JDBC clients that CDC still uses map binary and instant types regardless of the
old flags, and PostgreSQL `timestamptz` becomes nullable |
| Paimon | nested LTZ write columns stay TIMESTAMPTZ; LTZ partition values
keep pre-epoch and DST-fold instants in every zone; legacy ORC LTZ files route
to the SDK for MAP/MULTISET/ROW leaves and mixed splits |
| JDBC / HMS / Trino | per-dialect binary and timestamp mappings; calendar
functions on instants, TIMESTAMPTZ literals, PostgreSQL decoded NULLs and
dialect-specific binary literals stay local; Hive binary partition keys keep
their text type; Trino domains encode UTC micros and millisecond-aligned zoned
bounds; the JDBC writer sets the MySQL session to UTC before it prepares the
INSERT |
| Regression | a Paimon LTZ INSERT SELECT of two instants that fold to the
same Los Angeles time; a Spark cross-check of LTZ rows written in a non-UTC
session, including a pre-epoch instant; a native V1 read of the Paimon
timestamp-scale tables |
Separately, three `PaimonScanPlanProviderTest` cases set
`orc.timestamp-ltz.legacy-type`; the real key is
`orc.timestamp-ltz.legacy.type`, so the setting was ignored and the tests
passed only because the default is `true`.
**How this PR fixes it.** It ports each missing #68532 assertion to the
master class that now owns the behavior (mostly connector tests), adapted to
the connector APIs; follows master's regression style (`order_qt_` blocks plus
Spark/Doris cross-checks instead of literal asserts); and corrects the
misspelled option key.
**Results.** The 19 changed test classes pass locally. To check that the new
tests are not vacuous, six product-code mutations were applied across five
modules: dropping the binary-literal pushdown guard, an always-true NULL static
overwrite filter, no FIXED length check, parsing the NULL-min sentinel as a
date, ignoring MULTISET when routing legacy ORC LTZ, and initializing the MySQL
UTC session after the prepare. Each mutation failed exactly the corresponding
new test and nothing else. Both Paimon regression suites pass against the
iceberg docker environment, with the new `.out` blocks generated by the
framework and matching #68532's expected values (for example epoch micros
`1699173000123456` and `1699176600123456` for the fold).
Original author: @Gabriel39
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Unit tests: 19 changed classes pass locally (Iceberg connector 533,
Paimon connector 130, fe-core 83, jdbc-scanner 22, JDBC connector 17, Trino
connector 4, HMS 1 test cases); checkstyle passes. Regression:
`test_paimon_write_types` and `paimon_timestamp_types` pass locally against the
iceberg docker environment with FE/BE running in Asia/Shanghai (the CI time
zone); the `.out` files were generated by the regression framework and then
verified by a normal run.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
