morningman opened a new pull request, #69243:
URL: https://github.com/apache/doris/pull/69243
### What problem does this PR solve?
Issue Number: None
Related PR: #68786
Problem Summary:
**In short.** The V1 ORC reader rejects TIMESTAMPTZ values in year 0, which
Doris stores and the other read paths return. Found while comparing master with
branch-4.x after #68786; the branch-4.x V1 reader has no range check here.
**Background.** #68786 reads ORC `TIMESTAMP_INSTANT` as TIMESTAMPTZ and
added a UTC range check to the V1 ORC reader before it packs each value,
because ORC allows years that Doris cannot hold. Doris TIMESTAMPTZ holds UTC
years 0 through 9999: `TimestampTzValue` rejects only years below 0 or above
9999, and a TIMESTAMPTZ column stores `0000-01-01 00:00:00 +00:00`. In the
review of #68786 the JNI `VectorColumn` check was changed to reject only years
below 0, and master's ORC serde uses the same 0–9999 range.
**The problem, and what it cost.** The V1 check kept its lower bound at
0001-01-01T00:00:00Z (`MIN_SECONDS = -62135596800`). An ORC file with a year-0
value fails the scan with `Decoded TIMESTAMPTZ column '...' is outside the
Doris 0001-9999 range`, while the V2 reader returns the value.
**How this PR fixes it.** The lower bound becomes 0000-01-01T00:00:00Z
(`-62167219200`; year 0 is a leap year, 366 days before year 1). The upper
bound and the error for values outside the range are unchanged, and the message
now says 0000-9999.
**Results.** BE unit test on the V1 decode path:
| Value (UTC) | Before | After |
|---|---|---|
| 0000-01-01T00:00:00Z | DataQualityError | Decoded as year 0 |
| 0000-12-31T23:59:59Z | DataQualityError | Decoded as year 0 |
| 0001-01-01T00:00:00Z, 9999-12-31T23:59:59.999999Z | Decoded | Same |
| One second before year 0, 10000-01-01T00:00:00Z | DataQualityError | Same |
### Release note
Fix reading year-0 TIMESTAMPTZ values from ORC files with the V1 reader.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- `OrcReaderFillDataTest.TimestampTzV1RejectsOutOfRangeUtcYears` now
accepts 0000-01-01T00:00:00Z and 0000-12-31T23:59:59Z and rejects one second
before year 0. It fails on master and passes with this change; the other 7
`OrcReaderFillDataTest` cases pass. No regression test: none of the ORC
fixtures holds a year-0 `TIMESTAMP_INSTANT` value.
- Behavior changed:
- [ ] No.
- [x] Yes. Reading year-0 TIMESTAMPTZ values from ORC with the V1 reader
no longer fails.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
