hubgeter opened a new pull request, #69888:
URL: https://github.com/apache/doris/pull/69888
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
ORC `Int128` statistics already contain two's-complement bits, but the
Iceberg ORC writer negates negative values again before encoding them. A file
containing `[-100, -1]` consequently reports bounds `[100, 1]`, so a negative
predicate can prune a file that contains matching rows.
ORC footer statistics also trim trailing fractional zeros. For example,
`-1.00` is returned as `value=-1, scale=0`; a `DECIMAL(...,2)` manifest needs
the unscaled value `-100`. Preserving the sign alone therefore leaves incorrect
bounds.
Restore each bound to the declared column scale with the existing ORC
scaling utility, then serialize its original signed bits using minimal
big-endian encoding. Preserve necessary sign prefixes such as `128 -> 00 80`
and `-129 -> FF 7F`.
The fix applies to newly collected bounds. Existing incorrect metrics
require recomputation or data-file rewriting.
### Release note
Correct decimal bounds emitted by Iceberg ORC writes to prevent incorrect
predicate pruning for negative values and values with trailing fractional zeros.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Added BE tests cover 29 encoding cases, real ORC statistics at precisions
1/9/18/38, scale restoration through scale 38, nullable columns, and all-null
columns. The external regression checks manifest pruning for negative and mixed
decimal data across decimal32/64/128.
Local validation:
- An isolated codec harness built from the production function passed all 29
encoding cases.
- Five real ORC files passed independent Iceberg 1.12.0 manifest write/read
and predicate-pruning probes.
- clang-format 16, build hygiene, changed-line clang-tidy, and Groovy 4.0.19
syntax checks passed. Static analysis used real ASAN UT compilation commands
with PCH expanded as headers.
- The full BE-UT build was stopped during the monolithic dependency build
before tests executed. The external cluster regression was not run.
- Behavior changed:
- [ ] No.
- [x] Yes. Newly written ORC decimal bounds preserve the sign and the
column scale.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
