hubgeter opened a new pull request, #70214:
URL: https://github.com/apache/doris/pull/70214
### What problem does this PR solve?
Issue Number: N/A
Related PR: #68786
Problem Summary:
Writing an Iceberg STRING partition whose value is the literal `'null'` can
commit it as SQL NULL. For example, after inserting `(1, 'null')` and `(2,
NULL)`, partition pruning for `p = 'null'` can omit row 1. The writer routing
fix in #68786 separates the files, but commit fragments still use the same
textual token for both partition values.
Preserve nullable partition values through dynamic, static, and mixed
partition writers, and add an optional NULL bitmap to the commit protocol.
Decode it consistently when constructing data files, delete files, and identity
partition conflict filters. Keep the legacy token fallback for senders without
the bitmap, and preserve typed NULLs in existing JSON fragments. Reject
inconsistent bitmap or partition-field counts before committing metadata.
The complete correction requires upgraded FE and BE processes. It applies to
new writes and does not rewrite existing incorrect manifests. Older receivers
ignore the optional field and retain their previous behavior.
### Release note
Fix Iceberg writes incorrectly treating the literal STRING partition value
`'null'` as SQL NULL, which could cause partition-filtered queries to miss rows.
### Check List (For Author)
- Test
- [x] Regression test: Parquet and ORC dynamic writes, metadata
partitions, static and mixed partition overwrites, empty strings, and truncate
partitions; generated output and comparison both passed (28 query results).
- [x] Unit Test: 150 FE writer-helper and connector-transaction tests
passed. BE focused unit tests are building; not yet executed.
- [x] Manual test: `bash build.sh --be -j 6` (ASAN) and `bash build.sh
--fe -j 4` passed; formatting, header hygiene, and FE Checkstyle passed. Both
modified BE test files passed full-file clang-tidy; the final changed-line scan
is in progress.
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes. Newly written literal `'null'` and SQL NULL partitions retain
distinct manifest values and conflict filters; malformed commit partition
metadata is rejected.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
