eldenmoon opened a new pull request, #70001:
URL: https://github.com/apache/doris/pull/70001
### What problem does this PR solve?
Issue Number: N/A
Related PR: #69995
#69995 (`variant_escape_dots_in_keys`) sorts the sparse plan by bytes only
for escaped columns; whichever PR merges second removes that escaped-only sort,
since this PR sorts the sparse plan in all modes.
Problem Summary:
With variant_max_subcolumns_count=1, insert {"z":1,"a-b":1,"a":{"c":2}}
followed by rows containing only z. Direct a-b and a.c projections return
NULL/NULL even though the whole value retains both leaves. The trigger is a key
component that is a prefix of a sibling component continuing with a byte below
`.` (space, `!`, `"`, `#`, `$`, `%`, `&`, `'`, `(`, `)`, `*`, `+`, `,`, `-`),
with both paths in the same sparse bucket of one row. The writer orders paths
by components (a.c before a-b), while the reader binary-searches full path
bytes (a-b before a.c).
Sort the sparse plan by full path bytes after select_sparse_statistics,
preserving the statistics budget tie-break, statistics publication and
materialized order. This is a writer-only fix: the reader is unchanged, so it
adds no read cost. Newly written rows return 1/2 for the repro projections.
Rows written before this fix keep their order and are not repaired; Variant
V2 is not enabled by default in any release (branch-4.1/4.2 default
enable_variant_v2=false), so no reader compatibility path is added.
### Release note
Fix incorrect NULL results when querying newly written Variant V2 sparse
keys such as a-b beside nested a.c.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Unit Test: 140 affected tests across 6 suites; 138 passed and 2 existing
NestedGroup write-path tests skipped because the write path is unavailable in
this build. SparseRowsUseByteOrderWithoutChangingStatisticsSelection failed
with the HEAD writer and passed with the fix, including reads through the
unchanged assembler.
- Regression test: sparse_key_byte_order passed with -parallel 1 and no
-genOut, comparing the unchanged generated output for all 96 rows (DUP/MOW, 1/3
shards, ordinary/doc, before/after full compaction). Full variant_p0 ran 183
suites: 181 passed and 2 known unrelated failures (object-storage
InvalidAccessKeyId/code=403 and the old schema-change public=1 versus true
baseline). Both failures were independently rerun with -parallel 1 and fail the
same way on unmodified master (0976b2cd502).
- Manual test: the new ASAN build writes the repro table and returns 1/2 for
a-b/a.c; parent and whole-value reads also match.
- Validation: ASAN BE build, clang-format 16, build hygiene and git diff
--check passed. Clang-tidy passed for both changed C++ files with no
diagnostics on added lines.
- Behavior changed:
- [ ] No.
- [x] Yes. New Variant V2 sparse rows use byte order so sparse
projections return their stored values.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label (branch-4.1 and branch-4.2 have the same V2
sparse-plan ordering defect)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
