eldenmoon opened a new pull request, #70212:
URL: https://github.com/apache/doris/pull/70212
### What problem does this PR solve?
Issue Number: N/A
Related PR: #69995 (found during review; this defect also exists on master),
#70060 (independent internal-column-name classification fix)
Problem Summary:
Compaction can promote an ordinary Variant path from sparse storage while
its descendants stay sparse. `BinaryColumnExtractIterator` currently uses the
hierarchical `VariantAssembler` for this physical extraction. A row containing
only `a.b` then becomes a non-null value in the promoted `a` column as well.
With `variant_max_subcolumns_count=1`, three input rowsets each materialize
a different hot key and leave two scalar `a` values plus one `a.b` value
sparse. Across the rowsets, `a` is selected for promotion. Its physical
non-null count grows from 6 to 9 and compaction fails with `Path stats not
match`. The failure happens before replacing the input rowsets, so the old data
remains readable, but retries with the same selection fail again. This occurs
with ordinary keys and does not require dotted-key escaping.
Extract only the exact persisted path using the existing sparse-path binary
search and `decode_v1_storage_cells`. Keep the existing missing/SQL-NULL and
present Variant-null semantics, validate offsets before accessing cells, and
retain full object payloads stored at the exact path. Hierarchical query
assembly continues to use `VariantAssembler`.
Self-review: the change stays within physical sparse extraction, shared by
compaction/checksum readers. Query subtree assembly retains its own assembler.
Scratch vectors use Doris memory accounting; borrowed cells are decoded before
appending the owning result, and invalid offsets return Corruption before
access. No stored-format, protocol, schema, lock, publication or delete-bitmap
changes are required. Runtime validation uses a local shared-nothing cluster;
cloud callers were checked statically.
### Release note
Fix compaction failures when a Variant ancestor path is promoted from sparse
storage alongside its descendants.
### Check List (For Author)
- Test:
- ASAN BE/FE builds passed. Original BE: the new unit test fails in all
four path/read combinations; SQL reproduces the exact `6 -> 9` mismatch,
preserves the original rowsets, and reproduces F-04's `701/702/703 -> NULL`
reads separately.
- Fixed BE: **169 focused unit tests passed, 2 skipped**
(NestedGroup-dependent tests unavailable in this build). `clang-format` 16,
build hygiene and changed-line clang-tidy passed.
- `sparse_ancestor_promotion`, `sparse_key_byte_order` and
`regression_test_query_subcolumns` passed in comparison mode. New expectations
were generated by the repository runner and independently checked: **560
ordered row projections and 16 successful FULL compactions**; all before/after
pairs match.
- Retrying the original failing table on the fixed BE returns `[OK]`,
replaces four input rowsets with one output rowset, and preserves every
whole-value/subpath result. The same row values remain intact after restarting
FE and BE.
- Added BE coverage for exact-path promotion, descendant-only rows,
neighboring keys, exact object payloads, Variant nulls, missing values, nested
paths, batch reads and reordered rowid reads.
- Added `sparse_ancestor_promotion`: DUP/MOW, segment formats V2/V3, 1/3
sparse shards, two FULL compactions, ordered whole-value and subpath checks,
and explicit final compaction-status checks.
- Behavior changed: Yes. Physical sparse extraction no longer synthesizes
ancestor values from descendant paths.
- Does this need documentation: No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
