bobhan1 opened a new pull request, #70350:
URL: https://github.com/apache/doris/pull/70350
### Background
MoW compaction records a mapping from each input row to its output segment
and row ID. Delete bitmap conversion and inverted-index merging consume this
mapping, whose dense representation stores two `uint32_t` values for every
physical input row, including filtered rows.
### What problem does this PR solve?
Issue Number: None
Related PR: None
PR Type: Performance
Problem Summary:
Large tablets can require gigabytes of row-ID mapping memory and repeatedly
fail base compaction when allocations exceed the memory limit. This PR adds
opt-in compression during construction, using fixed 4096-source-row blocks with
delta varint64 and Zstd level 1. It adapts delete bitmap and SNII consumers to
bounded decoding while preserving validation and error propagation.
`enable_compressed_rowid_conversion` defaults to `false`, so existing
deployments keep their current representation unless explicitly enabled.
### Details
- **Change map:** 1,183 production lines changed, 644 test lines, 57
documentation lines, and no generated files. The compressed storage kernel
accounts for 625 production lines; source views and destination-prefix
translation for 258; SNII validation and posting access for 213; mode
selection, bitmap error propagation, configuration and build integration for
the remaining 87. Tests cover storage representations, consumer equivalence,
allocation accounting and performance observations.
- **Why it was slow:** Memory is the bottleneck: the dense array costs 8
bytes per physical input row, or 8 GB for one billion rows, before other
compaction allocations. Allocation failures can prevent compaction from
completing.
- **Optimization:** Seal fixed source-row blocks during merge, without
assuming a maximum number of rows per segment. EMPTY and LINEAR omit redundant
values; DELTA uses delta varints with Zstd only when beneficial; RAW64 caps
payload expansion. A shared 1 MiB paged arena avoids a separate allocation for
every block. Each source has a directory with 32 bytes per block and a lazy 32
KiB active builder; codec workspaces are bounded by one block. Finalization
releases builders and the encoder, then derives destination positions from
actual output segment row counts using 64-bit prefixes. SNII retains one
decoded block per prepared source and shares one decoder workspace per
executing plan. Significant allocations use Doris memory tracking; lookup
errors propagate as `Status` instead of being mistaken for deleted rows.
- **Results:** Four synthetic benchmarks each build and read all 262,144
physical source rows, using 1 or 16 sources, approximately 10% missing rows,
and two target-gap distributions. The dense array alone is 2,097,152 bytes. The
compressed directory is 2,048 bytes in each case, and sequential reads decode
exactly 64 blocks even when sources are interleaved.
| Sources | Gap parameter | Encoded payload (bytes) | Retained kernel
allocation (bytes) | Tracked build/overall peak (bytes) |
| ---: | ---: | ---: | ---: | ---: |
| 1 | 1 | 20,273 | 1,050,832 | 1,470,944 |
| 1 | 16 | 185,936 | 1,050,832 | 1,470,944 |
| 16 | 1 | 20,704 | 1,051,792 | 1,963,424 |
| 16 | 16 | 229,915 | 1,051,792 | 1,963,424 |
Retained allocation includes directory capacity, the entire 1 MiB arena page
and storage-kernel metadata. The consumer-tracker peak includes transient
allocations and reader workspaces; it is requested allocation size, not RSS.
These kernel measurements exclude allocator bookkeeping, benchmark
oracle/baseline copies, the enclosing source index/destination prefix and other
compaction state. The enclosing index and prefix also use tracked allocators
but are tested separately. A separate billion-row EMPTY-source test verifies
its complete 7,812,512-byte directory; it is not a billion-row compression
benchmark.
On the local ASAN/O0 build, construction took 65.4–95.3 ms, one cold lookup
per block took 3.5–10.3 ms, and sequential lookup took 80.0–101.4 ms versus
2.9–5.9 ms for the direct dense scan. The 4,096 grouped sparse lookups took
31.8–446.4 ms, and 4,096 random lookups took 211.5–446.5 ms. These are
diagnostic observations, with UT assertions included in sequential timing, not
optimized production throughput or end-to-end compaction results.
- **Applicability and cost:** Compression is limited to eligible MoW
UNIQUE-key compactions. Cluster-key tables, sequence-map tables, compactions
requiring CLucene and legacy inverted-index debug dumps retain DENSE. The
source streams must advance monotonically. Sparse/random access can repeatedly
decode blocks, and small maps can be dominated by arena tail space and
per-source workspaces. No spill or external CLucene API changes are introduced.
- **Behavior change:** The new dynamic BE setting enables an alternative
in-memory representation for eligible compactions. DENSE remains the default.
Bitmap visibility rules and SNII bounds, monotonicity, uniqueness and complete
destination coverage validation are retained; persistent formats and FE/BE
protocols are unchanged.
### Testing
Locally, 69/69 tests passed in `ASAN_UT` at commit
`fbedc9a49fd8e05e950f82b241099d9dfc8cc8cd`, based on master
`4c63661f60f96c694757000640bcebc832e8bb6d`:
```sh
bash run-be-ut.sh --run \
--filter='CompressedRowIdMapTest.*:CompressedRowIdMapBenchmark.*:SniiRowIdConversionTest.*:SniiPostingCursorTest.*:SniiIndexCompactionTest.*:TestRowIdConversion.*:VParquetWriterTest.HiveAndIcebergParquetPageIndexOptions'
\
-j32
```
Coverage includes all representations, deletion holes, wide ordinals, arena
page boundaries, independent cursors, backward seeks, malformed payloads,
allocation failures and balanced memory accounting. Consumer tests compare
compressed mappings with DENSE, verify physical segment IDs and visibility
filtering, and check that an in-memory SNII merge produces byte-identical
null/posting data to a rebuilt index. The additional Parquet case verifies the
upstream page-index test after rebasing.
Repository clang-format 16 checks on changed files,
`build-support/check-build-hygiene.sh` and `git diff --check` passed. Actual
local/cloud compaction, concurrent loading, lock-hold time and optimized
production timing are deferred to a separate validation phase. clang-tidy was
not run.
### Notes for Reviewers
See
[`be/test/storage/rowid_conversion_benchmark.md`](https://github.com/bobhan1/doris/blob/fbedc9a49fd8e05e950f82b241099d9dfc8cc8cd/be/test/storage/rowid_conversion_benchmark.md)
for reproducible commands and precise accounting boundaries. The small
benchmark's single arena page must not be extrapolated as a new page for every
262,144 rows in a larger shared map.
The explicit `cloud/config.h` include in `segment_read_ahead.cpp` fixes its
direct `is_cloud_mode` dependency exposed by changed unity-batch boundaries.
The compressed codec source is excluded from unity compilation because it uses
Zstd's static workspace API. Keep the setting disabled by default pending
actual compaction validation.
### Release note
An experimental `enable_compressed_rowid_conversion` BE setting can reduce
row-ID mapping memory for eligible MoW compactions. It is disabled by default.
### Check List (For Author)
- Behavior changed: Yes (see Details)
- Does this need documentation: Yes; the in-tree benchmark guide covers
developer use. User documentation will follow actual compaction validation.
- Pre-submission status: Partially verified: targeted ASAN unit tests pass;
actual compaction validation and optimized performance measurements are pending.
### AI Disclosure
- Tool: OpenAI Codex (GPT-6)
- Generated: Implementation, unit tests, benchmark, developer guide and PR
description.
- Human review: Pending.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
