github-actions[bot] commented on code in PR #66052: URL: https://github.com/apache/doris/pull/66052#discussion_r3767348977
########## be/src/common/config.cpp: ########## @@ -1309,6 +1329,56 @@ DEFINE_mDouble(inverted_index_ram_buffer_size, "512"); // -1 indicates not working. // Normally we should not change this, it's useful for testing. DEFINE_mInt32(inverted_index_max_buffered_docs, "-1"); +// G16-c: whether plain positions-tier (non-scoring) SNII indexes lay out freq +// regions. Freq bytes serve ONLY BM25 scoring, which the Doris integration +// does not reach yet (scoring_query has no production caller), so the default +// drops them (textbench: -2.2 GB index). Scoring-config indexes always write +// freq regardless. Applies at segment build (write side only); existing +// segments keep whatever layout they were written with (self-describing). +DEFINE_mBool(snii_positions_index_write_freq, "false"); +// G16-h: zstd levels for the SNII dict-block compression and the .prx window +// auto mode. Level 9 (vs the historical 3) shrinks the two largest compressed +// sections -- textbench: index -457 MB (0.918x -> 0.891x V3) -- for an import +// CPU cost inside the run-to-run variance band; zstd decode speed does not +// depend on the level, and warm/cold benches measured no query change. +// Write side only; segments self-describe their compression. +// Default 3 since the all-level-3 evaluation (2026-07-11, 4 corpora): vs +// level 9 the settled index grows only +0.6%..+6.3% (whole table +// +0.3%..+1.9%) while import index CPU drops 17-24% and full-compaction CPU +// 8-24%; settled cold-query latency is unchanged (interleaved A/B). The +// delta+varint-encoded payloads are high-entropy, so level 9's extra search +// buys almost no ratio. Raise only for size-critical deployments. +DEFINE_mInt32(snii_dict_block_zstd_level, "3"); +DEFINE_mInt32(snii_prx_zstd_level, "3"); +// Patch C prx tiering: zstd level for the prx region of DIRECT-LOAD segments +// only (stream/broker load, see IndexColumnWriter::set_direct_load). Inert at +// the defaults (both levels 3); it exists for size-critical deployments that +// RAISE snii_prx_zstd_level (e.g. 9) and still want cheap loads: compaction +// rewrites every segment at snii_prx_zstd_level, so SETTLED data (and the +// cold-query path over it) is unaffected by the load tier -- measured -290s +// (httplogs) / -204s (agentlogs) of import index CPU at 3 vs 9. Same clamp +// [3, 19]. Read at index flush like snii_prx_zstd_level (a mid-load change +// lands on in-flight segments); the direct-load BIT itself is captured once. +DEFINE_mInt32(snii_prx_zstd_level_direct_load, "3"); +// G16-d: target SNII dict block size in bytes; 0 uses the format default +// (64 KiB). Larger blocks compress better under the per-block zstd (the dict +// is the dominant physical section on high-cardinality corpora) at the cost +// of a larger fetch+decompress unit per cold dict-block miss. Write side +// only; the block size is self-described by the on-disk directory. +DEFINE_mInt32(snii_target_dict_block_bytes, "0"); +// Process-wide SNII index-build RAM budget across ALL live segment writers +// (G09); the largest writers are asked to spill early once the sum crosses it. Review Comment: [P1] Enable the documented aggregate memory protection by default. The adjacent contract says the default is 8 GiB, but `0` disables registration entirely. That leaves only per-writer spilling, even though the new limiter's rationale documents the exact 100+ writer case where roughly 41 GiB accumulated while no individual writer crossed its cap. Set the intended 8 GiB default (or correct the contract and rollout claims if disabling is deliberate) and cover production wiring. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
