morningman opened a new pull request, #70322: URL: https://github.com/apache/doris/pull/70322
### Background SNII is an inverted index storage format whose writer collects postings in memory in a `SpimiTermBuffer`, which charges its resident bytes to a `MemoryReporter`. Under the spill-threshold policy that index ingestion uses, the reporter's cap only triggers spills of the posting arena and never fails an allocation, because the persistent vocabulary (term strings, the intern set, the slot index) may exceed the cap by design. Two cases in `snii_streamed_session_test.cpp` check that finalizing the index and draining the buffer work, and are accounted exactly, while the vocabulary alone is above the cap. ### What problem does this PR solve? Issue Number: None Related PR: #66052 (added the two cases) PR Type: Test Problem Summary: `FinalizesAfterPersistentVocabularyCrossesSpillThreshold` and `TerminalDrainReleasesLookupStateWithoutUnderAccountingIds` fail on every run of a BE unit test build that uses libc++, such as the macOS build, before they reach what they test: their precondition that the vocabulary is above the 384 KiB cap does not hold. They insert a fixed 3000 terms, which is enough only with libstdc++, because the buffer charges each owned term string by `sizeof(std::string)`: 32 bytes with libstdc++, 24 with libc++. This PR raises the count to 4000, which keeps the vocabulary above the cap with both libraries without letting the in-memory case spill. The accounting is right on both libraries, so production code is unchanged. ### Details - **Failure:** With libc++, `ASSERT_GT(reporter.current_bytes(), kReporterCap)` fails in the first case (`actual: 371332 vs 393216`) and `ASSERT_GT(before_drain, kReporterCap)` in the second (`actual: 371188 vs 393216`), on every run, in the iteration that forces a spill. The BE UT pipeline builds with libstdc++ on Linux, and `be-ut-mac.yml` builds the BE on macOS without building or running the unit tests, so no pipeline has seen it since #66052. - **Cause:** The buffer charges its owned vocabulary as `owned_vocab_.capacity() * sizeof(std::string)` plus heap payloads, which these short terms don't have because they fit the small-string buffer. The capacity grows by explicit doubling and is 4096 for 3000 terms with either library, so libc++ accounts 8 × 4096 = 32,768 bytes less for the same input. After the forced spill frees the posting arena, 371,332 bytes are left with libc++, under the 393,216-byte cap; libstdc++ accounts 404,100 for the same input. The iteration that does not spill keeps its arena and is above the cap with both. - **Stability evidence:** The failure is deterministic, not timing related. With 4000 terms both cases pass with libc++, where the forced iteration accounts 421,332 bytes; libstdc++ accounts 32,768 bytes more for the same input. The posting arena, which decides whether the in-memory iteration spills, does not depend on the library: it reaches 64 KiB, below the 96 KiB (cap / 4) that a spill needs, so that iteration still never spills. ### Testing - `run-be-ut.sh --run --filter='SniiStreamedWriterSessionTest.*'`, locally on macOS arm64 with ASAN (libc++): all 41 cases pass with this change, and the two cases fail without it. - Two mutations of `spimi_term_buffer.cpp`, run locally against the new count: raising MEM_LIMIT_EXCEEDED once the reporter is over the cap, which treats the spill threshold as an allocation limit, fails both cases; keeping the intern set and the string ranks through the terminal drain fails the second case at `during_drain < before_drain` (519,296 vs 503,296), and leaves 208,000 bytes charged at the end of both. - These ran on `42cf4e3e5d1` plus this change; the SNII sources and tests are the same on `bb3156d30e6`, where this PR is based. The libstdc++ run is left to the BE UT pipeline. ### Notes for Reviewers - Why 4000: the count must stay at or below 4096, where the vector capacities, and with them the margins above, stay as measured. 3500 terms would leave only about 3 KB above the cap with libc++ (396,332 bytes). ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Partially verified: the libstdc++ (Linux) run is left to the BE UT pipeline ### AI Disclosure - Tool: Claude Code (Claude Opus 5.5) - Generated: the test change and this description - Human review: not stated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
