arunkumarucet opened a new pull request, #19760: URL: https://github.com/apache/pinot/pull/19760
## Problem A `count(*)` with a `TEXT_MATCH` filter only needs the size of the match set, but `TextMatchFilterOperator#getNumMatchingDocs` builds a full bitmap of every matching document and then reads its cardinality: ```java return _textIndexReader.getDocIds(_predicate.getValue(), _predicate.getOptions()).getCardinality(); ``` The cost of that grows with the number of matches, so it dominates a count over a high-frequency term on a large segment. Nothing survives the bitmap except a single integer. The fast-count path already exists and is already wired up: `AggregationPlanNode` routes to `FastFilteredCountOperator` when a filter reports `canOptimizeCount()`, and `TextMatchFilterOperator` already returns `true`. Only the underlying implementation still materialized the bitmap. ## Change - `TextIndexReader#getNumMatchingDocs(searchQuery, optionsString)`, defaulting to the existing `getDocIds(...).getCardinality()` so every other implementation is unaffected. - `LuceneTextIndexReader` implements it with `Weight#count` per leaf. Lucene answers a single-term query from index metadata without visiting documents. Where a leaf cannot be counted cheaply it returns `-1`, and we fall back to the existing path rather than letting Lucene iterate internally, so the collector's query-termination check is preserved and a slow count still honours the query timeout. - `TextMatchFilterOperator` asks for the count instead of the match set. - Query parsing moves into one shared method, so the count path and the doc-id path cannot interpret the same query differently. Otherwise `count(*)` could disagree with the rows its own filter returns. ## Measurements On a 1B-row OpenTelemetry log corpus (single node, 232 segments, Lucene text index on the log body), counting a high-frequency term took 183 ms, against 5 ms for an engine that reads the count from posting-list metadata. The gap tracked match count across queries: | Query | Filter | Matches | Before | |---|---|---|---| | `count(*) WHERE TEXT_MATCH(Body, 'error')` | single term, high frequency | ~100M | 36.6x slower | | `count(*) WHERE TEXT_MATCH(Body, 'failed AND order')` | conjunction | mid | 13.4x | | `count(*) WHERE TEXT_MATCH(Body, 'payment')` | single term, low frequency | ~1.7M | 8.8x | | 8-term conjunction | conjunction, low frequency | low | 3.8x | The effect is not visible at small scale: on the same workload at 1M rows, materializing the bitmap is cheap enough that the difference disappears. That is likely why it has gone unnoticed. ## Tests `LuceneTextIndexCountTest` pins the invariant that the short-circuit must return exactly what counting the materialized doc ids returns, across 16 query shapes: single terms (including one matching no documents and one matching every document), `AND`/`OR`/`NOT`, phrases, prefix, wildcard, fuzzy, regexp, and option-driven parsing (`parser=MATCH,minimumShouldMatch=2`, `allowLeadingWildcard=true`). This covers both the metadata path and the fallback. All 61 existing text-index tests in `pinot-segment-local` pass. ## Follow-up Multi-column text indexes keep the existing behaviour; `MultiColumnTextIndexReader` takes a different signature and is marked with a TODO. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
