Jackie-Jiang commented on code in PR #18758:
URL: https://github.com/apache/pinot/pull/18758#discussion_r4558702606
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/text/LuceneTextIndexReader.java:
##########
@@ -214,9 +220,24 @@ private File getTextIndexFile(File segmentIndexDir) {
return file;
}
+ @Override
+ public boolean isBuildOnDictionary() {
+ return _buildOnDictionary;
+ }
+
@Override
public ImmutableRoaringBitmap getDictIds(String searchQuery) {
- throw new UnsupportedOperationException("");
+ return getDictIds(searchQuery, null);
+ }
+
+ @Override
+ public ImmutableRoaringBitmap getDictIds(String searchQuery, @Nullable
String optionsString) {
+ if (_buildOnDictionary) {
+ // The translator resolves Lucene document IDs to the original
dictionary IDs, including after merges.
+ return getDocIds(searchQuery, optionsString, null);
Review Comment:
Optional improvement: preserve the existing query-scoped parser cache on the
dictionary path. Passing `null` here means identical queries are parsed again
for each segment, whereas the ordinary TEXT filter passes its query-scoped
cache. A backward-compatible `getDictIds(query, options, cache)` overload could
forward the evaluator factory's existing `QueryContext` and reuse the current
parsing-cache implementation.
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/TextMatchTransformFunction.java:
##########
@@ -106,19 +112,46 @@ public void init(List<TransformFunction> arguments,
Map<String, ColumnContext> c
}
_textIndexReader = indexReader;
+ _columnArg = columnArg;
+ _dictionary = dataSource.getDictionary();
+ _matchingRawValues = null;
}
public int[] transformToIntValuesSV(ValueBlock valueBlock) {
int length = valueBlock.getNumDocs();
initZeroFillingIntValuesSV(length);
- int[] docIds = valueBlock.getDocIds();
- MutableRoaringBitmap indexDocIds = _textIndexReader.isMultiColumn()
- ? ((MultiColumnTextIndexReader) _textIndexReader).getDocIds(_column,
_predicate, _options)
- : _textIndexReader.getDocIds(_predicate, _options);
+ int[] ids;
+ ImmutableRoaringBitmap matchingIds;
+ if (_textIndexReader.isBuildOnDictionary()) {
+ if (_columnArg.getDictionary() == null) {
+ // A standalone dictionary can coexist with a raw forward index.
+ if (_matchingRawValues == null) {
+ matchingIds = _textIndexReader.getDictIds(_predicate, _options);
+ _matchingRawValues = new
ObjectOpenHashSet<>(matchingIds.getCardinality());
+ for (int dictId : matchingIds.toArray()) {
+ _matchingRawValues.add(_dictionary.getStringValue(dictId));
+ }
+ }
+ String[] values = _columnArg.transformToStringValuesSV(valueBlock);
+ for (int i = 0; i < length; i++) {
+ if (_matchingRawValues.contains(values[i])) {
+ _intValuesSV[i] = 1;
+ }
+ }
+ return _intValuesSV;
+ }
+ matchingIds = _textIndexReader.getDictIds(_predicate, _options);
Review Comment:
Suggestion: lazily cache the matching dictionary-ID bitmap per transform
instance and reset it in `init()`. This currently runs a Lucene search and
allocates a bitmap/collector for every value block, although dictionary mode is
immutable and the predicate/options are fixed. The raw-forward branch already
caches its matching strings. This would make a large projection or
`SUM(TEXT_MATCH(...))` search once per segment rather than once per batch.
Please include a projection test exceeding 10,000 rows; the current 1,000-row
tests do not cross a batch boundary.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/TextIndexHandler.java:
##########
@@ -336,7 +336,15 @@ private void
createTextIndexForColumn(SegmentDirectory.Writer segmentWriter, Col
_fieldIndexConfigs.get(columnMetadata.getColumnName()),
columnMetadata);
ForwardIndexReaderContext readerContext =
forwardIndexReader.createContext();
TextIndexCreator textIndexCreator =
StandardIndexes.text().createIndexCreator(context, config)) {
- if (columnMetadata.isSingleValue()) {
+ if (config.isBuildOnDictionary()) {
Review Comment:
Suggestion: branch on dictionary mode before `createForwardIndexIfNeeded()`
and before opening the forward reader/context. This branch only reads
dictionary values, so it does not need a forward index. For a forward-disabled
column with an inverted index, the current setup reconstructs, serializes, and
later deletes an entire temporary forward index unnecessarily. Dictionary mode
can own just the dictionary and text creator, while marker handling, sealing,
and packing remain shared. A forward-disabled reload case would cover this.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/text/LuceneTextIndexCreator.java:
##########
@@ -153,7 +160,10 @@ public LuceneTextIndexCreator(String column, File
segmentIndexDir, boolean commi
// 1. Not the realtime index, i.e. commit is set to true
// 2. Happens during realtime segment conversion
// 3. Mutable segment not compacted
- _reuseMutableIndex = config.isReuseMutableIndex() && commit &&
realtimeConversion && !mutableSegmentCompacted;
+ // 4. Not a dictionary-based index: the mutable index is per-row, so a
dictionary-based immutable index must be
+ // rebuilt from the sealed dictionary rather than reusing the mutable
index.
+ _reuseMutableIndex = config.isReuseMutableIndex() && commit &&
realtimeConversion && !mutableSegmentCompacted
+ && !_buildOnDictionary;
Review Comment:
[P1] Preserve original text before disabling mutable-index reuse
With `buildOnDictionary=true`, `reuseMutableIndex=true`, and
`indexes.text.rawValue` configured (or legacy `noRawDataForTextIndex=true`),
`MutableSegmentImpl` still installs `SameValueMutableDictionary`, which retains
only the placeholder. Disabling reuse here makes `TextIndexType` rebuild the
text index from that placeholder, losing the original searchable terms. For
input `alpha`/`beta` and replacement `n`, `TEXT_MATCH(column, 'alpha')` matches
while consuming but stops matching after sealing. The new conversion test
enables reuse but omits replacement values.
Please reject this incompatible configuration or preserve original
dictionary values during ingestion, and add the combination to the conversion
regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]