xiangfu0 commented on code in PR #18758:
URL: https://github.com/apache/pinot/pull/18758#discussion_r4596251777


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/TextIndexHandler.java:
##########
@@ -201,12 +211,40 @@ private Set<String> 
getExistingColumns(SegmentDirectory.Reader segmentReader) {
   }
 
   private void validateRecoverySource(SegmentDirectory.Reader segmentReader, 
String column) {
-    if (!segmentReader.hasIndexFor(column, StandardIndexes.forward())) {
+    if 
(_fieldIndexConfigs.get(column).getConfig(StandardIndexes.text()).isBuildOnDictionary())
 {
+      if (!segmentReader.hasIndexFor(column, StandardIndexes.dictionary())) {
+        throw new IllegalStateException("Cannot rebuild Lucene text index for 
column: " + column
+            + ". Original dictionary must be retained; restore a compatible 
segment or refresh it from source data");

Review Comment:
   Fixed in `c11281aab6`. Preflight accepts a retained RAW forward index for an 
eligible single-value STRING column when the forward index handler can create 
the dictionary. The rebuild still checks for the dictionary and guards against 
legacy realtime placeholders before replacing the text index.
   
   Both new rollback cases failed before the fix and passed on JDK 25. They 
cover an incompatible Lucene index and an interruption marker, then verify 
dictionary matches and a stable reload. All 21 rollback and recovery cases 
passed.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/text/LuceneTextIndexReader.java:
##########
@@ -214,9 +220,30 @@ private File getTextIndexFile(File segmentIndexDir) {
     return file;
   }
 
+  @Override
+  public boolean isBuildOnDictionary() {
+    return _buildOnDictionary;
+  }
+
   @Override
   public ImmutableRoaringBitmap getDictIds(String searchQuery) {
-    throw new UnsupportedOperationException("");
+    return getDictIds(searchQuery, null);
+  }
+
+  @Override
+  public ImmutableRoaringBitmap getDictIds(String searchQuery, @Nullable 
String optionsString) {
+    return getDictIds(searchQuery, optionsString, null);
+  }
+
+  @Override
+  public ImmutableRoaringBitmap getDictIds(String searchQuery, @Nullable 
String optionsString,
+      @Nullable QueryScopedCache queryScopedCache) {
+    if (_buildOnDictionary) {
+      // The translator resolves Lucene document IDs to the original 
dictionary IDs, including after merges.
+      return getDocIds(searchQuery, optionsString, queryScopedCache);

Review Comment:
   Fixed in `c11281aab6`. `getDocIds()` and `getNumMatchingDocs()` now reject 
dictionary mode. Both ID APIs use the same private search helper, and 
`getDictIds()` still translates Lucene IDs to dictionary IDs. The SPI docs now 
distinguish row IDs and counts from dictionary IDs.
   
   All four `testDictionaryMappingAfterMerge` cases failed on the prior code 
and passed with the fix on JDK 25, covering directory and packed formats. The 
39 row count and eight query cache cases passed as well.



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/TextMatchTransformFunction.java:
##########
@@ -106,19 +114,50 @@ public void init(List<TransformFunction> arguments, 
Map<String, ColumnContext> c
     }
 
     _textIndexReader = indexReader;
+    _columnArg = columnArg;
+    _dictionary = dataSource.getDictionary();
+    _matchingRawValues = null;
+    _matchingDictIds = null;
   }
 
   public int[] transformToIntValuesSV(ValueBlock valueBlock) {
     int length = valueBlock.getNumDocs();
     initZeroFillingIntValuesSV(length);
 
-    int[] docIds = valueBlock.getDocIds();
-    MutableRoaringBitmap indexDocIds = _textIndexReader.isMultiColumn()
-        ? ((MultiColumnTextIndexReader) _textIndexReader).getDocIds(_column, 
_predicate, _options)
-        : _textIndexReader.getDocIds(_predicate, _options);
+    int[] ids;
+    ImmutableRoaringBitmap matchingIds;
+    if (_textIndexReader.isBuildOnDictionary()) {
+      if (_matchingDictIds == null) {
+        _matchingDictIds = _textIndexReader.getDictIds(_predicate, _options);
+      }
+      matchingIds = _matchingDictIds;
+      if (_columnArg.getDictionary() == null) {
+        // A standalone dictionary can coexist with a raw forward index.
+        if (_matchingRawValues == null) {
+          _matchingRawValues = new 
ObjectOpenHashSet<>(matchingIds.getCardinality());

Review Comment:
   Fixed in `c11281aab6`. Dictionary projections fill the active output with 
zeros or ones for no matches or all matches. Both cases skip forward reads and 
avoid building a set of raw strings. Null handling remains in the existing null 
path.
   
   `TextMatchTransformFunctionTest` verifies both forward representations and 
reuse of the output buffer. `DictionaryBasedTextIndexQueriesTest` verifies 
nullable SQL projections. All 23 cases across the two classes passed on JDK 25.



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/TextMatchTransformFunction.java:
##########
@@ -106,19 +114,50 @@ public void init(List<TransformFunction> arguments, 
Map<String, ColumnContext> c
     }
 
     _textIndexReader = indexReader;
+    _columnArg = columnArg;
+    _dictionary = dataSource.getDictionary();
+    _matchingRawValues = null;
+    _matchingDictIds = null;
   }
 
   public int[] transformToIntValuesSV(ValueBlock valueBlock) {
     int length = valueBlock.getNumDocs();
     initZeroFillingIntValuesSV(length);
 
-    int[] docIds = valueBlock.getDocIds();
-    MutableRoaringBitmap indexDocIds = _textIndexReader.isMultiColumn()
-        ? ((MultiColumnTextIndexReader) _textIndexReader).getDocIds(_column, 
_predicate, _options)
-        : _textIndexReader.getDocIds(_predicate, _options);
+    int[] ids;
+    ImmutableRoaringBitmap matchingIds;
+    if (_textIndexReader.isBuildOnDictionary()) {
+      if (_matchingDictIds == null) {
+        _matchingDictIds = _textIndexReader.getDictIds(_predicate, _options);

Review Comment:
   Fixed in `c11281aab6`. The transform now passes the query-scoped parser 
cache to `getDictIds()`. The match bitmap belongs to the transform, and 
initialization clears it and resets the cache reference.
   
   `TextMatchTransformFunctionTest` verifies that subsequent blocks reuse local 
matches and that reinitialization clears the previous query's cache and 
matches. All 10 cases passed on JDK 25.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to