Jackie-Jiang commented on code in PR #18758:
URL: https://github.com/apache/pinot/pull/18758#discussion_r4595739608


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/text/LuceneTextIndexReader.java:
##########
@@ -214,9 +220,30 @@ private File getTextIndexFile(File segmentIndexDir) {
     return file;
   }
 
+  @Override
+  public boolean isBuildOnDictionary() {
+    return _buildOnDictionary;
+  }
+
   @Override
   public ImmutableRoaringBitmap getDictIds(String searchQuery) {
-    throw new UnsupportedOperationException("");
+    return getDictIds(searchQuery, null);
+  }
+
+  @Override
+  public ImmutableRoaringBitmap getDictIds(String searchQuery, @Nullable 
String optionsString) {
+    return getDictIds(searchQuery, optionsString, null);
+  }
+
+  @Override
+  public ImmutableRoaringBitmap getDictIds(String searchQuery, @Nullable 
String optionsString,
+      @Nullable QueryScopedCache queryScopedCache) {
+    if (_buildOnDictionary) {
+      // The translator resolves Lucene document IDs to the original 
dictionary IDs, including after merges.
+      return getDocIds(searchQuery, optionsString, queryScopedCache);

Review Comment:
   [P3] Make the reader's ID contract explicit
   
   In dictionary mode, public `getDocIds()` returns dictionary IDs and 
`getNumMatchingDocs()` counts distinct values. Current planner callers dispatch 
correctly, but the API contract is misleading. Extract the shared search into a 
private helper and reject row-ID/count operations in dictionary mode, following 
the existing FST reader pattern.



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/TextMatchTransformFunction.java:
##########
@@ -106,19 +114,50 @@ public void init(List<TransformFunction> arguments, 
Map<String, ColumnContext> c
     }
 
     _textIndexReader = indexReader;
+    _columnArg = columnArg;
+    _dictionary = dataSource.getDictionary();
+    _matchingRawValues = null;
+    _matchingDictIds = null;
   }
 
   public int[] transformToIntValuesSV(ValueBlock valueBlock) {
     int length = valueBlock.getNumDocs();
     initZeroFillingIntValuesSV(length);
 
-    int[] docIds = valueBlock.getDocIds();
-    MutableRoaringBitmap indexDocIds = _textIndexReader.isMultiColumn()
-        ? ((MultiColumnTextIndexReader) _textIndexReader).getDocIds(_column, 
_predicate, _options)
-        : _textIndexReader.getDocIds(_predicate, _options);
+    int[] ids;
+    ImmutableRoaringBitmap matchingIds;
+    if (_textIndexReader.isBuildOnDictionary()) {
+      if (_matchingDictIds == null) {
+        _matchingDictIds = _textIndexReader.getDictIds(_predicate, _options);
+      }
+      matchingIds = _matchingDictIds;
+      if (_columnArg.getDictionary() == null) {
+        // A standalone dictionary can coexist with a raw forward index.
+        if (_matchingRawValues == null) {
+          _matchingRawValues = new 
ObjectOpenHashSet<>(matchingIds.getCardinality());

Review Comment:
   [P3] Short-circuit constant projection results
   
   When no dictionary values match, return the initialized zero array. When 
every dictionary value matches, fill the active output with ones. This avoids 
reading forward values for every block and avoids constructing the entire 
matching-string set for RAW match-all queries. Existing null propagation 
remains separate; the predicate evaluator already uses these shortcuts.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/TextIndexHandler.java:
##########
@@ -201,12 +211,40 @@ private Set<String> 
getExistingColumns(SegmentDirectory.Reader segmentReader) {
   }
 
   private void validateRecoverySource(SegmentDirectory.Reader segmentReader, 
String column) {
-    if (!segmentReader.hasIndexFor(column, StandardIndexes.forward())) {
+    if 
(_fieldIndexConfigs.get(column).getConfig(StandardIndexes.text()).isBuildOnDictionary())
 {
+      if (!segmentReader.hasIndexFor(column, StandardIndexes.dictionary())) {
+        throw new IllegalStateException("Cannot rebuild Lucene text index for 
column: " + column
+            + ". Original dictionary must be retained; restore a compatible 
segment or refresh it from source data");

Review Comment:
   [P2] Accept raw values while dictionary creation is pending
   
   Recovery validation runs before `ForwardIndexHandler` creates the 
dictionary. Migrating a single-value RAW column to dictionary text mode 
therefore fails when its text index is incompatible or has an interruption 
marker, despite an intact raw forward index containing the original values. 
Allow that recoverable source when dictionary creation is pending, and extend a 
RAW rollback test with `buildOnDictionary=true`.



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/TextMatchTransformFunction.java:
##########
@@ -106,19 +114,50 @@ public void init(List<TransformFunction> arguments, 
Map<String, ColumnContext> c
     }
 
     _textIndexReader = indexReader;
+    _columnArg = columnArg;
+    _dictionary = dataSource.getDictionary();
+    _matchingRawValues = null;
+    _matchingDictIds = null;
   }
 
   public int[] transformToIntValuesSV(ValueBlock valueBlock) {
     int length = valueBlock.getNumDocs();
     initZeroFillingIntValuesSV(length);
 
-    int[] docIds = valueBlock.getDocIds();
-    MutableRoaringBitmap indexDocIds = _textIndexReader.isMultiColumn()
-        ? ((MultiColumnTextIndexReader) _textIndexReader).getDocIds(_column, 
_predicate, _options)
-        : _textIndexReader.getDocIds(_predicate, _options);
+    int[] ids;
+    ImmutableRoaringBitmap matchingIds;
+    if (_textIndexReader.isBuildOnDictionary()) {
+      if (_matchingDictIds == null) {
+        _matchingDictIds = _textIndexReader.getDictIds(_predicate, _options);

Review Comment:
   [P3] Share parsed queries in projections too
   
   This two-argument call bypasses the query-scoped parser cache, so 
`SELECT`/`SUM(TEXT_MATCH(...))` still parses once per segment. Capture the 
cache through the existing `QueryContext` initialization overload and pass it 
to the three-argument `getDictIds()`, as filtering already does. Keep matching 
bitmaps local to each transform and reset the cache reference on 
reinitialization.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to