CrownChu opened a new pull request, #10771:
URL: https://github.com/apache/paimon/pull/10771
### Purpose
Allow ordinary scalar predicates to use **grouped indexes**, where the
indexed fields can be queried independently, without treating them as ordered
composite keys.
A grouped ES index such as `(embedding, category, price)` can answer a
predicate on `category` or `price` independently. Currently the single-field
scalar paths exclude this definition altogether, so a normal filtered scan can
silently fall back to reading data without opening the ES index. Metadata-only
coverage also cannot establish that every shard accepted the predicate.
This change:
- Adds `GlobalIndexerFactory.supportsGroupedIndexScalarQueries()` as an
opt-in capability, defaulting to `false`. Only the ES factory opts in; there is
no ES type-name special case in the core planner.
- Allows eligible grouped definitions in eager and reader-side scalar
queries, including search pre-filters.
- Tracks per-field metadata coverage for eligible definitions and uses
**evaluated coverage** when a reader declines a predicate, preserving uncovered
ranges in FULL/DETAIL mode.
- Makes vector/full-text scalar pre-filters consume evaluated coverage
rather than treating declined shards as empty matches.
- Keeps BTree composite-key planning and vector-primary-column selection
unchanged. This does not add an `es_search()` SQL API.
The additional ESLib module dependencies are test-scoped only.
### Tests
Added 23 test invocations:
- 16 real ES archive/table integration cases: scalar
equality/ranges/IN/NULL/AND/OR, FAST/FULL/DETAIL, eager and reader-side
execution, serialized splits, partial coverage and different definitions, and
unchanged vector-primary selection. Assertions compare against index-disabled
reads and verify that the ES archive was actually opened.
- 2 grouped scalar query cases checking that unsupported shards do not claim
evaluated coverage.
- 5 vector/full-text scalar pre-filter cases checking fallback before top-k
ranking.
Local validation on upstream base `b21f36cc56ee6bd11f43b7a2d7f43695445cd446`
uses JDK 11 with Checkstyle, Spotless and Enforcer enabled. The targeted
reactor command is:
```sh
mvn -B -ntp -Ppaimon-eslib -pl paimon-eslib -am \
-DwildcardSuites=none -DfailIfNoTests=false \
'-Dtest=ESIndex*Test,GlobalIndexQueryTest,GlobalIndexEvaluatorTest,CompositeBTreeTableTest,IndexQuerySplitTest,DataEvolutionBatchScanTest,FullTextSearchBuilderTest,VectorSearchBuilderTest,VectorSearchRowFilterExactnessTest'
\
package
```
`package` is used for a fresh worktree so the codegen-loader embedded
resources are prepared before the dependent tests run.
Result on upstream `b21f36cc56ee6bd11f43b7a2d7f43695445cd446`: **380 tests
passed, 0 failures, 0 errors, 0 skipped** (38 common + 252 core + 90 ESLib).
This is a targeted regression run, not the entire repository test suite.
### CI status
- The [fork CI run for this
commit](https://github.com/CrownChu/paimon/actions/runs/39207107696) passed all
Java jobs, including ESLib/JDK 11, Core/JDK 8 and 11, Spark, Flink and Java E2E
jobs.
- The full workflow is **not green**: `Python / Native CI` failed. The [same
job on the exact upstream
parent](https://github.com/apache/paimon/actions/runs/39183016486/job/117839677706),
without this change, reported **67 failed, 19154 passed, 131 skipped**. The
Python source tree and Python CI workflow are byte-identical between this
commit and its parent.
- The 52 failure markers visible in the fork log have the same per-file
progress sequences as the parent run. One confirmed baseline issue is that the
native-commit tracking wrapper in `paimon-python/conftest.py` does not accept
the new `batch` keyword passed by truncate operations. Other baseline failures
include native reader/test-fixture incompatibilities and update-order
assertions.
- These are historical CI comparisons, not a controlled rerun of the
complete Native suite: both jobs build the moving `paimon-rust` main branch and
resolved different Rust commits. The fork Native job ended before a final
pytest summary; its early termination is not yet explained. This PR does not
change Python/Rust code or suppress failing tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
