xiangfu0 commented on PR #19336: URL: https://github.com/apache/pinot/pull/19336#issuecomment-5434985328
Superseded by #19303, closing. #19303 now covers the same capability — making the mutable HNSW index filter-aware so consuming segments of upsert tables use filtered ANN instead of the exact-scan fallback — rebased onto master after #19297–#19301 landed, so it no longer carries the base-fix half this PR was stacked on. The two ideas from here that were worth keeping are folded in: - a single doc-id mechanism (numeric doc values drive both filtered traversal and hit translation, dropping the redundant stored field per document); - per-instance index directories, so two replicas of one segment in the same JVM cannot collide on the Lucene write lock — kept segment and column in the path so a leaked directory is still attributable. Two things were deliberately not carried over. This branch reimplements the Lucene filter iterator privately, while `HnswVectorIndexReader` already has one; #19303 extracts a shared base class instead, so there is one implementation rather than three. And this branch opens a fresh `DirectoryReader` from the writer on every search; #19303 refreshes a shared `SearcherManager` and only on the filtered path, leaving the unfiltered path on the cheaper last-committed view. Also noted for whoever revisits this area: the `org.jetbrains:annotations` dependency added to `pinot-segment-local/pom.xml` here is unused in that module. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
