weiqingy opened a new issue, #999: URL: https://github.com/apache/flink-agents/issues/999
### Search before asking - [X] I searched in the [issues](https://github.com/apache/flink-agents/issues) and found nothing similar. ### Description `ElasticsearchVectorStore.queryEmbedding` attaches its filter with `SearchRequest.Builder.postFilter(...)`, while the KNN clause itself carries no `filter` option. Elasticsearch applies `post_filter` after the kNN phase has already selected its `k` nearest hits, so the filter can only remove documents from that set. It never pulls in matching documents that fell outside the top `k`. A caller asking for 5 documents matching `user_id=alice` can get zero back, even with hundreds of alice's documents indexed, if the 5 nearest vectors happen to belong to other users. `get` and `delete` build the same filter map into a real query clause, so they are exhaustive. The three methods diverge, and the divergence is not visible from the API. Both filter forms are affected, since the unified `filters` map and a raw `filter_query` are merged into a single query by `combineQueryJson` before it is attached. Expected: `queryEmbedding` returns up to `k` documents that match the filter. Actual: it returns whichever of the unfiltered `k` nearest happen to match, which can be none. The Elasticsearch Java client exposes `KnnSearch.Builder.filter(...)`, which applies the filter during the kNN search rather than after it. This would be a user-visible behavior change. Callers who pass a filter to `queryEmbedding` today receive at most the matching subset of the top `k`, and would afterwards receive up to `k` matching documents. ### How to reproduce 1. Start a single-node Elasticsearch with `xpack.security.enabled=false`. 2. Index documents belonging to several users, each carrying metadata such as `user_id`. 3. Call `queryEmbedding` with `k` of 5, a filter for one user, and an embedding nearest to another user's documents. 4. Fewer than 5 documents come back, possibly none, even though more than 5 documents match the filter. Note that `ElasticsearchVectorStoreTest` is annotated `@Disabled("Should setup Elasticsearch server.")`, so the existing `testFiltersDsl` coverage does not run in CI and would not catch this. ### Version and environment `main`, elasticsearch-java 8.19.0. Noticed while reviewing #997. ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
