krickert commented on PR #15676: URL: https://github.com/apache/lucene/pull/15676#issuecomment-3929870490
So I already knew that heavy ANN sharding hurts recall fast, but made the mistake this round to think that would be good enough so as long as I don't make it worse. However - I want to try to attack the terrible recall over many shards whereas my focus before was on performance. I want to do another pass focused on adding global context at index time. I assumed that if I relied on base Lucene ANN behavior and avoided regressions, I would still be in a good place. But at 16 shards, around 15 to 18 percent recall is not acceptable. Right now, with shard-local indexing only, search-time collaboration alone does not reliably move us past the baseline ANN frontier. I think search-time collaboration is close to maxed out on shard-local ANN, so the next step is index-time shard awareness. I want to try a lightweight global ANN routing layer with compact cross-shard neighborhood metadata, which should be practical to distribute over gRPC, so shards can navigate the global manifold instead of only their local graph. I found recent probabilistic routing work from 2024 that looks closely related, and I want to apply the same idea at shard granularity by adding index-time global routing metadata so distributed ANN can avoid shard-blind traversal. The plan is to keep local HNSW local, but add a global routing spine that gives each shard better context before deep traversal. At high shard counts, both baseline and collaborative quality drop significantly in this setup. This attempt did not deliver the win I wanted yet, but it did clarify a concrete path to improve it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
