mikemccand commented on PR #15979: URL: https://github.com/apache/lucene/pull/15979#issuecomment-5104656163
Maybe we should finish a data blind (I can't help but be reminded of "blind dates", something totally different!!) quantization before trying to get dedup working with quantization? I.e. always rotate indexed and searched vectors by "deterministic random" Hadamard matrix (https://github.com/apache/lucene/pull/16092, also implemented in luceneutil now). After that we should implement data blind quantization (eventually making it default vector format if it performs well) which doesn't look at the incoming vectors to compute quantization, and then (I think?) wouldn't need per-vector correction factors. At that point it's just the quantized bits stored for each vector, and then dedup can just key off of that instead of the input full precision vectors? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
