mikemccand commented on PR #16092: URL: https://github.com/apache/lucene/pull/16092#issuecomment-5104592060
Note: luceneutil PR is merged, so benchmarks can precondition vectors (a way to pre-test before this PR is merged). @shubhamvishu is there a separate issue open to switch to data-blind quantization? I.e. we don't need to scan the incoming vectors to compute their distributions. Seems like it's just our current vectors format, but hardwired to what the math (law of large numbers) says all corpora must converge to, e.g. all centroids are 0 (because preconditioning should create isotropic vectors, so all dimensions look like perfect little gaussians with mean 0.0 and variance derived from dimension count), merging can just copy already quantized bytes, etc. Or is there another vector format that's already in progress / doing this ...? Hard to keep up! Separately, is there another issue open to encode residuals instead of the original full precision vectors? I.e. if we do the data-blind quantization (4 bit, 8 bit, etc.), we can subtract that quantized value from the original, leaving a residual vector (much smaller length, not unit sphere normalized by itself) which we could use fewer bits to encode fairly accurately. It's wasteful to store the full precision vectors, disregarding that you also have the quantized version available ... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
