mikemccand commented on PR #16030: URL: https://github.com/apache/lucene/pull/16030#issuecomment-5282246293
+1, I love that we are making progress on data-blind quantization. This should work well when incoming vectors are isotropic (all dimensions behave the same i.e. the histograms of their per-dimension values approximate little baby gaussians with mean 0) and variance as required to be on unit sphere at that dimensionality. But I think your average trained model won't just produce isotropic embeddings? For such cases (probably the common case? not sure), we have pre-conditioning / random Hadamard rotation (another PR in flight for this? -- yes #16092!) which should (usually? there are adversaries (intentional or otherwise) for any rotation matrix right?) scrub anisotropic vectors. Sort of like the [record and record players in GEB](https://en.wikipedia.org/wiki/G%C3%B6del,_Escher,_Bach) -- thank you [Kurt Gödel!](https://en.wikipedia.org/wiki/Kurt_G%C3%B6del)). These are all experimental vector codecs but I hope they eventually become default. If we always pre-condition then we almost always can do data blind quantization that is just as good as non-data-blind quantization. Does this PR make any effort / at least javadocs to explain that you should ensure your incoming vectors are isotropic? Or to spot check if they really seem to be isotropic? luceneutil has all sorts of smell detection ("smelling pipeline" a recent genai model called it!) now to detect all sorts of problems your otherwise very-opaque-to-humans vectors might have. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
