Hi Mike, I'm looking forward to that vector search is available at 9.0, thanks for your hard work on it.
> Alternatively we could simply expect users to > perform such normalization, and throw an error if vectors intended for > comparison using dot product (which is specified when adding a value) > are not unit-length. For simplicity, we could assume normalized vectors as inputs and just document it - without any checks? Meanwhile some utility functions (e.g., o.a.l.util.VectorUtil) for it could be helpful. Tomoko 2020年11月26日(木) 6:23 Michael Sokolov <[email protected]>: > I have been working on getting benchmarks working on the GloVe public > data set and spent a while chasing down a bug with VectorValues.search > that turned out to be a bug with the data (sort of)! When comparing > vectors using an angular (dot product) measure, one has to normalize > by the vectors' lengths. Given that the only purpose of such vectors > is to compare them using dot-product, it would be sensible to > normalize them *in advance* to unit length, rather than doing so for > every comparison, yet this is not how this dataset at least is > distributed on the internet, and widely-referenced benchmarking > software such as ann-benchmarks assumes that code will handle such > details internally. > > I'm trying to see how we should handle this use case. We could provide > a convenience function for normalizing while indexing. But should we? > Would it happen when creating an IndexableField? When flushing? It's a > little strange if you index a vector, and then retrieve it and its > value is different! Alternatively we could simply expect users to > perform such normalization, and throw an error if vectors intended for > comparison using dot product (which is specified when adding a value) > are not unit-length. But then again this is a somewhat costly > operation that is only a safety measure, and users who already > normalized their vectors would pay the cost needlessly. > > For now, I'm doing nothing, but I wonder if we could offer users some help > here. > > -Mike > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
