> We all need to be on the same page, grounded in reality, not fantasy, where if we set a limit of 1024 or 2048, that you can actually index vectors with that many dimensions and it actually works and scales
This is something that I agree with. When we test it, I think we should go in with the right expectations. For example, there's been an ask for use cases. The most immediate use case that I see is LLM memory in the higher range of vectors (up to 2048), rather than strictly search. I'm ok with this use case because of its unprecedented growth. Consider the fact that a lot of users and most of the capital expenditures on Lucene infrastructure goes to use cases that are not strictly application search. Analytics, as we all know, is a very common off-label use of Lucene. Another common use case is index intersection across a variety of text fields. It would be great if we could document a way to up the limit and test using Open AI's embeddings. If there are performance problems they should be understood as performance problems for a particular use case, and we as a community should invest in making it better based on user feedback. This leap of faith could have a very positive impact on all involved. Best, Marcus On Thu, Apr 6, 2023 at 9:30 AM Alessandro Benedetti <a.benede...@sease.io> wrote: > I am not sure I get the point to make the limit configurable: > > 1) if it is configurable, but default max to 1024, it means that we don't > enforce any limit aside the max integer behind the scenes. > So if you want to set a vector dimension for a field to 5000 you need to > first set a MAX compatible and then set the dimension to 5000 for a field. > > 2) if we remove the limit (just an example). The user can directly set the > dimension to 5000 for a field. > > It seems to me that setting the max limit as a configurable constant > brings all the same (negative?) considerations of removing the limit at all > + additional operations needed by the users to achieve the same results. > > I beg your pardon if I an missing something. > > On Thu, 6 Apr 2023, 17:02 Walter Underwood, <wun...@wunderwood.org> wrote: > >> If we find issues with larger limits, maybe have a configurable limit >> like we do for maxBooleanClauses. Maybe somebody wants to run with a 100G >> heap and do one query per second. >> >> Where I work (LexisNexis), we have high-value queries, but just not that >> many of them per second. >> >> wunder >> Walter Underwood >> wun...@wunderwood.org >> http://observer.wunderwood.org/ (my blog) >> >> On Apr 6, 2023, at 8:57 AM, Alessandro Benedetti <a.benede...@sease.io> >> wrote: >> >> To be clear Robert, I agree with you in not bumping it just to 2048 or >> whatever not motivated enough constant. >> >> But I disagree on the performance perspective: >> I mean I am absolutely positive in working to improve the current >> performances, but I think this is disconnected from that limit. >> >> Not all users need billions of vectors, maybe tomorrow a new chip is >> released that speed up the processing 100x or whatever... >> >> The limit as far as I know is not used to initialise or optimise any data >> structure, it's only used to raise an exception. >> >> I don't see a big problem in allowing 10k vectors for example but then >> majority of people won't be able to use such vectors because slow on the >> average computer. >> If we just get 1 new user, it's better than 0. >> Or well, if it's a reputation thing, than It's a completely different >> discussion I guess. >> >> >> On Thu, 6 Apr 2023, 16:47 Robert Muir, <rcm...@gmail.com> wrote: >> >>> Well, I'm asking ppl actually try to test using such high dimensions. >>> Based on my own experience, I consider it unusable. It seems other >>> folks may have run into trouble too. If the project committers can't >>> even really use vectors with such high dimension counts, then its not >>> in an OK state for users, and we shouldn't bump the limit. >>> >>> I'm happy to discuss/compromise etc, but simply bumping the limit >>> without addressing the underlying usability/scalability is a real >>> no-go, it is not really solving anything, nor is it giving users any >>> freedom or allowing them to do something they couldnt do before. >>> Because if it still doesnt work it still doesnt work. >>> >>> We all need to be on the same page, grounded in reality, not fantasy, >>> where if we set a limit of 1024 or 2048, that you can actually index >>> vectors with that many dimensions and it actually works and scales. >>> >>> On Thu, Apr 6, 2023 at 11:38 AM Alessandro Benedetti >>> <a.benede...@sease.io> wrote: >>> > >>> > As I said earlier, a max limit limits usability. >>> > It's not forcing users with small vectors to pay the performance >>> penalty of big vectors, it's literally preventing some users to use >>> Lucene/Solr/Elasticsearch at all. >>> > As far as I know, the max limit is used to raise an exception, it's >>> not used to initialise or optimise data structures (please correct me if >>> I'm wrong). >>> > >>> > Improving the algorithm performance is a separate discussion. >>> > I don't see a correlation with the fact that indexing billions of >>> whatever dimensioned vector is slow with a usability parameter. >>> > >>> > What about potential users that need few high dimensional vectors? >>> > >>> > As I said before, I am a big +1 for NOT just raise it blindly, but I >>> believe we need to remove the limit or size it in a way it's not a problem >>> for both users and internal data structure optimizations, if any. >>> > >>> > >>> > On Wed, 5 Apr 2023, 18:54 Robert Muir, <rcm...@gmail.com> wrote: >>> >> >>> >> I'd ask anyone voting +1 to raise this limit to at least try to index >>> >> a few million vectors with 756 or 1024, which is allowed today. >>> >> >>> >> IMO based on how painful it is, it seems the limit is already too >>> >> high, I realize that will sound controversial but please at least try >>> >> it out! >>> >> >>> >> voting +1 without at least doing this is really the >>> >> "weak/unscientifically minded" approach. >>> >> >>> >> On Wed, Apr 5, 2023 at 12:52 PM Michael Wechner >>> >> <michael.wech...@wyona.com> wrote: >>> >> > >>> >> > Thanks for your feedback! >>> >> > >>> >> > I agree, that it should not crash. >>> >> > >>> >> > So far we did not experience crashes ourselves, but we did not index >>> >> > millions of vectors. >>> >> > >>> >> > I will try to reproduce the crash, maybe this will help us to move >>> forward. >>> >> > >>> >> > Thanks >>> >> > >>> >> > Michael >>> >> > >>> >> > Am 05.04.23 um 18:30 schrieb Dawid Weiss: >>> >> > >> Can you describe your crash in more detail? >>> >> > > I can't. That experiment was a while ago and a quick test to see >>> if I >>> >> > > could index rather large-ish USPTO (patent office) data as >>> vectors. >>> >> > > Couldn't do it then. >>> >> > > >>> >> > >> How much RAM? >>> >> > > My indexing jobs run with rather smallish heaps to give space for >>> I/O >>> >> > > buffers. Think 4-8GB at most. So yes, it could have been the >>> problem. >>> >> > > I recall segment merging grew slower and slower and then simply >>> >> > > crashed. Lucene should work with low heap requirements, even if it >>> >> > > slows down. Throwing ram at the indexing/ segment merging problem >>> >> > > is... I don't know - not elegant? >>> >> > > >>> >> > > Anyway. My main point was to remind folks about how Apache works - >>> >> > > code is merged in when there are no vetoes. If Rob (or anybody >>> else) >>> >> > > remains unconvinced, he or she can block the change. (I didn't >>> invent >>> >> > > those rules). >>> >> > > >>> >> > > D. >>> >> > > >>> >> > > >>> --------------------------------------------------------------------- >>> >> > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>> >> > > For additional commands, e-mail: dev-h...@lucene.apache.org >>> >> > > >>> >> > >>> >> > >>> >> > >>> --------------------------------------------------------------------- >>> >> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>> >> > For additional commands, e-mail: dev-h...@lucene.apache.org >>> >> > >>> >> >>> >> --------------------------------------------------------------------- >>> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>> >> For additional commands, e-mail: dev-h...@lucene.apache.org >>> >> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: dev-h...@lucene.apache.org >>> >>> >> -- Marcus Eagan
