Bram Van Dam [bram.van...@intix.eu] wrote: [Solr cursors]
> Oh thanks, that's a pretty interesting read. The scale we're > investigating is several orders of magnitude larger than what was tested > there, so I'm still a bit worried. The beauty of the cursor is that it is has little to no overhead, relative to a standard top-X sorted search. A standard search uses a sliding window over the full result set, as does a cursor-search. Same amount of work. It is just a question of limits for the window. > The largest index I currently have access to is > about a billion documents in size. Paging there is a nightmare, but the > Solr version is too old to support cursors so I'm afraid I can't offer > any useful data. Non-cursor paging in Solr uses a sliding window sort with a heap that contains all documents up to the paging number. A heap is a very fine thing for sliding window sort, as long as it is small. But performance drops to horrible levels when it gets large as it is extremely RAM-cache unfriendly. > Does anyone have any performance data on multi-billion-document indexes? Sorry, no. I could do a test on our 7 billion documents index, but it would have to wait until the end of January. >Nobody will hit next 499 times, but a lot of our users skip to the last > page quite often. Maybe I should make *that* as hard as possible. Hmm. Issue a search with sort in reverse order, then reverse the returned list of documents? - Toke Eskildsen
