Hi Erick,
Here's some more details about our structure. First here's an example of
document in our index :
PrimaryKey = SJAsfsf353JHGada66GH6 (it's a hash)
DocType = X
Data = This is the data
SearchableContent = This is the data
DateCreated = <timestamp>
DateModified = <timestamp>
Counter1 = 17
Counter2 = 3
Average = 0.17
Cost = 200
The users are able to sort on almost all fields: Data, DateCreated,
DateModified, Counter1, Counter2, Average, Cost.
When we search, we always search on the 'SearchableContent' field and we
have at least one filter on the DocType (because we have many document types
in the same index). So a common search that would find the document above is
"data *AND DocType:X*" (we automatically add the "*AND DocType:X*" part
using Lucene Filters.
I would say that the number of unique terms in the field being sorted on is
very big - for example timestamps, almost all unique, counters, average,
cost, data... so if a query finds 10M results, it's almost 10M different
values to sort. About cache and warm-up queries : we don't use warm-up
queries -at all- because we have absolutely no idea of what users are going
to search for (they can search for absolutely anything). About "returning
10M" documents, right, we don't actually return the 10M documents, we use
pagination to return documents X to Y of the 10M (and the 10M was only an
example, it can be anywhere between 1K and 100M results). The pagination
usually works fine and fast, our problem is really sorting.
Our "Lucene Reader" process has 2GB of ram allowed, here's how I start it -
java -Xmx2048m -jar LuceneReader.jar
The problem really seems to be a ram problem, but I can't be 100% sure (any
help about how to be sure is welcome).
Our current idea of a solution would be to get maximum 250 results (the more
relevant ones; more results than that is totally useless in our system) so
the sort should work fine on a small data set like that, but we want to make
sure we're doing everything right before doing that so we don't run in the
same problems again.
Thank you very much; let me know if you need any more details!
- Mike
aka...@gmail.com
On Mon, Aug 16, 2010 at 4:01 PM, Erick Erickson <erickerick...@gmail.com>wrote:
> Let's back up a minute. The number of matched records is not
> important when sorting, what's important is the number of unique
> terms in the field being sorted. Do you have any figures on that? One
> very common sorting issue is sorting on very fine date time resolutions,
> although your examples don't include that...
>
> Now, cache loading is an issue. The very first time you sort on a field,
> all the values are read into a cache. Is it possible this is the source
> of your problems? You can cure this with warmup queries. The take-away
> is that measuring the response time for the first sorted query is
> very misleading.
>
> Although if you're sorting on many, many, many email addresses,
> that could be "interesting".
>
> The comment "returning 10,000,000 documents" is, I hope, a
> misstatement. If you're trying to *return* 10M docs sorting
> is irrelevant compared to assembling that many documents. If
> you're trying to return the first 100 of 10M documents, it should
> work.
>
> Overall, we need more details on what you're sorting and what
> you're trying to return as well as how you're measuring before
> we can say much....
>
> Along with how much memory you're giving your JVM to work with,
> what "exploding" means. Are you CPU bound? IO bound? Swapping?
> You need to characterize what is going wrong before worrying about
> solutions......
>
> Best
> Erick
>
> On Mon, Aug 16, 2010 at 3:08 PM, Michel Nadeau <aka...@gmail.com> wrote:
>
> > Hi,
> >
> > we are building an application using Lucene and we have HUGE data sets
> (our
> > index contains millions and millions and millions of documents), which
> > obviously cause us very important problems when sorting. In fact, we
> > disabled sorting completely because the servers were just exploding when
> > trying to sort results in RAM. The users using the system can search for
> > whatever they want, so we never know how many results will be returned -
> a
> > search can return 10 documents (no problem with sorting) or 10,000,000
> > (huge
> > sorting problems).
> >
> > I woke up this morning and had a flash : is it possible with Lucene to
> have
> > a "natural sorting" of documents? For example, let's say I have 3 columns
> I
> > want to be able to sort by : first name, last name, email; I would have 3
> > different indexes with the very same data but with a different primary
> key
> > for sorting. I know it's far fetched, and I have never seen anything like
> > that since I use Lucene, but we're just desperate... how people do to
> have
> > huge data sets, a lot of users, and sort!?
> >
> > Thanks,
> >
> > Mike
> >
>
