On Wed, 2015-05-13 at 09:22 +0000, Abhishek Gupta wrote:
> Yes we have that many documents (exact count: 522664425), but I am not
> sure why that matters because what I understood from documentation is
> that fc will only work on the documents filtered by filter query and
> query.
What the documentation does not mention explicitly is the UnInversion
that takes place on first call. If you look in your Solr-log after
"UnInverted", you will see how many milliseconds it takes at the
parameter "time".
For example:
UnInverted multi-valued field {field=lsubject,memSize=216343445,
tindexSize=1037315,time=36620,phase1=35868,nTerms=4440544,bigTerms=1,
termInstances=55196823,uses=0}
took 36620 milliseconds to UnInvert 4.440.544 terms.
The number of references from documents to terms is 55.196.823. If we
assume you have approximately 1 reference/document, you will have half a
billion references or about 10 times my number. 10 times 37 seconds is
quite close to the 300 seconds you state below. Of course our numbers
cannot be compared directly, but it means that your measurements passed
the sanity check.
> For my query there are only 137 documents for fc to work on and to
> make FieldCache.
The mapping structure from your 522.664.425 documents to the values in
your field (also in the higher millions, as I understand it) is
independent of your search result.
After the structure has been created, it is used to look up the terms
used by your 137 hits.
> Also subsequent calls are not fast:
> First call time: 297572
> Second call time (made with in 2 sec): 249287
Are you indexing while searching? Each time the index is changed, the
UnInversion will have to be re-done. facet.method=fcs seems a better
choice with an often-changing index of your size.
>
- Toke Eskildsen, State and University Library, Denmark
>
