Thats why I asked about multi-valued terms. If hes not using the enum
faceting method (which only makes sense with fewer uniques), and the
fields are not multi-valued, than it is using the FieldCache method.
Which of course does use the filterCache, and works best when the
filterCache size is the size of all the unique terms in all the fields
you are faceting on.
Perhaps his machine can't handle that, but certainly he would benefit
heavily from the cache. And if you had tons of uniques and a small
cache, you would see exactly what he is seeing. Probably best to see how
it goes before seeing if you have to optimize based on garbage
collection (and I beleive that he could run in to resource issues, whats
why I said give it a shot :) ). He may just need more resources than he
can get, but I'm fairly sure he needs the resources. Unless its
multivalued fields and its using the UninvertedField - not sure how the
filterCache plays into that.
- Mark
Grant Ingersoll wrote:
On Jun 17, 2009, at 10:32 PM, Mark Miller wrote:
Right, so if you are on 1.3 or early 1.4 dev, with so many uniques,
you should be using the FieldCache method of faceting. The RAM
depends on the number of documents and number of uniques terms mostly.
With 1.4 you may be using an UninvertedField though (are your facet
fields multivalued or tokenized?), and I know much less about that.
I'd try a cache size of 10,000 and see how it goes.
I'm not so sure about that, my guess is your going to get hammered on
garbage collection when you do commits with something that big.
Let's take a step back. These are LRU cache's, the fact that you have
a zero hit ratio does not mean caching isn't working or that you
necessarily need a bigger cache. It suggests to me that your
application is not the type that can benefit from caching of
filters. My understanding is that in certain cases with the new 1.4
faceting, it ends up using the filterCache as well. I believe the
admin will give stats on the number of big terms, etc.
Perhaps you can give a bit more detail about your application and why
you think that cache ratio is causing slower faceting.
Have you actually done some profiling/timings on the faceting?
- Mark
Manepalli, Kalyan wrote:
Got that, if its number of cache entries, definitely its very low. I
have around 10,000 unique items to facet on. Does the RAM size
depend on Document size.
Thanks,
Kalyan Manepalli
-----Original Message-----
From: Mark Miller [mailto:markrmil...@gmail.com]
Sent: Wednesday, June 17, 2009 7:13 PM
To: solr-user@lucene.apache.org
Subject: Re: FilterCache issue
Its been a while since I've thought about this sort of thing, but it
looks like your cache is way too small and things get evicted before
being used. How many uniques are you faceting on? 512 is the number of
cache entries, not the size in kb/mb.
Try raising it - perhaps a lot ;) But consider that you have to have
the
RAM to accommodate as well ...
What version of Solr are you using?
--
- Mark
http://www.lucidimagination.com
Manepalli, Kalyan wrote:
Hi,
I am seeing an issue with the filtercache setting on my
solr app which is causing slower faceting.
Here is the configuration.
<filterCache class="solr.LRUCache" size="512" initialSize="512"
autowarmCount="256"/>
Statistics:
description: LRU Cache(maxSize=512, initialSize=512,
autowarmCount=256,
regenerator=org.apache.solr.search.solrindexsearche...@8d41f2)
stats: lookups : 979692
hits : 6904
hitratio : 0.00
inserts : 973531
evictions : 972978
size : 512
warmupTime : 1479
cumulative_lookups : 61660491
cumulative_hits : 516057
cumulative_hitratio : 0.00
cumulative_inserts : 61170111
cumulative_evictions : 61153787
As we can see the cache hit ratio is almost zero. How do I improve
the filter cache.
Also wanted to know what does the size mean. Is it number of
documents or the memory size (kb/mb)
Any suggestions in this regard will be very helpful.
Thanks,
Kalyan Manepalli
--
- Mark
http://www.lucidimagination.com
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
using Solr/Lucene:
http://www.lucidimagination.com/search
--
- Mark
http://www.lucidimagination.com
