Re: Solr cache size information
Thanks a lot for these answers! Elisabeth 2011/12/4 Erick Erickson > See below: > > On Thu, Dec 1, 2011 at 10:57 AM, elisabeth benoit > wrote: > > Hello, > > > > If anybody can help, I'd like to confirm a few things about Solr's caches > > configuration. > > > > If I want to calculate cache size in memory relativly to cache size in > > solrconfig.xml > > > > For Document cache > > > > size in memory = size in solrconfig.xml * average size of all fields > > defined in fl parameter ??? > > pretty much. > > > > > For Filter cache > > > > size in memory = size in solrconfig.xml * WHAT (the size of an id) ??? (I > > don't use facet.enum method) > > > > It Depends(tm). Solr tries to do the best thing here, depending upon > how many docs match the filter query. One method puts in a bitset for > each > entry, which is (maxDocs/8) bytes. maxDocs is reported on the admin/stats > page. > > If the filter cache only hits a few documents, the size is smaller than > that. > > You can think of this cache as a map where the key is the > filter query (which is how they're re-used and how autowarm > works) and the value for each key is the bitset or list. The > size of the map is bounded by the size in solrconfig.xml. > > > For Query result cache > > > > size in memory = size in solrconfig.xml * the size of an id ??? > > > Pretty much. This is the maximum size, but each entry is > the query plus a list of IDs that's up to > long. This cache is, by and large, the least of your worries. > > > > > > I would also like to know relation between solr's caches sizes and JVM > max > > size? > > Don't quite know what you're asking for here. There's nothing automatic > that's sensitive to whether the JVM memory limits are about to be exceeded. > If the caches get too big, OOMs happen. > > > > > If anyone has an answer or a link for further reading to suggest, it > would > > be greatly appreciated. > > > There's some information here: http://wiki.apache.org/solr/SolrCaching, > but > it often comes down to "try your app and monitor" > > Here's a work-in-progress that Grant is working on, be aware that it's > for trunk, not 3x. > http://java.dzone.com/news/estimating-memory-and-storage > > > Best > Erick > > > Thanks, > > Elisabeth >
Re: Solr cache size information
See below: On Thu, Dec 1, 2011 at 10:57 AM, elisabeth benoit wrote: > Hello, > > If anybody can help, I'd like to confirm a few things about Solr's caches > configuration. > > If I want to calculate cache size in memory relativly to cache size in > solrconfig.xml > > For Document cache > > size in memory = size in solrconfig.xml * average size of all fields > defined in fl parameter ??? pretty much. > > For Filter cache > > size in memory = size in solrconfig.xml * WHAT (the size of an id) ??? (I > don't use facet.enum method) > It Depends(tm). Solr tries to do the best thing here, depending upon how many docs match the filter query. One method puts in a bitset for each entry, which is (maxDocs/8) bytes. maxDocs is reported on the admin/stats page. If the filter cache only hits a few documents, the size is smaller than that. You can think of this cache as a map where the key is the filter query (which is how they're re-used and how autowarm works) and the value for each key is the bitset or list. The size of the map is bounded by the size in solrconfig.xml. > For Query result cache > > size in memory = size in solrconfig.xml * the size of an id ??? > Pretty much. This is the maximum size, but each entry is the query plus a list of IDs that's up to long. This cache is, by and large, the least of your worries. > > I would also like to know relation between solr's caches sizes and JVM max > size? Don't quite know what you're asking for here. There's nothing automatic that's sensitive to whether the JVM memory limits are about to be exceeded. If the caches get too big, OOMs happen. > > If anyone has an answer or a link for further reading to suggest, it would > be greatly appreciated. > There's some information here: http://wiki.apache.org/solr/SolrCaching, but it often comes down to "try your app and monitor" Here's a work-in-progress that Grant is working on, be aware that it's for trunk, not 3x. http://java.dzone.com/news/estimating-memory-and-storage Best Erick > Thanks, > Elisabeth
RE: Solr cache size information
> For Filter cache > > size in memory = size in solrconfig.xml * WHAT (the size of an id) ??? > (I > don't use facet.enum method) > As I understand it, size is the number queries that will be cached. My short experience means that the memory consumed will be data dependent. If you have a huge number of documents matched in a FQ, then the size consumed will be very large, if you get a single match then the cached result will take much less memory. I don't know if there is a way you can bound the cache by memory rather than results. I think all of the solr caches behave this way, but I am not sure. NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
