Re: What controls field cache size and eviction rates?
Should say -- Can anyone confirm if it's right *still*, since the article is 10 years old :) On Fri, Mar 5, 2021 at 10:36 AM Stephen Lewis Bianamara < stephen.bianam...@gmail.com> wrote: > Hi SOLR Community, > > Just following up here with an update. I found this article which goes > into depth on the field cache though stops short of discussing how it > handles eviction. Can anyone confirm if this info is right? > > https://lucidworks.com/post/scaling-lucene-and-solr/ > > > Also, can anyone speak to how the field cache handles evictions? > > Best, > Stephen > > On Wed, Feb 24, 2021 at 4:43 PM Stephen Lewis Bianamara < > stephen.bianam...@gmail.com> wrote: > >> Hi SOLR Community, >> >> I've been trying to understand how the field cache in SOLR manages >> its evictions, and it is not easily readable from the code or documentation >> the simple question of when and how something gets evicted from the field >> cache. This cache also doesn't show hit ratio, total hits, eviction ratio, >> total evictions, etc... in the web UI. >> >> For example: I've observed that if I write one document and trigger a >> query with a sort on the field, it will generate two entries in the field >> cache. Then if I repush the document, the entries get removed, but will >> otherwise stay there seemingly forever. If my query matches 2 docs, same >> thing but with 4 entries (2 each). Then, if I rewrite one of the docs, >> those two entries go away but not the two from the first one. This >> obviously implies that there are implications to write throughput >> performance based on this cache, so the fact that it is not configurable by >> the user and doesn't have very clear documentation is a bit worrisome. >> >> Can someone here help out and explain how the filter cache handles >> evictions, or perhaps send me the documentation if I missed it? >> >> >> Thanks! >> Stephen >> >
Re: What controls field cache size and eviction rates?
Hi SOLR Community, Just following up here with an update. I found this article which goes into depth on the field cache though stops short of discussing how it handles eviction. Can anyone confirm if this info is right? https://lucidworks.com/post/scaling-lucene-and-solr/ Also, can anyone speak to how the field cache handles evictions? Best, Stephen On Wed, Feb 24, 2021 at 4:43 PM Stephen Lewis Bianamara < stephen.bianam...@gmail.com> wrote: > Hi SOLR Community, > > I've been trying to understand how the field cache in SOLR manages > its evictions, and it is not easily readable from the code or documentation > the simple question of when and how something gets evicted from the field > cache. This cache also doesn't show hit ratio, total hits, eviction ratio, > total evictions, etc... in the web UI. > > For example: I've observed that if I write one document and trigger a > query with a sort on the field, it will generate two entries in the field > cache. Then if I repush the document, the entries get removed, but will > otherwise stay there seemingly forever. If my query matches 2 docs, same > thing but with 4 entries (2 each). Then, if I rewrite one of the docs, > those two entries go away but not the two from the first one. This > obviously implies that there are implications to write throughput > performance based on this cache, so the fact that it is not configurable by > the user and doesn't have very clear documentation is a bit worrisome. > > Can someone here help out and explain how the filter cache handles > evictions, or perhaps send me the documentation if I missed it? > > > Thanks! > Stephen >
What controls field cache size and eviction rates?
Hi SOLR Community, I've been trying to understand how the field cache in SOLR manages its evictions, and it is not easily readable from the code or documentation the simple question of when and how something gets evicted from the field cache. This cache also doesn't show hit ratio, total hits, eviction ratio, total evictions, etc... in the web UI. For example: I've observed that if I write one document and trigger a query with a sort on the field, it will generate two entries in the field cache. Then if I repush the document, the entries get removed, but will otherwise stay there seemingly forever. If my query matches 2 docs, same thing but with 4 entries (2 each). Then, if I rewrite one of the docs, those two entries go away but not the two from the first one. This obviously implies that there are implications to write throughput performance based on this cache, so the fact that it is not configurable by the user and doesn't have very clear documentation is a bit worrisome. Can someone here help out and explain how the filter cache handles evictions, or perhaps send me the documentation if I missed it? Thanks! Stephen
Re: Question about filter cache size
On Fri, Oct 3, 2014 at 6:38 PM, Peter Keegan wrote: >> it will be cached as hidden:true and then inverted > Inverted at query time, so for best query performance use fq=hidden:false, > right? Yep. -Yonik http://heliosearch.org - native code faceting, facet functions, sub-facets, off-heap data
Re: Question about filter cache size
> it will be cached as hidden:true and then inverted Inverted at query time, so for best query performance use fq=hidden:false, right? On Fri, Oct 3, 2014 at 3:57 PM, Yonik Seeley wrote: > On Fri, Oct 3, 2014 at 3:42 PM, Peter Keegan > wrote: > > Say I have a boolean field named 'hidden', and less than 1% of the > > documents in the index have hidden=true. > > Do both these filter queries use the same docset cache size? : > > fq=hidden:false > > fq=!hidden:true > > Nope... !hidden:true will be smaller in the cache (it will be cached > as hidden:true and then inverted) > The downside is that you'll pay the cost of that inversion. > > -Yonik > http://heliosearch.org - native code faceting, facet functions, > sub-facets, off-heap data >
Re: Question about filter cache size
On Fri, Oct 3, 2014 at 4:35 PM, Shawn Heisey wrote: > On 10/3/2014 1:57 PM, Yonik Seeley wrote: >> On Fri, Oct 3, 2014 at 3:42 PM, Peter Keegan wrote: >>> Say I have a boolean field named 'hidden', and less than 1% of the >>> documents in the index have hidden=true. >>> Do both these filter queries use the same docset cache size? : >>> fq=hidden:false >>> fq=!hidden:true >> >> Nope... !hidden:true will be smaller in the cache (it will be cached >> as hidden:true and then inverted) >> The downside is that you'll pay the cost of that inversion. > > I would think that unless it's using hashDocSet, the cached data for > every filter would always be the same size. The wiki says that > hashDocSet is no longer used for filter caching as of 1.4.0. Is that > actually true? Yes, SortedIntDocSet is used instead. It stores an int per match (i.e. 4 bytes per match). This change was made so in-order traversal could be done efficiently. -Yonik http://heliosearch.org - native code faceting, facet functions, sub-facets, off-heap data
Re: Question about filter cache size
On 10/3/2014 1:57 PM, Yonik Seeley wrote: > On Fri, Oct 3, 2014 at 3:42 PM, Peter Keegan wrote: >> Say I have a boolean field named 'hidden', and less than 1% of the >> documents in the index have hidden=true. >> Do both these filter queries use the same docset cache size? : >> fq=hidden:false >> fq=!hidden:true > > Nope... !hidden:true will be smaller in the cache (it will be cached > as hidden:true and then inverted) > The downside is that you'll pay the cost of that inversion. I would think that unless it's using hashDocSet, the cached data for every filter would always be the same size. The wiki says that hashDocSet is no longer used for filter caching as of 1.4.0. Is that actually true? Is my understanding of filterCache completely out of touch with reality? https://wiki.apache.org/solr/SolrCaching#The_hashDocSet_Max_Size This does bring to mind an optimization that might help memory usage in cases where either a very small or very large percentage of documents match the filter: do run-length encoding on the bitset. If the RLE representation is at least N percent smaller than the bitset, use that representation instead. I think the first iteration of an RLE option would have it always on or always off, controlled in solrconfig.xml. A config mode where Solr attempts RLE on every bitset and periodically reports efficiency statistics would be pretty nice. That data might be useful to define default thresholds for a future automatic mode. Thanks, Shawn
Re: Question about filter cache size
On Fri, Oct 3, 2014 at 3:42 PM, Peter Keegan wrote: > Say I have a boolean field named 'hidden', and less than 1% of the > documents in the index have hidden=true. > Do both these filter queries use the same docset cache size? : > fq=hidden:false > fq=!hidden:true Nope... !hidden:true will be smaller in the cache (it will be cached as hidden:true and then inverted) The downside is that you'll pay the cost of that inversion. -Yonik http://heliosearch.org - native code faceting, facet functions, sub-facets, off-heap data
Question about filter cache size
Say I have a boolean field named 'hidden', and less than 1% of the documents in the index have hidden=true. Do both these filter queries use the same docset cache size? : fq=hidden:false fq=!hidden:true Peter
Re: Solr Filter Cache Size
What Erick said. That's a giant Filter Cache. Have a look at these Solr metrics and note the Filter Cache in the middle: http://www.flickr.com/photos/otis/8409088080/ Note how small the cache is and how high the hit rate is. Those are stats for http://search-lucene.com/ and http://search-hadoop.com/ where you can see facets on the right that and up being used as filter queries. Most Solr apps I've seen had small Filter Caches. Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Wed, Mar 5, 2014 at 3:34 PM, Erick Erickson wrote: > This, BTW, is an ENORMOUS number cached queries. > > Here's a rough guide: > Each entry will be (length of query) + maxDoc/8 bytes long. > > Think of the filterCache as a map where the key is the query > and the value is a bitmap large enough to hold maxDoc bits. > > BTW, I'd kick this back to the default (512?) and periodically check > it with the admin>>plugins/stats page to see what kind of hit ratio > I have and adjust from there. > > Best, > Erick > > On Mon, Mar 3, 2014 at 11:00 AM, Benjamin Wiens > wrote: > > How can we calculate how much heap memory the filter cache will consume? > We > > understand that in order to determine a good size we also need to > evaluate > > how many filterqueries would be used over a certain time period. > > > > > > > > Here's our setting: > > > > > > > > > > > class="solr.FastLRUCache" > > > > size="30" > > > > initialSize="30" > > > > autowarmCount="5"/> > > > > > > > > According to the post below, 53 GB of RAM would be needed just by the > > filter cache alone with 1.4 million Docs. Not sure if this true and how > > this would work. > > > > > > > > Reference: > > > http://stackoverflow.com/questions/2004/solr-filter-cache-fastlrucache-takes-too-much-memory-and-results-in-out-of-mem > > > > > > > > We filled the filterquery cache with Solr Meter and had a JVM Heap Size > of > > far less than 53 GB. > > > > > > > > Can anyone chime in and enlighten us? > > > > > > > > Thank you! > > > > > > Ben Wiens & Benjamin Mosior >
Re: Solr Filter Cache Size
This, BTW, is an ENORMOUS number cached queries. Here's a rough guide: Each entry will be (length of query) + maxDoc/8 bytes long. Think of the filterCache as a map where the key is the query and the value is a bitmap large enough to hold maxDoc bits. BTW, I'd kick this back to the default (512?) and periodically check it with the admin>>plugins/stats page to see what kind of hit ratio I have and adjust from there. Best, Erick On Mon, Mar 3, 2014 at 11:00 AM, Benjamin Wiens wrote: > How can we calculate how much heap memory the filter cache will consume? We > understand that in order to determine a good size we also need to evaluate > how many filterqueries would be used over a certain time period. > > > > Here's our setting: > > > > > class="solr.FastLRUCache" > > size="30" > > initialSize="30" > > autowarmCount="5"/> > > > > According to the post below, 53 GB of RAM would be needed just by the > filter cache alone with 1.4 million Docs. Not sure if this true and how > this would work. > > > > Reference: > http://stackoverflow.com/questions/2004/solr-filter-cache-fastlrucache-takes-too-much-memory-and-results-in-out-of-mem > > > > We filled the filterquery cache with Solr Meter and had a JVM Heap Size of > far less than 53 GB. > > > > Can anyone chime in and enlighten us? > > > > Thank you! > > > Ben Wiens & Benjamin Mosior
Solr Filter Cache Size
How can we calculate how much heap memory the filter cache will consume? We understand that in order to determine a good size we also need to evaluate how many filterqueries would be used over a certain time period. Here's our setting: According to the post below, 53 GB of RAM would be needed just by the filter cache alone with 1.4 million Docs. Not sure if this true and how this would work. Reference: http://stackoverflow.com/questions/2004/solr-filter-cache-fastlrucache-takes-too-much-memory-and-results-in-out-of-mem We filled the filterquery cache with Solr Meter and had a JVM Heap Size of far less than 53 GB. Can anyone chime in and enlighten us? Thank you! Ben Wiens & Benjamin Mosior
Re: How to decide proper cache size at load testing?
You simply need to monitor and adjust. Both during testing and in production because search patterns change over time. Hook up alerting to it to get notified of high evictions and low cache hit rate so you don't have to actively look at stats all day. Here is the graph of Query Cache metrics for http://search-lucene.com/ for example: https://apps.sematext.com/spm-reports/s.do?k=eDcirzHG7i Otis Solr & ElasticSearch Support http://sematext.com/ On May 2, 2013 5:14 PM, "Furkan KAMACI" wrote: > I read that at wiki: > > Sometimes a smaller cache size will help avoid full garbage collections at > the cost of more evictions. Load testing should be used to help determine > proper cache sizes throughout the searching/indexing lifecycle. > > Could anybody give me an example scenario of how can I make a test, what > should I do and find a proper cache size at load testing? >
How to decide proper cache size at load testing?
I read that at wiki: Sometimes a smaller cache size will help avoid full garbage collections at the cost of more evictions. Load testing should be used to help determine proper cache sizes throughout the searching/indexing lifecycle. Could anybody give me an example scenario of how can I make a test, what should I do and find a proper cache size at load testing?
Re: Solr cache size information
Thanks a lot for these answers! Elisabeth 2011/12/4 Erick Erickson > See below: > > On Thu, Dec 1, 2011 at 10:57 AM, elisabeth benoit > wrote: > > Hello, > > > > If anybody can help, I'd like to confirm a few things about Solr's caches > > configuration. > > > > If I want to calculate cache size in memory relativly to cache size in > > solrconfig.xml > > > > For Document cache > > > > size in memory = size in solrconfig.xml * average size of all fields > > defined in fl parameter ??? > > pretty much. > > > > > For Filter cache > > > > size in memory = size in solrconfig.xml * WHAT (the size of an id) ??? (I > > don't use facet.enum method) > > > > It Depends(tm). Solr tries to do the best thing here, depending upon > how many docs match the filter query. One method puts in a bitset for > each > entry, which is (maxDocs/8) bytes. maxDocs is reported on the admin/stats > page. > > If the filter cache only hits a few documents, the size is smaller than > that. > > You can think of this cache as a map where the key is the > filter query (which is how they're re-used and how autowarm > works) and the value for each key is the bitset or list. The > size of the map is bounded by the size in solrconfig.xml. > > > For Query result cache > > > > size in memory = size in solrconfig.xml * the size of an id ??? > > > Pretty much. This is the maximum size, but each entry is > the query plus a list of IDs that's up to > long. This cache is, by and large, the least of your worries. > > > > > > I would also like to know relation between solr's caches sizes and JVM > max > > size? > > Don't quite know what you're asking for here. There's nothing automatic > that's sensitive to whether the JVM memory limits are about to be exceeded. > If the caches get too big, OOMs happen. > > > > > If anyone has an answer or a link for further reading to suggest, it > would > > be greatly appreciated. > > > There's some information here: http://wiki.apache.org/solr/SolrCaching, > but > it often comes down to "try your app and monitor" > > Here's a work-in-progress that Grant is working on, be aware that it's > for trunk, not 3x. > http://java.dzone.com/news/estimating-memory-and-storage > > > Best > Erick > > > Thanks, > > Elisabeth >
Re: Solr cache size information
See below: On Thu, Dec 1, 2011 at 10:57 AM, elisabeth benoit wrote: > Hello, > > If anybody can help, I'd like to confirm a few things about Solr's caches > configuration. > > If I want to calculate cache size in memory relativly to cache size in > solrconfig.xml > > For Document cache > > size in memory = size in solrconfig.xml * average size of all fields > defined in fl parameter ??? pretty much. > > For Filter cache > > size in memory = size in solrconfig.xml * WHAT (the size of an id) ??? (I > don't use facet.enum method) > It Depends(tm). Solr tries to do the best thing here, depending upon how many docs match the filter query. One method puts in a bitset for each entry, which is (maxDocs/8) bytes. maxDocs is reported on the admin/stats page. If the filter cache only hits a few documents, the size is smaller than that. You can think of this cache as a map where the key is the filter query (which is how they're re-used and how autowarm works) and the value for each key is the bitset or list. The size of the map is bounded by the size in solrconfig.xml. > For Query result cache > > size in memory = size in solrconfig.xml * the size of an id ??? > Pretty much. This is the maximum size, but each entry is the query plus a list of IDs that's up to long. This cache is, by and large, the least of your worries. > > I would also like to know relation between solr's caches sizes and JVM max > size? Don't quite know what you're asking for here. There's nothing automatic that's sensitive to whether the JVM memory limits are about to be exceeded. If the caches get too big, OOMs happen. > > If anyone has an answer or a link for further reading to suggest, it would > be greatly appreciated. > There's some information here: http://wiki.apache.org/solr/SolrCaching, but it often comes down to "try your app and monitor" Here's a work-in-progress that Grant is working on, be aware that it's for trunk, not 3x. http://java.dzone.com/news/estimating-memory-and-storage Best Erick > Thanks, > Elisabeth
RE: Solr cache size information
> For Filter cache > > size in memory = size in solrconfig.xml * WHAT (the size of an id) ??? > (I > don't use facet.enum method) > As I understand it, size is the number queries that will be cached. My short experience means that the memory consumed will be data dependent. If you have a huge number of documents matched in a FQ, then the size consumed will be very large, if you get a single match then the cached result will take much less memory. I don't know if there is a way you can bound the cache by memory rather than results. I think all of the solr caches behave this way, but I am not sure. NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
Solr cache size information
Hello, If anybody can help, I'd like to confirm a few things about Solr's caches configuration. If I want to calculate cache size in memory relativly to cache size in solrconfig.xml For Document cache size in memory = size in solrconfig.xml * average size of all fields defined in fl parameter ??? For Filter cache size in memory = size in solrconfig.xml * WHAT (the size of an id) ??? (I don't use facet.enum method) For Query result cache size in memory = size in solrconfig.xml * the size of an id ??? I would also like to know relation between solr's caches sizes and JVM max size? If anyone has an answer or a link for further reading to suggest, it would be greatly appreciated. Thanks, Elisabeth
Re: Cache size
You can dump the heap and analyze it with a tool like jhat. IBM's heap analyzer is also a very good tool and if i'm not mistaken people also use one that comes with Eclipse. On Tuesday 08 February 2011 16:35:35 Mehdi Ben Haj Abbes wrote: > Hi folks, > > Is there any way to know the size *in bytes* occupied by a cache (filter > cache, doc cache ...)? I don't find such information within the stats page. > > Regards -- Markus Jelsma - CTO - Openindex http://www.linkedin.com/in/markus17 050-8536620 / 06-50258350
Cache size
Hi folks, Is there any way to know the size *in bytes* occupied by a cache (filter cache, doc cache ...)? I don't find such information within the stats page. Regards -- Mehdi BEN HAJ ABBES
Re: Solr query result cache size and "expire" property
: please help - how can I calculate queryresultcache size (how much RAM should : be dedicated for that). I have 1,5 index size, 4 mio docs. : QueryResultWindowSize is 20. : Could I use "expire" property on the documents in this cache? There is no "expire" property, items are automaticly removed from the cache if the cache gets full, and the entire cache is thrown out when a new searcher is loaded (that's the only time it would make sense to "expire" anything) honestly: trial and error is typically the best bet for sizing your queryResultsCache ... the size of your index is much less relevant then the types of queries you get. If you typically only have 200 unique queries over and over again, and no one ever does any ohter queries, then any number abot 200 is going to be essentially the same. if you have 200 queries thta get a *lot* and 100 other queries that get hit once or twice ver ... then something ~250 is probably a good idea ... any more is probably just a waste of ram, any less is probably a waste of CPU. -Hoss
Solr query result cache size and "expire" property
Hi all! please help - how can I calculate queryresultcache size (how much RAM should be dedicated for that). I have 1,5 index size, 4 mio docs. QueryResultWindowSize is 20. Could I use "expire" property on the documents in this cache? regards, Stanislaw
Re: maximum recommended document cache size
The general recommendation is to watch the caches during normal user searches and keep increasing the size until evictions start happening. This may or may not work for your situation. The problem is that the eviction rate does not show "lifetime in cache". So if 90% of the cache sits there indefinitely and the remaining 10% churns, the cache is fine but you'll show zillions of evictions. On Thu, May 13, 2010 at 10:38 AM, Nagelberg, Kallin wrote: > I am trying to tune my Solr setup so that the caches are well warmed after > the index is updated. My documents are quite small, usually under 10k. I > currently have a document cache size of about 15,000, and am warming up 5,000 > with a query after each indexing. Autocommit is set at 30 seconds, and my > caches are warming up easily in just a couple of seconds. I've read of > concerns regarding garbage collection when your cache is too large. Does > anyone have experience with this? Ideally I would like to get 90% of all > documents from the last month in memory after each index, which would be > around 25,000. I'm doing extensive load testing, but if someone has > recommendations I'd love to hear them. > > Thanks, > -Kallin Nagelberg > -- Lance Norskog goks...@gmail.com
maximum recommended document cache size
I am trying to tune my Solr setup so that the caches are well warmed after the index is updated. My documents are quite small, usually under 10k. I currently have a document cache size of about 15,000, and am warming up 5,000 with a query after each indexing. Autocommit is set at 30 seconds, and my caches are warming up easily in just a couple of seconds. I've read of concerns regarding garbage collection when your cache is too large. Does anyone have experience with this? Ideally I would like to get 90% of all documents from the last month in memory after each index, which would be around 25,000. I'm doing extensive load testing, but if someone has recommendations I'd love to hear them. Thanks, -Kallin Nagelberg
Re: Query on Cache size.
On Mon, Dec 14, 2009 at 7:17 PM, kalidoss < kalidoss.muthuramalin...@sifycorp.com> wrote: > Hi, > > We have enabled the query result cache, its 512 entries, > > we have calculated the size used for cache : > page size about 1000bytes, (1000*512)/1024/1024 = .48MB > > The query result cache is a map of (q, sort, n) to ordered list of Lucene docids. Assuming queryResultWindowSize iw 20 and an average user does not go beyond 20 results, your memory usage of the values in this map is approx 20*sizeof(int)*512. Add some more for keys, map, references etc. -- Regards, Shalin Shekhar Mangar.
Query on Cache size.
Hi, We have enabled the query result cache, its 512 entries, we have calculated the size used for cache : page size about 1000bytes, (1000*512)/1024/1024 = .48MB If we increase the cache count to 10 then the memory used for cache about. (1000*10)/1024/1024 = 96MB Is my calculation is correct? my system has 2.5GB of RAM for Solr and 1GB for OS. If am wrong, let me know how to calculate the memory? Thanks, Kalidoss.m, Get your world in your inbox! Mail, widgets, documents, spreadsheets, organizer and much more with your Sifymail WIYI id! Log on to http://www.sify.com ** DISCLAIMER ** Information contained and transmitted by this E-MAIL is proprietary to Sify Limited and is intended for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If this is a forwarded message, the content of this E-MAIL may not have been sent with the authority of the Company. If you are not the intended recipient, an agent of the intended recipient or a person responsible for delivering the information to the named recipient, you are notified that any use, distribution, transmission, printing, copying or dissemination of this information in any way or in any manner is strictly prohibited. If you have received this communication in error, please delete this mail & notify us immediately at ad...@sifycorp.com
Re: Cache size clarification
On Jan 28, 2008, at 6:05 PM, Alex Benjamen wrote: I need some clarification on the cache size parameters in the solrconfig. Suppose I'm using these values: A lot of this is here: http://wiki.apache.org/solr/SolrCaching
Cache size clarification
I need some clarification on the cache size parameters in the solrconfig. Suppose I'm using these values: What does size="5" mean... Is this 5 bytes, kilobytes, megabytes... or is it the number of documents that can be cached? In other words, how do I calculate the memory usage based on the size setting? If the size is "1000", how much physical memory will this cache occupy Also, suppose I see the following in the stats section for queryResultCache: lookups : 24616 hits : 21102 hitratio : 0.85 inserts : 8456 evictions : 0 size : 3510 cumulative_lookups : 24616 cumulative_hits : 21102 cumulative_hitratio : 0.85 cumulative_inserts : 8456 cumulative_evictions : 0 Based on this, what would be the recommended setting forLRU Cache maxSize=??? -Thx
Re: Cache size and Heap size
: > I know this is a lot and I'm going to decrease it, I was just experimenting, : > but I need some guidelines of how to calculate the right size of the cache. : : Each filter that matches more than ~3000 documents will occupy maxDocs/8 bytes : of memory. Certain kinds of faceting require one entry per unique value in a FWIW: the magic number 3000 is the example value of the config option ... it can be tweaked if you think you can tune it to a value that makes more sense given the nature of the types of DocSets you are dealing with, but i wouldn't bother (there are probably a lot of better ways you can spend your time to tweak performance) -Hoss
Re: Cache size and Heap size
On 16-Jan-08, at 11:15 AM, [EMAIL PROTECTED] wrote: I'm using Tomcat. I set Max Size = 5Gb and I checked in profiler that it's actually uses whole memory. There is no significant memory use by other applications. Whole change was I increased the size of cache to: LRU Cache(maxSize=1048576, initialSize=1048576, autowarmCount=524288, [EMAIL PROTECTED]) autowarmcount > maxSize certainly doesn't make sense. I know this is a lot and I'm going to decrease it, I was just experimenting, but I need some guidelines of how to calculate the right size of the cache. Each filter that matches more than ~3000 documents will occupy maxDocs/8 bytes of memory. Certain kinds of faceting require one entry per unique value in a field. The best way to tune this is to monitor your cache hit/expunge statistics for the filter cache (on the solr admin statistics screen). -Mike
Re: Cache size and Heap size
I'm using Tomcat. I set Max Size = 5Gb and I checked in profiler that it's actually uses whole memory. There is no significant memory use by other applications. Whole change was I increased the size of cache to: LRU Cache(maxSize=1048576, initialSize=1048576, autowarmCount=524288, [EMAIL PROTECTED]) I know this is a lot and I'm going to decrease it, I was just experimenting, but I need some guidelines of how to calculate the right size of the cache. Thank you Gene - Original Message From: Daniel Alheiros <[EMAIL PROTECTED]> To: solr-user@lucene.apache.org Sent: Wednesday, January 16, 2008 10:48:50 AM Subject: Re: Cache size and Heap size Hi Gene. Have you set your app server / servlet container to use allocate some of this memory to be used? You can define the maximum and minimum heap size adding/replacing some parameters on the app server initialization: -Xmx1536m -Xms1536m Which app server / servlet container are you using? Regards, Daniel Alheiros On 16/1/08 15:23, "Evgeniy Strokin" <[EMAIL PROTECTED]> wrote: > Hello,.. > I have relatively large RAM (10Gb) on my server which is running Solr. I > increased Cache settings and start to see OutOfMemory exceptions, specially on > facet search. > Is anybody has some suggestions how Cache settings related to Memory > consumptions? What are optimal settings? How they could be calculated? > > Thank you for any advise, > Gene http://www.bbc.co.uk/ This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated. If you have received it in error, please delete it from your system. Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately. Please note that the BBC monitors e-mails sent or received. Further communication will signify your consent to this.
Re: Cache size and Heap size
Hi Gene. Have you set your app server / servlet container to use allocate some of this memory to be used? You can define the maximum and minimum heap size adding/replacing some parameters on the app server initialization: -Xmx1536m -Xms1536m Which app server / servlet container are you using? Regards, Daniel Alheiros On 16/1/08 15:23, "Evgeniy Strokin" <[EMAIL PROTECTED]> wrote: > Hello,.. > I have relatively large RAM (10Gb) on my server which is running Solr. I > increased Cache settings and start to see OutOfMemory exceptions, specially on > facet search. > Is anybody has some suggestions how Cache settings related to Memory > consumptions? What are optimal settings? How they could be calculated? > > Thank you for any advise, > Gene http://www.bbc.co.uk/ This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated. If you have received it in error, please delete it from your system. Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately. Please note that the BBC monitors e-mails sent or received. Further communication will signify your consent to this.
Cache size and Heap size
Hello,.. I have relatively large RAM (10Gb) on my server which is running Solr. I increased Cache settings and start to see OutOfMemory exceptions, specially on facet search. Is anybody has some suggestions how Cache settings related to Memory consumptions? What are optimal settings? How they could be calculated? Thank you for any advise, Gene
