This question is related to Out of Memory Errors that I am seeing on my Solr Cloud Setup - I am running Solr 4.5.1.
Here is how my setup looks: 1. Have 6 Solr Tomcat Nodes distributed across 3 Servers - i.e. 2 nodes per server 2. Each tomcat node has been allocated 2 GB RAM - XmX setting Have two Collections: 1. UserGroupMappings (2 shards x 3 replicas = 6 Cores) 2. GroupCustomerMappings (2 shards x 3 replicas = 6 Cores) Each Solr Tomcat Node contains 2 cores - one from each of the 2 collection. Every Solr Tomcat Node manages a combined index size (from 2 cores) of approximately 6 GB. Here is the approximate structure of my Collections (have eliminated some fields for ease of representation): 1. UserGroupCollection - UserId, CompanyName, GroupName Each User is affiliated with exactly one company - but could service multiple Groups 2. GroupCustomerMappings - CustomerId, FirstName, LastName, Address, CustomerNameAndAddress (Copy Field), GroupName (multi-valued field), Company Name A Customer could be affiliated with multiple companies - and within a company could be affiliated with one or more groups >From my FE - when a user performs a search - he / she is looking for customers by entering their name or address. My application performs two queries: Query 1: Fetch all Groups for current user from UserGroupCollection q= *:* fq= UserId:<current user id> and CompanyName:<current user's company> fl = GroupName Query 2: Search for Customers within groups returned from Query 1 q= CustomerNameAndAddress: <user entered search term> fq= CompanyName:<current user's company> and (GroupName: <first groupName returned by Query 1> OR GroupName: <second groupName returned by Query 1> OR .... ) fl = CustomerId, FirstName, LastName, Address Have been load testing the application with 100 unique users. Have been noticing that every now and then some of my nodes run out of Memory. Did some research on the net and came across this article - http://teaspoon-consulting.com/articles/solr-cache-tuning.html - that explains the structure of the query cache, filter cache and document cache. In specific it states that: 1. Each entry in the Query Cache holds an array of integers containing DocIds that are returned as part of the search results. 2. Each entry in the Filter Cache holds an array of bits, and the size of the array equals the total number of documents in the current core (this was a real eye opener - points at the fact that the memory requirements of this cache increases for larger indices) Based on the above information should I consider Query 1 to something as follows: Query 1 (Modified) q= UserId:<current user id> and CompanyName:<current user's company> fl = GroupName Basically, I have gotten rid of the "fq" completely and put the query parameters in the "q". I am doing this because I know that Each User in my system manages a max of 50 Groups - which means a max of 50 Doc Ids (integers) in each query cache entry - as compared to 500,000 bits per filter cache entry - which is a lot more memory. I get the feeling that the original Query 1 is probably the semantically right solution (also I dont care about scoring for Query 1 - since its just a data fetch than a search) - but modified Query 1 will be much more performant. Is this a correct decision that I am making on Query 1 ? Also on Query 2 - I am looking to modify it as follows: Query 2 (Modified) q= CustomerNameAndAddress: <user entered search term> and CompanyName:<current user's company> fq= CompanyName:<current user's company> and (GroupName: <first groupName returned by Query 1> OR GroupName: <second groupName returned by Query 1> OR .... ) fl = CustomerId, FirstName, LastName, Address The only change I am making here is to add the "CompanyName:<current user's company>" which was originally only in the "fq" to the "q" as well (please note that I am NOT removing this parameter from the fq). The reason I am doing this is - if a user on the system searches for a person named "john" - I don't want the Query Cache to hold docIds for John's across all companies. I just want the query cache to hold DocIds for all John's in context of the current company. Is this a correct decision on Query 2 ? The one thing I am NOT very sure about is whether its appropriate / justifiable in my Use Case to have the same query parameter "CompanyName" in both the "q" and "fq". Also, need to mention that I fell into the trap of setting extremely huge cache sizes for the Solr Caches - Had set the sizes of each of the 3 caches (query, filter and document caches) to 500,000 entries - which is way too extreme. I am going to reduce that number to something between 500 - 1000 per cache based on query patterns that I see in the live system. I did a memory dump on one of my solr tomcat nodes - when it was on the verge of running out of memory (with my extreme cache settings) - and the biggest consumers of memory happened to be: class [Ljava.lang.Object; - 9,692 instances, Size - 721,392,616 class [B - 25,285 instances, Size - 91,429,183 class org.apache.lucene.search.ScoreDoc - 3,431,025 instances, Size - 41,172,300 class [I - 3,556 instances, Size - 34,444,276 class [C -92,970 instances, Size - 13,751,378 I am trying to co-relate / maps these classes to one of the 3 caches - filter, query and coument caches. Is it fair to assume the following: 1. "class [I" or the array of integers maps to the query cache ?? -- Its highly possible that I ran 3556 unique queries 2. "class [B" or the array of bytes maps to the filter cache ?? -- I am actually a little surprised here - I am seeing 25,285 instances - I was expecting a number more like 100 - since I had 100 unique users on my load test - and the way my system works - I have a unique filter query per user - so that makes a total of 100 filter cache entries. Also, I have no clue on what the class [Ljava.lang.Object - array of objects and class [C - array of characters represent ? Are they related to any of the caches or any thing else in Solr ? The object array is the biggest consumer of heap space - what could it be ?? Also - why am I seeing so many instances (3.4 Million) of "org.apache.lucene.search.ScoreDoc" ? Is this being stored in one of the caches ?? -- View this message in context: http://lucene.472066.n3.nabble.com/Query-on-Solr-Caches-OOM-Errors-and-how-q-and-fq-affect-Solr-Cache-memory-consumption-tp4163814.html Sent from the Solr - User mailing list archive at Nabble.com.