On Sun, Oct 25, 2015, at 05:43 PM, Salman Ansari wrote: > Thanks guys for your responses. > > That's a very very large cache size. It is likely to use a VERY large > amount of heap, and autowarming up to 4096 entries at commit time might > take many *minutes*. Each filterCache entry is maxDoc/8 bytes. On an > index core with 70 million documents, each filterCache entry is at least > 8.75 million bytes. Multiply that by 16384, and a completely full cache > would need about 140GB of heap memory. 4096 entries will require 35GB. > I don't think this cache is actually storing that many entries, or you > would most certainly be running into OutOfMemoryError exceptions. > > True, however, I have tried with the default filtercache at the beginning > but the problem was still there. So, I don't think that is how I should > increase the performance of my Solr. Moreover, as you mentioned, when I > change the configuration, I should be running out of memory but that did > not happen. Do you think my Solr has not taken the latest configs? I have > restarted the Solr btw. > > Lately I have been trying different ways to improve this and I have > created > a brand new index on the same machine using 2 shards and it had few > entries > (about 5) and the performance was booming, I got the results back in 42 > ms > sometimes. What concerns me is that may be I am loading too much into one > index so that is why this is killing the performance. Is there a > recommended index size/document number and size that I should be looking > at > to tune this? Any other ideas other than increasing the memory size as I > have already tried this?
The optimal index size is down to the size of segments on disk. New segments are created when hard commits occur, and existing on-disk segments may get merged in the background when the segment count gets too high. Now, if those on-disk segments get too large, copying them around at merge time can get prohibitive, especially if your index is changing frequently. Splitting such an index into shards is one approach to dealing with this issue. Upayavira
