I have already indexed all the documents in Solr and not indexing anymore. So the problem I am running in is after all the documents are indexed. I am using Solr cloud with two shards and two replicas for each shard but on the same machine. Is there anywhere I can look at the relation between index size and machine specs and its effect on Solr query performance?
Regards, Salman On Mon, Oct 26, 2015 at 5:55 PM, Upayavira <u...@odoko.co.uk> wrote: > > > On Sun, Oct 25, 2015, at 05:43 PM, Salman Ansari wrote: > > Thanks guys for your responses. > > > > That's a very very large cache size. It is likely to use a VERY large > > amount of heap, and autowarming up to 4096 entries at commit time might > > take many *minutes*. Each filterCache entry is maxDoc/8 bytes. On an > > index core with 70 million documents, each filterCache entry is at least > > 8.75 million bytes. Multiply that by 16384, and a completely full cache > > would need about 140GB of heap memory. 4096 entries will require 35GB. > > I don't think this cache is actually storing that many entries, or you > > would most certainly be running into OutOfMemoryError exceptions. > > > > True, however, I have tried with the default filtercache at the beginning > > but the problem was still there. So, I don't think that is how I should > > increase the performance of my Solr. Moreover, as you mentioned, when I > > change the configuration, I should be running out of memory but that did > > not happen. Do you think my Solr has not taken the latest configs? I have > > restarted the Solr btw. > > > > Lately I have been trying different ways to improve this and I have > > created > > a brand new index on the same machine using 2 shards and it had few > > entries > > (about 5) and the performance was booming, I got the results back in 42 > > ms > > sometimes. What concerns me is that may be I am loading too much into one > > index so that is why this is killing the performance. Is there a > > recommended index size/document number and size that I should be looking > > at > > to tune this? Any other ideas other than increasing the memory size as I > > have already tried this? > > The optimal index size is down to the size of segments on disk. New > segments are created when hard commits occur, and existing on-disk > segments may get merged in the background when the segment count gets > too high. Now, if those on-disk segments get too large, copying them > around at merge time can get prohibitive, especially if your index is > changing frequently. > > Splitting such an index into shards is one approach to dealing with this > issue. > > Upayavira >
