On Mon, 2015-11-02 at 14:34 +0530, Modassar Ather wrote: > No! This is a single big machine with 12 shards on it. > Around 370 gb on the single machine.
Okay. I guess your observation of 400% for a single core is with top and looking at that core's entry? If so, the 400% can be explained by excessive garbage collection. You could turn GC-logging on to check that. With a bit of luck GC would be the cause of the slow down. > Few simple queries are returned with in a couple of seconds. But the > more complex queries with proximity and wild cards have taken more > than 3-4 minutes and some times some queries have timed out too where > time out is set to 5 minutes. The proximity information seems relevant here. > - How many results do you return (the rows-parameter)? > It is the default one 10. Grouping is enabled on a field. If you have group.ngroups=true that would be heavy (and require a lot of memory), but as your non-wildcard searches with many hits are fast, that is probably not the problem here. Toke: > If you are using spinning drives and only have 32GB of RAM in total in > each machine, you are probably struggling just to keep things running. > > As mentioned above this is a big machine with 370+ gb of RAM and Solr > (12 nodes total) is assigned 336 GB. The rest is still a good for > other system activities. Assuming the storage is spinning drives, it is quite a small machine, measured by cache memory vs. index size: You have 30-40GB free for disk cache and your index is 1TB, so ~3%. Unless you have a great deal of stored content, 3% for disk caching means that there will be a high amount of IO during a search. It works for you when the queries are simple field:term, but I am not surprised that it doesn't work well in other cases. By nature, truncated queries touches a lot of terms, which means a lot of lookups. I have no in-depth knowledge on how these lookups are performed, but I guesstimate that it involves IO-intensive lookups. Coincidentally we also run a machine with multiple Solrs, terabytes of index data and not much memory (< 1%) for disk cache. One difference being that it is backed by SSDs. I tried doing a few ad-hoc searches with grouping turned on (search terms are Danish words): q=ostekiks 38,646 hits, 530 ms. q=ost* 49,713,655 hits, 2,190 ms. q=køer mælk 1,232,445 hits, 767 ms. q=kat mad* 10,926,107 hits, 4624 ms. q="kaniner harer"~50 161,009 hits, 726 ms. q=kantarel 337,279 hits, 455 ms. q=deres kan* 245,719,036 hits, 13,565 ms. This was with Solr 4.10. No special garbage collection activity occurred. Heap usage stayed well below 8GB per Solr, which is the standard behaviour of our system. In short, I could not replicate your observed special activity based on the queries you have described. I have no reason to believe that Solr 5.3 should perform worse in this aspect. The SSDs are probably part of the explanation, but I suspect we are missing something else. It should not make a difference (as your non-truncated queries are fast), but could you try to reduce the slow request to the simplest possible? No grouping, faceting or other special processing, just q=network se* - Toke Eskildsen, State and University Library, Denmark
