Re: Mem allocation - SOLR vs OS

2011-01-20 Thread Salman Akram
I will be looking into JConsole. One more question regarding caching. When we talk about warm-up queries does that mean that some of the complex queries (esp those which require high I/O e.g. phrase queries) will really be very slow (on lets say an index of 200GB) if they are not cached? I am

Re: Mem allocation - SOLR vs OS

2011-01-20 Thread Otis Gospodnetic
, 2011 7:26:39 AM Subject: Re: Mem allocation - SOLR vs OS I will be looking into JConsole. One more question regarding caching. When we talk about warm-up queries does that mean that some of the complex queries (esp those which require high I/O e.g. phrase queries) will really be very slow

Re: Mem allocation - SOLR vs OS

2011-01-19 Thread Erick Erickson
You're better off using two cores on the same Solr instance rather than two instances of Tomcat, that way you avoid some overhead. The usual advice is to monitor the Solr caches, particularly for evictions and size the Solr caches accordingly. You can see these from the admin/stats page and also

Re: Mem allocation - SOLR vs OS

2011-01-19 Thread Salman Akram
Actually we don't have much load on the server (like the usage currently is quite low) but user queries are very complex e.g. long phrases/multiple proximity/wildcard etc so I know these values need to be tried out but I wanted to see whats the right 'start' so that I am not way off. Also

Re: Mem allocation - SOLR vs OS

2011-01-19 Thread Markus Jelsma
You only need so much for Solr so it can do its thing. Faceting can take quite some memory on a large index but sorting can be a really big RAM consumer. As Erick pointed out, inspect and tune the cache settings and adjust RAM allocated to the JVM if required. Using tools like JConsole you can

Re: Mem allocation - SOLR vs OS

2011-01-19 Thread Salman Akram
We do have sorting but not faceting. OK so I guess there is no 'hard and fast rule' as such so I will play with it and see. Thanks for the help On Wed, Jan 19, 2011 at 11:48 PM, Markus Jelsma markus.jel...@openindex.iowrote: You only need so much for Solr so it can do its thing. Faceting can

Re: Mem allocation - SOLR vs OS

2011-01-19 Thread Markus Jelsma
Sorting on field X will build an array of the size of maxDoc. The data type equals the one used by the field you're sorting on. Also, if you have a very high amount of deletes per update it might be a good idea to optimize as well since it reduces maxDoc to the number of documents that actually

Mem allocation - SOLR vs OS

2011-01-18 Thread Salman Akram
Hi, I know this is a subjective topic but from what I have read it seems more RAM should be spared for OS caching and much less for SOLR/Tomcat even on a dedicated SOLR server. Can someone give me an idea about the theoretically ideal proportion b/w them for a dedicated Windows server with 32GB

Re: Mem allocation - SOLR vs OS

2011-01-18 Thread Salman Akram
In case it helps there are two SOLR indexes (160GB and 700GB) on the machine. Also these are separate indexes and not shards so would it help to put them on two separate Tomcat servers on same machine? This way I think one index won't be affecting others cache. On Wed, Jan 19, 2011 at 12:00 PM,