Hi Shawn, Thanks for your reply. After following your suggestions we were able to index 30k documents. I have some queries:
1) What is stored in the RAM while only indexing is going on? How to calculate the RAM/heap requirements for our documents? 2) The document cache, filter cache, etc...are populated while querying. Correct me if I am wrong. Are there any caches that are populated while indexing? Thanks, Rahul On Sat, Jan 26, 2013 at 11:46 PM, Shawn Heisey <s...@elyograg.org> wrote: > On 1/26/2013 12:55 AM, Rahul Bishnoi wrote: > >> Thanks for quick reply and addressing each point queried. >> >> Additional asked information is mentioned below: >> >> OS = Ubuntu 12.04 (64 bit) >> Sun Java 7 (64 bit) >> Total RAM = 8GB >> >> SolrConfig.xml is available at http://pastebin.com/SEFxkw2R >> >> > Rahul, > > The MaxPermGenSize could be a contributing factor. The documents where > you have 1000 words are somewhat large, though your overall index size is > pretty small. I would try removing the MaxPermGenSize option and see what > happens. You can also try reducing the ramBufferSizeMB in solrconfig.xml. > The default in previous versions of Solr was 32, which is big enough for > most things, unless you are indexing HUGE documents like entire books. > > It looks like you have the cache sizes under <query> at values close to > default. I wouldn't decrease the documentCache any - in fact an increase > might be a good thing there. As for the others, you could probably reduce > them. The filterCache size I would start at 64 or 128. Watch your cache > hitratios to see whether the changes make things remarkably worse. > > If that doesn't help, try increasing the -Xmx option - first 3072m, next > 4096m. You could go as high as 6GB and not run into any OS cache problems > with your small index size, though you might run into long GC pauses. > > Indexing, especially big documents, is fairly memory intensive. Some > queries can be memory intensive as well, especially those using facets or a > lot of clauses. > > Under normal operation, I could probably get away with a 3GB heap size, > but I have it at 8GB because otherwise a full reindex (full-import from > mysql) runs into OOM errors. > > Thanks, > Shawn > >
