Set up your JVM to produce the heap dumps in case of OOM and try to analyze them with a profiler like YourKit. This could give you some ideas on what takes memory and what potentially could be reduced. Sometimes the cache settings could be adjusted without significant performance toll etc. See what on query side and on indexing side could be downsized. In some case you might need to modify the Lucene source code to adjust the internal cache I/O buffers size, for example. But look for low hanging fruits first. Use 32-bit JVM if possible, of course.
-Alexander On Mon, Sep 20, 2010 at 5:58 AM, Kjetil Ødegaard <kjetil.odega...@gmail.com> wrote: > On Thu, Sep 16, 2010 at 11:48 AM, Peter Karich <peat...@yahoo.de> wrote: > >> Hi Kjetil, >> >> is this custom component (which performes groub by + calcs stats) >> somewhere available? >> I would like to do something similar. Would you mind to share if it >> isn't already available? >> >> The grouping stuff sounds similar to >> https://issues.apache.org/jira/browse/SOLR-236 >> >> where you can have mem problems too ;-) or see: >> https://issues.apache.org/jira/browse/SOLR-1682 >> >> > Thanks for the links! These patches seem to provide somewhat similar > functionality, I'll investigate if they're implemented in a similar way too. > > We've developed this component for a client, so while I'd like to share it I > can't make any promises. Sorry. > > >> > Any tips or similar experiences? >> >> you want to decrease memory usage? > > > Yes. Specifically, I would like to keep the heap at 4 GB. Unfortunately I'm > still seeing some OutOfMemoryErrors so I might have to up the heap size > again. > > I guess what I'm really wondering is if there's a way to keep memory use > down, while at the same time not sacrificing the performance of our queries. > The queries have to run through all values for a field in order to calculate > the sum, so it's not enough to just cache a few values. > > The code which fetches values from the index uses > FieldCache.DEFAULT.getStringIndex for a field, and then indexes like this: > > FieldType fieldType = searcher.getSchema().getFieldType(fieldName); > fieldType.indexedToReadable(stringIndex.lookup[stringIndex.order[documentId]]); > > Is there a better way to do this? Thanks. > > > ---Kjetil >
