543M documents? On a single machine? How big is the index anyway? I think you're running up against physical memory limitations, the number of segments is a red herring. You're at a point where you need to shard your index to multiple machines I'd guess. Or perhaps that point was some time ago <G>.....
Best Erick On Thu, Mar 17, 2011 at 2:39 PM, danomano <dshopk...@earthlink.net> wrote: > Hi folks, I ran into problem today where I am no longer able to execute any > queries :( due to Out of Memory issues. > > I am in the process of investigating the use of different mergeFactors, or > even different merge policies all together. > My question is if I have many segments (i.e. smaller sized segments), will > that also reduce the total memory in RAM required for searching? (my System > is currently allocated 8GB ram and has a ~255GB index). (I'm not fully up > on the 'default merge policy' but I believe with a mergeFactor of 10, that > would mean each segment should be approaching about 25Gb? with ~543 million > documents > > of note: this is all running on 1 server. > > As seen below. > > SEVERE: java.lang.OutOfMemoryError: Java heap space > at > org.apache.lucene.search.cache.LongValuesCreator.fillLongValues(LongValuesCreator.java:141) > at > org.apache.lucene.search.cache.LongValuesCreator.validate(LongValuesCreator.java:84) > at > org.apache.lucene.search.cache.LongValuesCreator.create(LongValuesCreator.java:74) > at > org.apache.lucene.search.cache.LongValuesCreator.create(LongValuesCreator.java:37) > at > org.apache.lucene.search.FieldCacheImpl$Cache.createValue(FieldCacheImpl.java:155) > at > org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:188) > at > org.apache.lucene.search.FieldCacheImpl.getLongs(FieldCacheImpl.java:337) > at > org.apache.lucene.search.FieldComparator$LongComparator.setNextReader(FieldComparator.java:504) > at > org.apache.lucene.search.TopFieldCollector$OneComparatorNonScoringCollector.setNextReader(TopFieldCollector.java:97) > at > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:207) > at org.apache.lucene.search.Searcher.search(Searcher.java:101) > at > org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1389) > at > org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1285) > at > org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:344) > at > org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:273) > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:210) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1324) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:337) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:240) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) > at > com.openmarket.servletfilters.LogToCSVFilter.doFilter(LogToCSVFilter.java:89) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) > at > com.openmarket.servletfilters.GZipAutoDeflateFilter.doFilter(GZipAutoDeflateFilter.java:66) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) > ...etc > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Segments-and-Memory-Correlate-tp2694747p2694747.html > Sent from the Solr - User mailing list archive at Nabble.com. >
