I don't expect having equal values to make a noticeable difference, except possibly in some corner cases. Setting them equal is mostly for avoiding surprises...
Erick On Thu, Apr 10, 2014 at 7:17 AM, Atanas Atanasov <atanaso...@gmail.com> wrote: > Thanks for the tip, > > I already set the core properties. Now tomcat has only 27 threads after > start up, which is awesome. > Works fine, first search is not noticeably slower than before. > I'll put equal values for Xmx and Xms and see if there will be any > difference. > > Regards, > Atanas > > > On Thu, Apr 10, 2014 at 5:11 PM, Erick Erickson > <erickerick...@gmail.com>wrote: > >> Trying to fit 1,000 cores in 6G of memory is... interesting. That's a >> lot of stuff in a small amount of memory. I hope these cores' indexes >> are tiny. >> >> The lazy-loading bit for cores has a price. The first user in will pay >> the warmup penalty for that core while it loads. This may or may not >> be noticeable but be aware of it. You may or may not want autowarming >> in place. >> >> You can also specify how many cores are kept in memory at one time, >> they go into an LRU cache and are aged out after they serve their last >> outstanding request. >> >> BTW, current Java practice seems to be setting Xmx and Xms to the same >> value, 6G in your case. >> >> Good Luck! >> Erick >> >> On Thu, Apr 10, 2014 at 12:14 AM, Atanas Atanasov <atanaso...@gmail.com> >> wrote: >> > Thanks for the quick responses, >> > I have allocated 1GB min and 6 GB max memory to Java. The cache settings >> > are the default ones (maybe this is a good point to start). >> > All cores share the same schema and config. >> > I'll try setting the >> > loadOnStartup=*false* transient=*true *options for each core and see what >> > will happen. >> > >> > Those are the exceptions from the log files: >> > SEVERE: Servlet.service() for servlet [default] in context with path >> > [/solrt] threw exception >> > java.lang.IllegalStateException: Cannot call sendError() after the >> response >> > has been committed >> > at >> > >> org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:450) >> > at >> > >> org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:695) >> > at >> > >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:383) >> > at >> > >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158) >> > at >> > >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) >> > at >> > >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) >> > at >> > >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) >> > at >> > >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) >> > at >> > >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) >> > at >> > >> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) >> > at >> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) >> > at >> > >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) >> > at >> > >> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) >> > at >> > >> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040) >> > at >> > >> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) >> > at >> > >> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315) >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) >> > at java.lang.Thread.run(Unknown Source) >> > >> > AND >> > >> > SEVERE: null:ClientAbortException: java.net.SocketException: Software >> > caused connection abort: socket write error >> > at >> org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:371) >> > at >> org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:333) >> > at >> > >> org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:101) >> > at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) >> > at sun.nio.cs.StreamEncoder.flush(Unknown Source) >> > at java.io.OutputStreamWriter.flush(Unknown Source) >> > at org.apache.solr.util.FastWriter.flush(FastWriter.java:137) >> > at >> > >> org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:648) >> > at >> > >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:375) >> > at >> > >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158) >> > at >> > >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) >> > at >> > >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) >> > at >> > >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) >> > at >> > >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) >> > at >> > >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) >> > at >> > >> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) >> > at >> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) >> > at >> > >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) >> > at >> > >> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) >> > at >> > >> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040) >> > at >> > >> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) >> > at >> > >> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313) >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) >> > at java.lang.Thread.run(Unknown Source) >> > Caused by: java.net.SocketException: Software caused connection abort: >> > socket write error >> > at java.net.SocketOutputStream.socketWrite0(Native Method) >> > at java.net.SocketOutputStream.socketWrite(Unknown Source) >> > at java.net.SocketOutputStream.write(Unknown Source) >> > at >> > >> org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215) >> > at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480) >> > at >> > >> org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:119) >> > at >> > >> org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:799) >> > at org.apache.coyote.Response.action(Response.java:174) >> > at >> org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:366) >> > ... 24 more >> > >> > >> > >> > On Thu, Apr 10, 2014 at 9:51 AM, Alexandre Rafalovitch >> > <arafa...@gmail.com>wrote: >> > >> >> Are you using all those cores at once? If not, there is a recent >> >> settings to allow solr to load cores on demand. >> >> >> >> If you are using them all, perhaps you need to look into splitting >> >> them to different machines (horizontal scaling). >> >> >> >> What about your caches? How many additional structures you have >> >> configured for each core? How much memory you allocated to the Java >> >> process. You are probably running out of memory and thrashing with a >> >> swap. I am not even sure Java process can access that much memory in >> >> one process. You might be better off running multiple Tomcat/Solr >> >> instances on the same machine with different subsets of cores. >> >> >> >> Regards, >> >> Alex. >> >> P.s. This is general advice, I don't know the specific issues around >> >> that version of Solr/Tomcat. >> >> Personal website: http://www.outerthoughts.com/ >> >> Current project: http://www.solr-start.com/ - Accelerating your Solr >> >> proficiency >> >> >> >> >> >> On Thu, Apr 10, 2014 at 1:40 PM, Atanas Atanasov <atanaso...@gmail.com> >> >> wrote: >> >> > Hi, guys, >> >> > >> >> > I need some help. After updating to SOLR 4.4 the tomcat process is >> >> > consuming about 2GBs of memory, the CPU usage is about 40% after the >> >> start >> >> > for about 10 minutes. However, the bigger problem is, I have about >> 1000 >> >> > cores and seems that for each core a thread is created. The process >> has >> >> > more than 1000 threads and everything is extremely slow. Creating or >> >> > unloading a core even without documents takes about 20 minutes. >> Searching >> >> > is more or less good, but storing also takes a lot. >> >> > Is there some configuration I missed or that I did wrong? There aren't >> >> many >> >> > calls, I use 64 bit tomcat 7, SOLR 4.4, latest 64 bit Java. The >> machine >> >> has >> >> > 24 GBs of RAM, a CPU with 16 cores and is running Windows Server 2008 >> R2. >> >> > Index is uppdated every 30 seconds/10 000 documents. >> >> > I haven't checked the number of threads before the update, because I >> >> didn't >> >> > have to, it was working just fine. Any suggestion will be highly >> >> > appreciated, thank you in advance. >> >> > >> >> > Regards, >> >> > Atanas >> >> >>
