Hi Atanas, I have a question that, how do know that how much thread does the tomcat has?
Thanks Aman Tandon On Thu, Apr 10, 2014 at 7:53 PM, Erick Erickson <erickerick...@gmail.com>wrote: > I don't expect having equal values to make a noticeable difference, > except possibly in some corner cases. Setting them equal is mostly for > avoiding surprises... > > Erick > > On Thu, Apr 10, 2014 at 7:17 AM, Atanas Atanasov <atanaso...@gmail.com> > wrote: > > Thanks for the tip, > > > > I already set the core properties. Now tomcat has only 27 threads after > > start up, which is awesome. > > Works fine, first search is not noticeably slower than before. > > I'll put equal values for Xmx and Xms and see if there will be any > > difference. > > > > Regards, > > Atanas > > > > > > On Thu, Apr 10, 2014 at 5:11 PM, Erick Erickson <erickerick...@gmail.com > >wrote: > > > >> Trying to fit 1,000 cores in 6G of memory is... interesting. That's a > >> lot of stuff in a small amount of memory. I hope these cores' indexes > >> are tiny. > >> > >> The lazy-loading bit for cores has a price. The first user in will pay > >> the warmup penalty for that core while it loads. This may or may not > >> be noticeable but be aware of it. You may or may not want autowarming > >> in place. > >> > >> You can also specify how many cores are kept in memory at one time, > >> they go into an LRU cache and are aged out after they serve their last > >> outstanding request. > >> > >> BTW, current Java practice seems to be setting Xmx and Xms to the same > >> value, 6G in your case. > >> > >> Good Luck! > >> Erick > >> > >> On Thu, Apr 10, 2014 at 12:14 AM, Atanas Atanasov <atanaso...@gmail.com > > > >> wrote: > >> > Thanks for the quick responses, > >> > I have allocated 1GB min and 6 GB max memory to Java. The cache > settings > >> > are the default ones (maybe this is a good point to start). > >> > All cores share the same schema and config. > >> > I'll try setting the > >> > loadOnStartup=*false* transient=*true *options for each core and see > what > >> > will happen. > >> > > >> > Those are the exceptions from the log files: > >> > SEVERE: Servlet.service() for servlet [default] in context with path > >> > [/solrt] threw exception > >> > java.lang.IllegalStateException: Cannot call sendError() after the > >> response > >> > has been committed > >> > at > >> > > >> > org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:450) > >> > at > >> > > >> > org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:695) > >> > at > >> > > >> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:383) > >> > at > >> > > >> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158) > >> > at > >> > > >> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > >> > at > >> > > >> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > >> > at > >> > > >> > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) > >> > at > >> > > >> > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) > >> > at > >> > > >> > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) > >> > at > >> > > >> > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) > >> > at > >> > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) > >> > at > >> > > >> > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) > >> > at > >> > > >> > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) > >> > at > >> > > >> > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040) > >> > at > >> > > >> > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) > >> > at > >> > > >> > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315) > >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > >> > at java.lang.Thread.run(Unknown Source) > >> > > >> > AND > >> > > >> > SEVERE: null:ClientAbortException: java.net.SocketException: Software > >> > caused connection abort: socket write error > >> > at > >> > org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:371) > >> > at > >> org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:333) > >> > at > >> > > >> > org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:101) > >> > at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) > >> > at sun.nio.cs.StreamEncoder.flush(Unknown Source) > >> > at java.io.OutputStreamWriter.flush(Unknown Source) > >> > at org.apache.solr.util.FastWriter.flush(FastWriter.java:137) > >> > at > >> > > >> > org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:648) > >> > at > >> > > >> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:375) > >> > at > >> > > >> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158) > >> > at > >> > > >> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > >> > at > >> > > >> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > >> > at > >> > > >> > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) > >> > at > >> > > >> > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) > >> > at > >> > > >> > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) > >> > at > >> > > >> > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) > >> > at > >> > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) > >> > at > >> > > >> > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) > >> > at > >> > > >> > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) > >> > at > >> > > >> > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040) > >> > at > >> > > >> > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) > >> > at > >> > > >> > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313) > >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > >> > at java.lang.Thread.run(Unknown Source) > >> > Caused by: java.net.SocketException: Software caused connection abort: > >> > socket write error > >> > at java.net.SocketOutputStream.socketWrite0(Native Method) > >> > at java.net.SocketOutputStream.socketWrite(Unknown Source) > >> > at java.net.SocketOutputStream.write(Unknown Source) > >> > at > >> > > >> > org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215) > >> > at > org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480) > >> > at > >> > > >> > org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:119) > >> > at > >> > > >> > org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:799) > >> > at org.apache.coyote.Response.action(Response.java:174) > >> > at > >> > org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:366) > >> > ... 24 more > >> > > >> > > >> > > >> > On Thu, Apr 10, 2014 at 9:51 AM, Alexandre Rafalovitch > >> > <arafa...@gmail.com>wrote: > >> > > >> >> Are you using all those cores at once? If not, there is a recent > >> >> settings to allow solr to load cores on demand. > >> >> > >> >> If you are using them all, perhaps you need to look into splitting > >> >> them to different machines (horizontal scaling). > >> >> > >> >> What about your caches? How many additional structures you have > >> >> configured for each core? How much memory you allocated to the Java > >> >> process. You are probably running out of memory and thrashing with a > >> >> swap. I am not even sure Java process can access that much memory in > >> >> one process. You might be better off running multiple Tomcat/Solr > >> >> instances on the same machine with different subsets of cores. > >> >> > >> >> Regards, > >> >> Alex. > >> >> P.s. This is general advice, I don't know the specific issues around > >> >> that version of Solr/Tomcat. > >> >> Personal website: http://www.outerthoughts.com/ > >> >> Current project: http://www.solr-start.com/ - Accelerating your Solr > >> >> proficiency > >> >> > >> >> > >> >> On Thu, Apr 10, 2014 at 1:40 PM, Atanas Atanasov < > atanaso...@gmail.com> > >> >> wrote: > >> >> > Hi, guys, > >> >> > > >> >> > I need some help. After updating to SOLR 4.4 the tomcat process is > >> >> > consuming about 2GBs of memory, the CPU usage is about 40% after > the > >> >> start > >> >> > for about 10 minutes. However, the bigger problem is, I have about > >> 1000 > >> >> > cores and seems that for each core a thread is created. The process > >> has > >> >> > more than 1000 threads and everything is extremely slow. Creating > or > >> >> > unloading a core even without documents takes about 20 minutes. > >> Searching > >> >> > is more or less good, but storing also takes a lot. > >> >> > Is there some configuration I missed or that I did wrong? There > aren't > >> >> many > >> >> > calls, I use 64 bit tomcat 7, SOLR 4.4, latest 64 bit Java. The > >> machine > >> >> has > >> >> > 24 GBs of RAM, a CPU with 16 cores and is running Windows Server > 2008 > >> R2. > >> >> > Index is uppdated every 30 seconds/10 000 documents. > >> >> > I haven't checked the number of threads before the update, because > I > >> >> didn't > >> >> > have to, it was working just fine. Any suggestion will be highly > >> >> > appreciated, thank you in advance. > >> >> > > >> >> > Regards, > >> >> > Atanas > >> >> > >> >
