Re: Tomcat creates a thread for each SOLR core
Hi to all, I have some other information that might be useful. Installing previous java did not work. The creation of a core does not take 20 minutes, it seems that it happens right away but there is no response and looks stuck. After I restart the apache tomcat the core is there and working without any problems. In the SOLR admin panel this is what thread dump shows: http-nio-8080-exec-2 (27) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@53dbcec - sun.misc.Unsafe.park(Native Method) - java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) - java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) - org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85) - org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31) - java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) - java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) - java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) - java.lang.Thread.run(Thread.java:744) 21091.3352ms 20420.5309ms and - sun.management.ThreadImpl.getThreadInfo1(Native Method) - sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:172) - org.apache.solr.handler.admin.ThreadDumpHandler.handleRequestBody(ThreadDumpHandler.java:69) - org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) - org.apache.solr.core.SolrCore.execute(SolrCore.java:1904) - org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:659) - org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362) - org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158) - org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) - org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) - org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) - org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) - org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:136) - org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:74) - org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610) - org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) - org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:526) - org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1017) - org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:652) - org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:222) - org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1575) - org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1533) - java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) - java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) - java.lang.Thread.run(Thread.java:744) they both seem to be stuck. Just a clarification if someone decides to reproduce it: 1. 64 bit apache tomcat installation (7 or 8) - solr 4.4 deployed, any JVM. 2. Create 1000 cores. 3. Restart Apache Tomcat. 4. Create 1 core. I hope I explained it well, if not just ask. Thanks in advance, Atanas On Tue, Apr 15, 2014 at 9:58 AM, Atanas Atanasov atanaso...@gmail.comwrote: Hello again, Current situation is, after setting the two options in order not to load the cores on start up and ramBufferSizeMB=32 Tomcat is stable, responsive, threads reach 60 as a maximum. Browsing and storing are fast. I should note that I have many cores with small amount of documents. Unfortunately the problem with the creation of a new core taking 20 minutes still exists. Next step will be downgrading to Java 7u25. Any other suggestions will be highly appreciated. Thanks in advance. P.S previous SOLR version from which I updated was 3.6. Regards, Atanas Atanasov
Re: Tomcat creates a thread for each SOLR core
Hello again, Current situation is, after setting the two options in order not to load the cores on start up and ramBufferSizeMB=32 Tomcat is stable, responsive, threads reach 60 as a maximum. Browsing and storing are fast. I should note that I have many cores with small amount of documents. Unfortunately the problem with the creation of a new core taking 20 minutes still exists. Next step will be downgrading to Java 7u25. Any other suggestions will be highly appreciated. Thanks in advance. P.S previous SOLR version from which I updated was 3.6. Regards, Atanas Atanasov On Thu, Apr 10, 2014 at 6:06 PM, Shawn Heisey s...@elyograg.org wrote: On 4/10/2014 12:40 AM, Atanas Atanasov wrote: I need some help. After updating to SOLR 4.4 the tomcat process is consuming about 2GBs of memory, the CPU usage is about 40% after the start for about 10 minutes. However, the bigger problem is, I have about 1000 cores and seems that for each core a thread is created. The process has more than 1000 threads and everything is extremely slow. Creating or unloading a core even without documents takes about 20 minutes. Searching is more or less good, but storing also takes a lot. Is there some configuration I missed or that I did wrong? There aren't many calls, I use 64 bit tomcat 7, SOLR 4.4, latest 64 bit Java. The machine has 24 GBs of RAM, a CPU with 16 cores and is running Windows Server 2008 R2. Index is uppdated every 30 seconds/10 000 documents. I haven't checked the number of threads before the update, because I didn't have to, it was working just fine. Any suggestion will be highly appreciated, thank you in advance. If creating a core takes 20 minutes, that sounds to me like the JVM is doing constant full garbage collections to free up enough memory for basic system operation. It could also be explained by temporary work threads having to wait to execute because the servlet container will not allow them to run. When indexing is happening, each core will set aside some memory for buffering index updates. By default, the value of ramBufferSizeMB is 100. If all your cores are indexing at once, multiply the indexing buffer by 1000, and you'll require 100GB of heap memory. You'll need to greatly reduce that buffer size. This buffer was 32MB by default in 4.0 and earlier. If you are not setting this value, this change sounds like it might fully explain what you are seeing. https://issues.apache.org/jira/browse/SOLR-4074 What version did you upgrade from? Solr 4.x is a very different beast than earlier major versions. I believe there may have been some changes made to reduce memory usage in versions after 4.4.0. The jetty that comes with Solr is configured to allow 10,000 threads. Most people don't have that many, even on a temporary basis, but bad things happen when the servlet container will not allow Solr to start as many as it requires. I believe that the typical default maxThreads value you'll find in a servlet container config is 200. Erick's right about a 6GB heap being very small for what you are trying to do. Putting 1000 cores on one machine is something I would never try. If it became a requirement I had to deal with, I wouldn't try it unless the machine had a lot more CPU cores, hundreds of gigabytes of RAM, and a lot of extremely fast disk space. If this worked before a Solr upgrade, I'm amazed. Congratulations to you for fine work! NB: Oracle Java 7u25 is what you should be using. 7u40 through 7u51 have known bugs that affect Solr/Lucene. These should be fixed by 7u60. A pre-release of that is available now, and it should be generally available in May 2014. Thanks, Shawn
Tomcat creates a thread for each SOLR core
Hi, guys, I need some help. After updating to SOLR 4.4 the tomcat process is consuming about 2GBs of memory, the CPU usage is about 40% after the start for about 10 minutes. However, the bigger problem is, I have about 1000 cores and seems that for each core a thread is created. The process has more than 1000 threads and everything is extremely slow. Creating or unloading a core even without documents takes about 20 minutes. Searching is more or less good, but storing also takes a lot. Is there some configuration I missed or that I did wrong? There aren't many calls, I use 64 bit tomcat 7, SOLR 4.4, latest 64 bit Java. The machine has 24 GBs of RAM, a CPU with 16 cores and is running Windows Server 2008 R2. Index is uppdated every 30 seconds/10 000 documents. I haven't checked the number of threads before the update, because I didn't have to, it was working just fine. Any suggestion will be highly appreciated, thank you in advance. Regards, Atanas
Re: Tomcat creates a thread for each SOLR core
...@gmail.comwrote: Are you using all those cores at once? If not, there is a recent settings to allow solr to load cores on demand. If you are using them all, perhaps you need to look into splitting them to different machines (horizontal scaling). What about your caches? How many additional structures you have configured for each core? How much memory you allocated to the Java process. You are probably running out of memory and thrashing with a swap. I am not even sure Java process can access that much memory in one process. You might be better off running multiple Tomcat/Solr instances on the same machine with different subsets of cores. Regards, Alex. P.s. This is general advice, I don't know the specific issues around that version of Solr/Tomcat. Personal website: http://www.outerthoughts.com/ Current project: http://www.solr-start.com/ - Accelerating your Solr proficiency On Thu, Apr 10, 2014 at 1:40 PM, Atanas Atanasov atanaso...@gmail.com wrote: Hi, guys, I need some help. After updating to SOLR 4.4 the tomcat process is consuming about 2GBs of memory, the CPU usage is about 40% after the start for about 10 minutes. However, the bigger problem is, I have about 1000 cores and seems that for each core a thread is created. The process has more than 1000 threads and everything is extremely slow. Creating or unloading a core even without documents takes about 20 minutes. Searching is more or less good, but storing also takes a lot. Is there some configuration I missed or that I did wrong? There aren't many calls, I use 64 bit tomcat 7, SOLR 4.4, latest 64 bit Java. The machine has 24 GBs of RAM, a CPU with 16 cores and is running Windows Server 2008 R2. Index is uppdated every 30 seconds/10 000 documents. I haven't checked the number of threads before the update, because I didn't have to, it was working just fine. Any suggestion will be highly appreciated, thank you in advance. Regards, Atanas
Re: Tomcat creates a thread for each SOLR core
Thanks for the tip, I already set the core properties. Now tomcat has only 27 threads after start up, which is awesome. Works fine, first search is not noticeably slower than before. I'll put equal values for Xmx and Xms and see if there will be any difference. Regards, Atanas On Thu, Apr 10, 2014 at 5:11 PM, Erick Erickson erickerick...@gmail.comwrote: Trying to fit 1,000 cores in 6G of memory is... interesting. That's a lot of stuff in a small amount of memory. I hope these cores' indexes are tiny. The lazy-loading bit for cores has a price. The first user in will pay the warmup penalty for that core while it loads. This may or may not be noticeable but be aware of it. You may or may not want autowarming in place. You can also specify how many cores are kept in memory at one time, they go into an LRU cache and are aged out after they serve their last outstanding request. BTW, current Java practice seems to be setting Xmx and Xms to the same value, 6G in your case. Good Luck! Erick On Thu, Apr 10, 2014 at 12:14 AM, Atanas Atanasov atanaso...@gmail.com wrote: Thanks for the quick responses, I have allocated 1GB min and 6 GB max memory to Java. The cache settings are the default ones (maybe this is a good point to start). All cores share the same schema and config. I'll try setting the loadOnStartup=*false* transient=*true *options for each core and see what will happen. Those are the exceptions from the log files: SEVERE: Servlet.service() for servlet [default] in context with path [/solrt] threw exception java.lang.IllegalStateException: Cannot call sendError() after the response has been committed at org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:450) at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:695) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:383) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) AND SEVERE: null:ClientAbortException: java.net.SocketException: Software caused connection abort: socket write error at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:371) at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:333) at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:101) at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) at sun.nio.cs.StreamEncoder.flush(Unknown Source) at java.io.OutputStreamWriter.flush(Unknown Source) at org.apache.solr.util.FastWriter.flush(FastWriter.java:137) at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:648) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:375) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service
Re: Tomcat creates a thread for each SOLR core
Hi, I see the threads of the tomcat7.exe process in the Windows Task manager. Regards, Atanas Atanasov On Thu, Apr 10, 2014 at 5:28 PM, Aman Tandon amantandon...@gmail.comwrote: Hi Atanas, I have a question that, how do know that how much thread does the tomcat has? Thanks Aman Tandon On Thu, Apr 10, 2014 at 7:53 PM, Erick Erickson erickerick...@gmail.com wrote: I don't expect having equal values to make a noticeable difference, except possibly in some corner cases. Setting them equal is mostly for avoiding surprises... Erick On Thu, Apr 10, 2014 at 7:17 AM, Atanas Atanasov atanaso...@gmail.com wrote: Thanks for the tip, I already set the core properties. Now tomcat has only 27 threads after start up, which is awesome. Works fine, first search is not noticeably slower than before. I'll put equal values for Xmx and Xms and see if there will be any difference. Regards, Atanas On Thu, Apr 10, 2014 at 5:11 PM, Erick Erickson erickerick...@gmail.com wrote: Trying to fit 1,000 cores in 6G of memory is... interesting. That's a lot of stuff in a small amount of memory. I hope these cores' indexes are tiny. The lazy-loading bit for cores has a price. The first user in will pay the warmup penalty for that core while it loads. This may or may not be noticeable but be aware of it. You may or may not want autowarming in place. You can also specify how many cores are kept in memory at one time, they go into an LRU cache and are aged out after they serve their last outstanding request. BTW, current Java practice seems to be setting Xmx and Xms to the same value, 6G in your case. Good Luck! Erick On Thu, Apr 10, 2014 at 12:14 AM, Atanas Atanasov atanaso...@gmail.com wrote: Thanks for the quick responses, I have allocated 1GB min and 6 GB max memory to Java. The cache settings are the default ones (maybe this is a good point to start). All cores share the same schema and config. I'll try setting the loadOnStartup=*false* transient=*true *options for each core and see what will happen. Those are the exceptions from the log files: SEVERE: Servlet.service() for servlet [default] in context with path [/solrt] threw exception java.lang.IllegalStateException: Cannot call sendError() after the response has been committed at org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:450) at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:695) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:383) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) AND SEVERE: null:ClientAbortException: java.net.SocketException: Software caused connection abort: socket write error at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:371) at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:333) at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:101) at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) at sun.nio.cs.StreamEncoder.flush(Unknown Source) at java.io.OutputStreamWriter.flush(Unknown Source) at org.apache.solr.util.FastWriter.flush(FastWriter.java:137) at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:648
