[jira] [Commented] (SOLR-8727) Limit Threadpools by default
[ https://issues.apache.org/jira/browse/SOLR-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287554#comment-15287554 ] ASF GitHub Bot commented on SOLR-8727: -- Github user bjoernhaeuser closed the pull request at: https://github.com/apache/lucene-solr/pull/11 > Limit Threadpools by default > > > Key: SOLR-8727 > URL: https://issues.apache.org/jira/browse/SOLR-8727 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Affects Versions: 5.2.1 >Reporter: Björn Häuser >Assignee: Noble Paul > > Yesterday we had a problem in our prodution cluster, it was running out of > native threads: > {code} > null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create > new native thread > at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:593) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:465) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) > at org.eclipse.jetty.server.Server.handle(Server.java:497) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) > at > org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) > at java.lang.Thread.run(Unknown Source) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:135) > at java.util.concurrent.ExecutorCompletionService.submit(Unknown Source) > at > org.apache.solr.handler.component.HttpShardHandler.submit(HttpShardHandler.java:250) > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:352) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450) > ... 22 more > {code} > After digging a little bit through the source code I found several > ThreadPools which a default maxCoreSize of Integer.MAX_VALUE. I think we > should figure out a better default then this. > Going to create the corresponding pull reuquest on github for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8727) Limit Threadpools by default
[ https://issues.apache.org/jira/browse/SOLR-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15174465#comment-15174465 ] Mark Miller commented on SOLR-8727: --- bq. But I also would say that consuming as much threads as possible from the OS is also not good user behaviour. Indeed, and we would like to limit the number of threads and connections used much more so than today - work in progress :) Some complications abound. SOLR-7344 is close and should be a big step forward though. > Limit Threadpools by default > > > Key: SOLR-8727 > URL: https://issues.apache.org/jira/browse/SOLR-8727 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Affects Versions: 5.2.1 >Reporter: Björn Häuser >Assignee: Noble Paul > > Yesterday we had a problem in our prodution cluster, it was running out of > native threads: > {code} > null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create > new native thread > at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:593) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:465) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) > at org.eclipse.jetty.server.Server.handle(Server.java:497) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) > at > org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) > at java.lang.Thread.run(Unknown Source) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:135) > at java.util.concurrent.ExecutorCompletionService.submit(Unknown Source) > at > org.apache.solr.handler.component.HttpShardHandler.submit(HttpShardHandler.java:250) > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:352) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450) > ... 22 more > {code} > After digging a little bit through the source code I found several > ThreadPools which a default maxCoreSize of Integer.MAX_VALUE. I think we > should figure out a better default then this. > Going to create the corresponding pull reuquest on github for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8727) Limit Threadpools by default
[ https://issues.apache.org/jira/browse/SOLR-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15173376#comment-15173376 ] Björn Häuser commented on SOLR-8727: Done :) Thank you! > Limit Threadpools by default > > > Key: SOLR-8727 > URL: https://issues.apache.org/jira/browse/SOLR-8727 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Affects Versions: 5.2.1 >Reporter: Björn Häuser >Assignee: Noble Paul > > Yesterday we had a problem in our prodution cluster, it was running out of > native threads: > {code} > null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create > new native thread > at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:593) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:465) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) > at org.eclipse.jetty.server.Server.handle(Server.java:497) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) > at > org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) > at java.lang.Thread.run(Unknown Source) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:135) > at java.util.concurrent.ExecutorCompletionService.submit(Unknown Source) > at > org.apache.solr.handler.component.HttpShardHandler.submit(HttpShardHandler.java:250) > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:352) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450) > ... 22 more > {code} > After digging a little bit through the source code I found several > ThreadPools which a default maxCoreSize of Integer.MAX_VALUE. I think we > should figure out a better default then this. > Going to create the corresponding pull reuquest on github for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8727) Limit Threadpools by default
[ https://issues.apache.org/jira/browse/SOLR-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15163064#comment-15163064 ] Noble Paul commented on SOLR-8727: -- Yeah this could be marked as a duplicate of solr-7344 > Limit Threadpools by default > > > Key: SOLR-8727 > URL: https://issues.apache.org/jira/browse/SOLR-8727 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Affects Versions: 5.2.1 >Reporter: Björn Häuser >Assignee: Noble Paul > > Yesterday we had a problem in our prodution cluster, it was running out of > native threads: > {code} > null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create > new native thread > at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:593) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:465) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) > at org.eclipse.jetty.server.Server.handle(Server.java:497) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) > at > org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) > at java.lang.Thread.run(Unknown Source) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:135) > at java.util.concurrent.ExecutorCompletionService.submit(Unknown Source) > at > org.apache.solr.handler.component.HttpShardHandler.submit(HttpShardHandler.java:250) > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:352) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450) > ... 22 more > {code} > After digging a little bit through the source code I found several > ThreadPools which a default maxCoreSize of Integer.MAX_VALUE. I think we > should figure out a better default then this. > Going to create the corresponding pull reuquest on github for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8727) Limit Threadpools by default
[ https://issues.apache.org/jira/browse/SOLR-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15163047#comment-15163047 ] Björn Häuser commented on SOLR-8727: Well, I was never aware of a problem of distributed locks in solr. So I was just aiming at the smallest possible fix to ensure that there are no threadpools which dont have a limit. After reading the linked issue I can see the problem which is imposed by limiting the threadpools. But I also would say that consuming as much threads as possible from the OS is also not good user behaviour. In this scenario (with the linked stack trace) I am not able to see the underlying problem. Sure, most of the threadpools can be limited through configuration :) Should we close this issue here and rather solve 7344? > Limit Threadpools by default > > > Key: SOLR-8727 > URL: https://issues.apache.org/jira/browse/SOLR-8727 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Affects Versions: 5.2.1 >Reporter: Björn Häuser >Assignee: Noble Paul > > Yesterday we had a problem in our prodution cluster, it was running out of > native threads: > {code} > null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create > new native thread > at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:593) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:465) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) > at org.eclipse.jetty.server.Server.handle(Server.java:497) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) > at > org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) > at java.lang.Thread.run(Unknown Source) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:135) > at java.util.concurrent.ExecutorCompletionService.submit(Unknown Source) > at > org.apache.solr.handler.component.HttpShardHandler.submit(HttpShardHandler.java:250) > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:352) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450) > ... 22 more > {code} > After digging a little bit through the source code I found several > ThreadPools which a default maxCoreSize of Integer.MAX_VALUE. I think we > should figure out a better default then this. > Going to create the corresponding pull reuquest on github for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) ---
[jira] [Commented] (SOLR-8727) Limit Threadpools by default
[ https://issues.apache.org/jira/browse/SOLR-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15163024#comment-15163024 ] Mark Miller commented on SOLR-8727: --- I think this is probably the right path for trying to deal with how many threads we currently have to allow: https://issues.apache.org/jira/browse/SOLR-7344 > Limit Threadpools by default > > > Key: SOLR-8727 > URL: https://issues.apache.org/jira/browse/SOLR-8727 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Affects Versions: 5.2.1 >Reporter: Björn Häuser >Assignee: Noble Paul > > Yesterday we had a problem in our prodution cluster, it was running out of > native threads: > {code} > null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create > new native thread > at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:593) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:465) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) > at org.eclipse.jetty.server.Server.handle(Server.java:497) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) > at > org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) > at java.lang.Thread.run(Unknown Source) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:135) > at java.util.concurrent.ExecutorCompletionService.submit(Unknown Source) > at > org.apache.solr.handler.component.HttpShardHandler.submit(HttpShardHandler.java:250) > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:352) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450) > ... 22 more > {code} > After digging a little bit through the source code I found several > ThreadPools which a default maxCoreSize of Integer.MAX_VALUE. I think we > should figure out a better default then this. > Going to create the corresponding pull reuquest on github for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8727) Limit Threadpools by default
[ https://issues.apache.org/jira/browse/SOLR-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15163020#comment-15163020 ] Mark Miller commented on SOLR-8727: --- And this deals with distributed deadlock issues how? > Limit Threadpools by default > > > Key: SOLR-8727 > URL: https://issues.apache.org/jira/browse/SOLR-8727 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Affects Versions: 5.2.1 >Reporter: Björn Häuser >Assignee: Noble Paul > > Yesterday we had a problem in our prodution cluster, it was running out of > native threads: > {code} > null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create > new native thread > at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:593) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:465) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) > at org.eclipse.jetty.server.Server.handle(Server.java:497) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) > at > org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) > at java.lang.Thread.run(Unknown Source) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:135) > at java.util.concurrent.ExecutorCompletionService.submit(Unknown Source) > at > org.apache.solr.handler.component.HttpShardHandler.submit(HttpShardHandler.java:250) > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:352) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450) > ... 22 more > {code} > After digging a little bit through the source code I found several > ThreadPools which a default maxCoreSize of Integer.MAX_VALUE. I think we > should figure out a better default then this. > Going to create the corresponding pull reuquest on github for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8727) Limit Threadpools by default
[ https://issues.apache.org/jira/browse/SOLR-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15162868#comment-15162868 ] Björn Häuser commented on SOLR-8727: How should that look like? I can add this > Limit Threadpools by default > > > Key: SOLR-8727 > URL: https://issues.apache.org/jira/browse/SOLR-8727 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Affects Versions: 5.2.1 >Reporter: Björn Häuser >Assignee: Noble Paul > > Yesterday we had a problem in our prodution cluster, it was running out of > native threads: > {code} > null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create > new native thread > at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:593) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:465) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) > at org.eclipse.jetty.server.Server.handle(Server.java:497) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) > at > org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) > at java.lang.Thread.run(Unknown Source) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:135) > at java.util.concurrent.ExecutorCompletionService.submit(Unknown Source) > at > org.apache.solr.handler.component.HttpShardHandler.submit(HttpShardHandler.java:250) > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:352) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450) > ... 22 more > {code} > After digging a little bit through the source code I found several > ThreadPools which a default maxCoreSize of Integer.MAX_VALUE. I think we > should figure out a better default then this. > Going to create the corresponding pull reuquest on github for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8727) Limit Threadpools by default
[ https://issues.apache.org/jira/browse/SOLR-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15162859#comment-15162859 ] Noble Paul commented on SOLR-8727: -- {{Integer.MAX_VALUE}} is definitely bad. 128 seems like a small value . BTW , it should also be possible to control it through a sys prop > Limit Threadpools by default > > > Key: SOLR-8727 > URL: https://issues.apache.org/jira/browse/SOLR-8727 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Affects Versions: 5.2.1 >Reporter: Björn Häuser >Assignee: Noble Paul > > Yesterday we had a problem in our prodution cluster, it was running out of > native threads: > {code} > null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create > new native thread > at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:593) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:465) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) > at org.eclipse.jetty.server.Server.handle(Server.java:497) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) > at > org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) > at java.lang.Thread.run(Unknown Source) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:135) > at java.util.concurrent.ExecutorCompletionService.submit(Unknown Source) > at > org.apache.solr.handler.component.HttpShardHandler.submit(HttpShardHandler.java:250) > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:352) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450) > ... 22 more > {code} > After digging a little bit through the source code I found several > ThreadPools which a default maxCoreSize of Integer.MAX_VALUE. I think we > should figure out a better default then this. > Going to create the corresponding pull reuquest on github for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8727) Limit Threadpools by default
[ https://issues.apache.org/jira/browse/SOLR-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15162809#comment-15162809 ] ASF GitHub Bot commented on SOLR-8727: -- GitHub user bjoernhaeuser opened a pull request: https://github.com/apache/lucene-solr/pull/11 Limit threadpools by default to 128 Integer.MAX_VALUE could kill the VM instead of providing useful error messages. 128 is a wild guess by myself. Let's discuss a good default value for this :) Jira: https://issues.apache.org/jira/browse/SOLR-8727 You can merge this pull request into a Git repository by running: $ git pull https://github.com/rebuy-de/lucene-solr limit-threadpools Alternatively you can review and apply these changes as the patch at: https://github.com/apache/lucene-solr/pull/11.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #11 commit cff552d1dc4e7bc7892d9521355c651c2ed2ad06 Author: Björn Häuser Date: 2016-02-24T10:44:09Z Limit threadpools by default to 128 Integer.MAX_VALUE could kill the VM instead of providing useful error messages. 128 is a wild guess by myself. Let's discuss a good default value for this :) > Limit Threadpools by default > > > Key: SOLR-8727 > URL: https://issues.apache.org/jira/browse/SOLR-8727 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Affects Versions: 5.2.1 >Reporter: Björn Häuser > > Yesterday we had a problem in our prodution cluster, it was running out of > native threads: > {code} > null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create > new native thread > at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:593) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:465) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) > at org.eclipse.jetty.server.Server.handle(Server.java:497) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) > at > org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) > at java.lang.Thread.run(Unknown Source) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:135) > at java.util.concurrent.ExecutorCompletionService.submit(Unknown Source) > at > org.apache.solr.handler.component.HttpShardHandler.submit(HttpShardHandler.java:250) > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:352) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064) > at org.apache.solr.s