Hi All,
I have recently upgraded solr to 8.4.1 and have installed solr as service in
linux machine. Once I start my service, it will be up for 15-18hours and
suddenly stops without us shutting down. In solr.log I found below error. Can
somebody guide me what values should I be increasing in Linux machine?
Earlier, open file limit was not set and now I have increased. Below are my
system configuration for solr:
JVM memory: 8GB
RAM: 32GB
Open file descriptor count: 500000
Ulimit -v - unlimited
Ulimit -m - unlimited
ERROR STACK TRACE:
2020-03-06 12:08:03.071 ERROR (qtp1691185247-21) [ x:product]
o.a.s.s.HttpSolrCall null:java.lang.RuntimeException:
java.lang.OutOfMemoryError: unable to create new native thread
at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:752)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:603)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:505)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at
org.apache.solr.handler.dataimport.DataImporter.runAsync(DataImporter.java:466)
at
org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:205)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
... 36 more
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method) ~[?:1.8.0_242]
at java.lang.Thread.start(Thread.java:717) ~[?:1.8.0_242]
at
org.apache.solr.handler.dataimport.DataImporter.runAsync(DataImporter.java:466)
~[?:?]
at
org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:205)
~[?:?]
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
~[?:?]
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596) ~[?:?]
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
~[?:?]
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
~[?:?]
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
~[?:?]
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
~[?:?]
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
~[jetty-security-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
~[jetty-rewrite-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at org.eclipse.jetty.server.Server.handle(Server.java:505)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
Thanks and Regards,
Srinivas Kashyap
________________________________
DISCLAIMER:
E-mails and attachments from Bamboo Rose, LLC are confidential.
If you are not the intended recipient, please notify the sender immediately by
replying to the e-mail, and then delete it without making copies or using it in
any way.
No representation is made that this email or any attachments are free of
viruses. Virus scanning is recommended and is the responsibility of the
recipient.
Disclaimer
The information contained in this communication from the sender is
confidential. It is intended solely for use by the recipient and others
authorized to receive it. If you are not the recipient, you are hereby notified
that any disclosure, copying, distribution or taking action in relation of the
contents of this information is strictly prohibited and may be unlawful.
This email has been scanned for viruses and malware, and may have been
automatically archived by Mimecast Ltd, an innovator in Software as a Service
(SaaS) for business. Providing a safer and more useful place for your human
generated data. Specializing in; Security, archiving and compliance. To find
out more visit the Mimecast website.
