Hi, On Fri, Jan 5, 2018 at 6:50 PM, Sarath Prabath Redlapalli Jaya <[email protected]> wrote: > Hi, > > We've been running a java server with Jetty(Dropwizard) and the load of > system per node would be ~ 10k QPS as of now. > > We ran Load Test, Connection Test and Peak Test(No Stress test yet.), but > couldn't reproduce what's happening on production when we chose to go live > > Below is the error that spits out continuosly > > [2018-01-05 16:05:00,874] [] [WARN] MSG - o.e.j.u.t.QueuedThreadPool > org.eclipse.jetty.util.thread.QueuedThreadPool@dw{STARTED,4000<=4000<=4000,i=0,q=10000} > rejected > CEP:SocketChannelEndPoint@62241285{/127.0.0.1:49625<->/127.0.0.1:7000,OPEN,fill=FI,flush=-,to=0/30000}{io=1/0,kio=1,kro=1}->HttpConnection@23c692fe[p=HttpParser{s=START,0 > of > 0},g=HttpGenerator@544fccad{s=START}]=>HttpChannelOverHttp@2c630bfe{r=0,c=false,a=IDLE,uri=null}:runFillable:BLOCKING > [2018-01-05 16:05:00,874] [] [WARN] MSG - o.e.j.u.t.s.EatWhatYouKill > java.util.concurrent.RejectedExecutionException: > CEP:SocketChannelEndPoint@62241285{/127.0.0.1:49625<->/127.0.0.1:7000,OPEN,fill=FI,flush=-,to=0/30000}{io=1/0,kio=1,kro=1}->HttpConnection@23c692fe[p=HttpParser{s=START,0 > of > 0},g=HttpGenerator@544fccad{s=START}]=>HttpChannelOverHttp@2c630bfe{r=0,c=false,a=IDLE,uri=null}:runFillable:BLOCKING > at > org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:385) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:249) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) > at > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:243) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:679) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:597) > at java.lang.Thread.run(Thread.java:745) > > Tried to find an answer, but couldn't. Can anyone help. At this point the > server stops accepting new connections and doesn't serve new http requests. > > But Jetty QueuedThreadPool Metrics show that thread Utilization is 100%
What Jetty version ? Have you tried 9.4.8 ? Are you using a bounded thread pool ? If it replicates with 9.4.8, can you take a server dump as explained here: https://www.eclipse.org/jetty/documentation/9.4.x/jetty-dump-tool.html -- Simone Bordet ---- http://cometd.org http://webtide.com Developer advice, training, services and support from the Jetty & CometD experts. _______________________________________________ jetty-users mailing list [email protected] To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/jetty-users
