Hi,
We've been running a java server with Jetty(Dropwizard) and the load of
system per node would be ~ 10k QPS as of now.
We ran Load Test, Connection Test and Peak Test(No Stress test yet.), but
couldn't reproduce what's happening on production when we chose to go live
Below is the error that spits out continuosly
[2018-01-05 16:05:00,874] [] [WARN] MSG - o.e.j.u.t.QueuedThreadPool
org.eclipse.jetty.util.thread.QueuedThreadPool@dw{STARTED,4000<=4000<=4000,i=0,q=10000}
rejected CEP:SocketChannelEndPoint@62241285{/127.0.0.1:49625<->/127.
0.0.1:7000,OPEN,fill=FI,flush=-,to=0/30000}{io=1/0,kio=1,
kro=1}->HttpConnection@23c692fe[p=HttpParser{s=START,0 of
0},g=HttpGenerator@544fccad{s=START}]=>HttpChannelOverHttp@2
c630bfe{r=0,c=false,a=IDLE,uri=null}:runFillable:BLOCKING
[2018-01-05 16:05:00,874] [] [WARN] MSG - o.e.j.u.t.s.EatWhatYouKill
java.util.concurrent.RejectedExecutionException:
CEP:SocketChannelEndPoint@62241285{/127.0.0.1:49625<->/127.0.0.1:7000
,OPEN,fill=FI,flush=-,to=0/30000}{io=1/0,kio=1,
kro=1}->HttpConnection@23c692fe[p=HttpParser{s=START,0 of
0},g=HttpGenerator@544fccad{s=START}]=>HttpChannelOverHttp@2
c630bfe{r=0,c=false,a=IDLE,uri=null}:runFillable:BLOCKING
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(Queue
dThreadPool.java:385)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProd
uce(EatWhatYouKill.java:249)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produc
e(EatWhatYouKill.java:140)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(Ea
tWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$Reserve
dThread.run(ReservedThreadExecutor.java:243)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued
ThreadPool.java:679)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedT
hreadPool.java:597)
at java.lang.Thread.run(Thread.java:745)
Tried to find an answer, but couldn't. Can anyone help. At this point the
server stops accepting new connections and doesn't serve new http requests.
But Jetty QueuedThreadPool Metrics show that thread Utilization is 100%
Thanks
_______________________________________________
jetty-users mailing list
[email protected]
To change your delivery options, retrieve your password, or unsubscribe from
this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
