Jesse Barnum <jsb_tom...@360works.com> wrote: Thanks for such a well written question. All the relevant information is available and presented clearly and logically.
>Problem summary: >My nio polling threads are using too much CPU time. Are you sure that is the real problem? It sounds like the occasional slowness is the problem and the polling threads are the suspected - emphasis on the suspected - root cause. >Application overview: >My application has from 1,300 - 4,000 users connected at any given >time. Each user sends about 200 bytes, then waits 30 seconds, then >sends about 200 bytes, and this just loops for each user. >Each user connects with SSL, and we use a long keepalive to ensure that >the HTTP connection doesn't close, so that we don't have to renegotiate >SSL. How long should the application take to process those 200 bytes? I'm wondering what the peak number of concurrent requests, rather than connections, might be. >Configuration: >Ubuntu 12.0.4 with Tomcat 7.0.35, 1.75 gigs of RAM. > >We are using Apache with SSL and mod_proxy_ajp to forward requests to >Tomcat. It has MPM module enabled, with 500 ThreadsPerChild, so we >typically have from 3-9 Apache instances running. > >> <IfModule mpm_worker_module> >> ServerLimit 12 >> ThreadLimit 1000 >> >> StartServers 1 >> MinSpareThreads 25 >> MaxRequestsPerChild 0 >> MaxSpareThreads 500 >> ThreadsPerChild 500 >> MaxClients 5000 >> </IfModule> >> > > >> ProxyPass /WSMRegister ajp://localhost:8009/WSMRegister > >We are using the AJP NIO connector on port 8009 on Tomcat with 15 >worker threads: > >> <!-- Define an AJP 1.3 Connector on port 8009 --> >> <Connector port="8009" >> protocol="org.apache.coyote.ajp.AjpNioProtocol" >> redirectPort="8443" >> minSpareThreads="1" >> maxThreads="15" >> scheme="https" >> secure="true" >> URIDecoding="UTF-8" >> proxyName="secure2.360works.com" >> proxyPort="443" /> > >Problem detail: >lsof is currently showing 564 open sockets between Apache and Tomcat on >port 8009, with 1,352 users connected to Apache. >The two threads consuming the most CPU time in Tomcat are >"NioBlockingSelector.BlockPoller-2 / 15" and >"ajp-nio-8009-ClientPoller-0 / 25". Between them, they are taking 20% >of all CPU time for the Tomcat process. I get a few times a day when >our monitoring software reports slow response times, and I'd like to >solve this. How much actual processor time are those threads taking? The 20% is only relative and it would be helpful to know what it is relative to. Do you have access logs available for those times? The number of concurrent requests would be a useful number. When working that out you need to be sure if the time in the access log is the time the request started or the time it finished. >Some guesses at solutions: >I'm guessing that the high CPU usage is because they are polling all >564 open sockets constantly? Would it make sense to reduce the number >of open sockets? I didn't configure any maximum and I don't know how to >reduce this number. I'm also concerned that reducing that might negate >any benefits by increasing the number of sockets opening and closing >between ajp_mod_proxy and the NIO AJP connector. Some rough numbers. With a peak of 9 httpd children with 500 threads each - assume at least 4000 connected clients. With a request every 30 seconds that is roughly 133 requests a second assuming they are perfectly evenly distributed which they won't be. With 15 worker threads on Tomcat each one will be handling roughly 9 requests a second, again assuming even distribution. That means a request has about 110ms to complete or you run the risk of running out of threads. This includes the time to process the request and a little overhead for Tomcat to recycle the thread. I wouldn't be surprised for peak loads to be at least 2-3 times higher due to the uneven distribution of requests in time. That means requests have more like 35-40ms to complete. I suspect that you are running out of worker threads on the Tomcat side. Increasing it from 15 to 30 wouldn't do any harm. Of course, this is just guess work. You'd need to look at your access logs to be sure. >Maybe it's already running at optimal performance and I just need to >throw hardware at it, but it seems like a solvable problem, because the >actual worker threads are not doing much at all. I agree it sounds like configuration at this point. Mark --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
