Thanks for all the tips. Today I tried the AJP connector and we had no problems. After watching a very informative presentation on tomcat performance tuning by Filip Hanik and reading the docs, I am guessing this was more of a function of the higher default max thread count of the ajp connector. The higher default seemed more inline with the recommendations of the presentation. My next step is to go back to the http connector with a max thread count of 200 ( the ajp default).
However, the fact that my system already needs 200 threads for processing with minimal CPU usage makes me think that I will not be able to effectively utilize a large % of available CPU with the thread per request model. i.e., i would need to set the max thread count excessively high to utilize most of the cpu. I have generated stack dumps throughout the day and practically all running threads are doing socket i/o. almost none are performing application logic. I know tomcat has some comet support, but in my limited investigation that seems designed for ajax heavy applications. Would it still be useful in my scenario (highly concurrent small requests served very fast from cache). It seems like it would still have value by allowing async reads and writes. On Wed, Apr 8, 2009 at 3:08 PM, Rainer Jung <rainer.j...@kippdata.de> wrote: > On 08.04.2009 08:23, feedly team wrote: >> We are running apache and tomcat on the same machine (using the http >> connector) and logging requests in both. Occasionally (maybe 1% of >> requests) we see 502 response in the apache log spread fairly evenly >> throughout the day. these requests don't appear in the tomcat log >> produced by the access valve. in the apache error log I see: >> >> proxy: error reading status line from remote server 127.0.0.1 >> proxy: Error reading from remote server returned by [url] >> >> The requests are a mix of methods (GET/POST/PUT) of pretty small >> bodies and responses, generally under 1k. >> >> We added request time (%T) in the apache logs and many of the requests >> have 0 response time so I don't think it's a timeout issue. Reading >> the tomcat connector documentation, I tried increasing the maxThreads >> to 100 and backlog to 1000 in the http connector since it seemed like >> with the default settings a burst of traffic may cause this issue. >> This didn't solve the problem. I have done periodic thread dumps and >> it looks like the number of http threads doesn't get past 80. >> >> concerning machine resources, cpu is not loaded at all (~10% >> utilization). i checked file descriptor usage, it's about 400 and >> there is a per process limit of 1024. >> >> I am not sure if this should be posted to the apache list or tomcat >> list, but thought i would start here since to me it seems more likely >> that I am not configuring tomcat correctly as opposed to some problem >> in apache. I have searched the web quite a bit, but nothing seems to >> fit. The next thing I will try is to use the AJP connector. But even >> if it gets rid of the problem, I am worried it may reappear once >> traffic increases. Has anyone experienced a similar problem? > > I guess it's the usual HTTP Keep-Alive race with mod_proxy_http. > The connections between Apache and Tomcat are only reused for a certain > amount of time. What happens every now and then is that httpd thinks it > has a valid connection and sends the next data, but in parallel Tomcat > detects the HTTP KeepAlive has timed out and closes the connection. > > There have been improvements in newer 2.2.x versions (latest is 2.2.11). > You should especially look at the various pool size and timeout > configuration attributes on the mod_proxy documentation page and also at > the Keepalive configuration values for the Tomcat HTTP connector you are > using. > >> other stuff: >> OS: CentOS >> apache: 2.2.3, using mod_proxy >> tomcat: 6.0.16 >> using netstat, i see a moderate number (~80) of tomcat's sockets in >> the CLOSE_WAIT state, not sure if this is relevant. > > I assume Apache and Tomcat are on separate systems, and you notice > CLOSE_WAIT when doing netstat on the Tomcat system and the connections > are proxy connections? Then this means Apache has closed a connection > and Tomcat would still use it. Investigate the related timeout > parameters as suggested above. > > Regards, > > Rainer > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
