Re: [Announce] Enhanced ISAPI Redirector + Tomcat Connector Binaries (and IIS 6.0)
Tim, Have you had experience testing your version of the ISAPI redirector DLL with IIS 6.0? What specific IIS configuration settings take the fullest advantage of the persistent HTTP connections that your version offers? Also, I've seen zero discussion thus far on whether version 1.2.25 of the DLL should work with IIS in worker process isolation mode or in IIS 5 isolation mode. Do you have any insight on this? Anyone have any insight on this? :) -Travis Haagen - Original Message - From: Tim Whittington [EMAIL PROTECTED] To: users@tomcat.apache.org; [EMAIL PROTECTED] Sent: Thursday, December 20, 2007 5:20 PM Subject: [Announce] Enhanced ISAPI Redirector + Tomcat Connector Binaries Hi all We've been using Tomcat and Tomcat Connectors in our company products for almost 10 years now, and it's a very important part of our technology stack. Over those years we've contributed some enhancements and fixes to the IIS connector/ISAPI Redirector, and most of these have been accepted back into the trunk. There's one in particular that we feel is very useful that hasn't been accepted though, which is the addition of chunked encoding support to the ISAPI Redirector, which allows IIS to use HTTP keep alives between the browser and IIS - we've found this has major scalability benefits on high transaction volume sites. http://issues.apache.org/bugzilla/show_bug.cgi?id=35297 http://issues.apache.org/bugzilla/show_bug.cgi?id=35297 tracks the proposed patch for this, but it's been waiting for 2+ years for a 1.3 development branch to open to make it available to the wider community. We feel this is an important enough feature for us to maintain internally and use in many production sites over the years, and we think other people will find it useful, so we've now taken the step of making it available to the community at large in source + binary form on SourceForge. Our patches are all APLv2, as per the Tomcat source, and are available in the SourceForge SVN repo. https://sourceforge.net/projects/timsjk/ is the SF site. We've also taken the opportunity to make available the compiled binaries we produce in-house for many platforms (we support a large number), and I hope this is of use to the community at large. You'll find binaries for Apache 2.0, 2.0-prefork, 2.2, 2.2-prefork and IIS 5/6 on AIX PPC64, HP-UX PA-RISC2, HP-UX IA64, Linux-x86, Linux-x64, Windows x86 and Window x64 (with some obvious exceptions) available from the SourceForge project download page. We've provided builds of our enhanced ISAPI redirector as well as an unpatched ISAPI redirector. This isn't a fork of Tomcat Connectors - the work the Tomcat devs are doing on Tomcat Connectors in recent times, in particular Rainer and Mladen, is great, and we continue to value, use and support it - apart from the IIS connector we use unpatched mod_jk connectors everywhere. If/when the 1.3 development branch opens and the chunked encoding support is added to the main trunk we'll wind up our patches and see if there's a better way to distribute the binaries. If you're going to try out or use the enhanced ISAPI redirector, please be responsible when reporting bugs and test against the unpatched version before logging them against Tomcat. If they turn out to be due to our patches, then report them on the SF project. cheers tim - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat and IIS losing their link
It appears that my problem with IIS losing its link with Tomcat was actually an application problem, as several of you suggested. The JMX thread dump (very helpful over terminal services) indicated that 100 threads were locked up with Hibernate trying to get a database connection from DBCP. Upon further research I learned that Hibernate (and probably everything else) works better with the c3p0 pooling API, so I switched all my applications over to c3p0 and all is well (for now). Thanks for all the help, Travis Haagen - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Quality check mod_jk 1.2.26-dev
I may have found a problem with 1.2.26-dev. I installed 'isapi_redirect.dll' on my server and only one of two workers would respond to HTTP requests after that (ajp13w, but not ajp13lmsw). I then reinstalled the 1.2.25 version of the DLL and it worked fine. I'm running this on Windows 2003 server with IIS 6.0. Each worker is a seperate Tomcat instance running on localhost. Here are my settings for the isapi_redirect.dll: == workers.properties == worker.list=wlb,wlms,jkstatus worker.ajp13w.type=ajp13 worker.ajp13w.host=localhost worker.ajp13w.port=8009 worker.ajp13w.lbfactor=1 #The next three properties allow mod_jk to efficiently determine if a connection is in error or not responding fast enough due to whatever reason. worker.ajp13w.connect_timeout=1 worker.ajp13w.prepost_timeout=1 worker.ajp13w.socket_timeout=10 #This value (in seconds) must equal server.xml's connectionTimeout of 10 minutes worker.ajp13w.connection_pool_timeout=600 worker.ajp13w.connection_pool_size=350 worker.ajp13lms.type=ajp13 worker.ajp13lms.host=localhost worker.ajp13lms.port=8010 worker.ajp13lms.lbfactor=1 #The next three properties allow mod_jk to efficiently determine if a connection is in error or not responding fast enough due to whatever reason. worker.ajp13lms.connect_timeout=1 worker.ajp13lms.prepost_timeout=1 worker.ajp13lms.socket_timeout=10 #This value (in seconds) must equal server.xml's connectionTimeout of 10 minutes worker.ajp13lms.connection_pool_timeout=600 worker.ajp13lms.connection_pool_size=350 worker.wlb.type=lb worker.wlb.balance_workers=ajp13w worker.wlms.type=lb worker.wlms.balance_workers=ajp13lms worker.jkstatus.type=status == uriworkermap.properties == /jkmanager=jkstatus /cas/*=wlb /ep5sf/*=wlb /ep5cm/*=wlb /lms.mydomain.com/*=wlms /www.mydomain.net/*=wlms == isapi_redirect.properties == extension_uri=/jakarta/isapi_redirect.dll log_file=D:\tomcat_iis_connector\logs\isapi_redirect.log log_level=warn worker_file=D:\tomcat_iis_connector\conf\workers.properties worker_mount_file=D:\tomcat_iis_connector\conf\uriworkermap.properties - Original Message - From: Rainer Jung [EMAIL PROTECTED] To: [EMAIL PROTECTED] Cc: users@tomcat.apache.org Sent: Tuesday, December 18, 2007 6:46 AM Subject: Quality check mod_jk 1.2.26-dev Hi all, The next version of mod_jk is approaching its release. A code snapshot (revision 605168) is available at: http://people.apache.org/~rjung/mod_jk-dev/ It is in the same format as a release download, so easy to build. Under the same URL you can find the updated documentation. It would be nice, if we could get some testing feedback from the community. Feel free to post any observations. Depending on your feedback we plan to tag the release at the end of the week. The most important changes are: Features/Improvements - - allow '*' for the worker name in exclusion rules (resp. JkUnMount) which will override all workers - add new value All for JkMountCopy httpd directive - enhance URI to worker map listing in status worker for Apache httpd. We now list maps for all virtual servers and not only the one, in which JKStatus itself was called. - enhance URI to worker map listing. Update stale uriworkermap.properties immediately. - remove trace logging from compiled code via new --disable-trace configure switch - maintain idle connections in decreasing (LRU) slot order - create JK_WORKER_ROUTE and JK_REQUEST_DURATION notes for httpd access log even if no JkRequestLogFormat is set - cleanup init and destroy of server configuration for httpd - remove global configuration items from per server configuration - remove unused attributes secret_key and automount/JkAutoMount - cleanup of jk_uri_worker_map - some refactorings of jk_connect.c, jk_ajp_common.c, jk_lb_worker.c Fixes - - BZ 43229: Load balancer does not do fail over after reply timeouts. - BZ 42038: Correct overlay of mounts and unmounts for IIS - BZ 42003: Need to increase the maximum buffer size for HTTP header from 16k to 64k. - BZ 43516: Memory leak for Apache httpd module of size 8KB for every virtual host without JK directive after each restart - BZ 43873: Fix small memory leak occuring during httpd restart - BZ 43647: Content-Type changes unexpectedly from text/html to text/pla (ingnore flush packets before we received the response headers) - fixed crash during startup when using worker configuration inheritance (attribute reference) and log level debug - match header names exactly against pre defined constants - repair detailed Apache httpd version display in jk status worker - repair broken apxs auto-detection in configure - correct parameter validation of JkStatusUpdateTask (jkstatus ant task) Docs - Small additions to JkShmFile documentation - BZ 43684: Replace JkMountFile by
Re: Tomcat and IIS losing their link
Thanks for the responses thus far. I've been trying to get more information about the problem, but haven't been very successfull. I can't seem to create thread dumps, because I have to use Terminal Services. Is there a way to get a thread dump even though I'm using Terminal Services? Tomcat keeps locking up every afternoon, during the work week, so its definitely related to server traffic. Sometimes it locks up twice in an afternoon. I found this article about IIS 6.0 application health monitoring and I was wondering if anyone know if the items listed in the section 'ISAPI Extensions Can Declare Themselves Unhealthy' are implemented in the ISAPI redirector? http://help.alentus.com/article.aspx?id=10958cNode=7F8F8Y Here's what the ISAPI redirector logs say (the following errors are simply repeated until Tomcat is restarted). My log level is currently set to 'WARN': Mon Dec 17 13:48:34.484 2007] [4340:5744] [error] jk_ajp_common.c (966): (ajp13w) can't receive the response message from tomcat, network problems or tomcat (127.0.0.1:8009) is down (errno=60) [Mon Dec 17 13:48:34.484 2007] [4340:5744] [error] jk_ajp_common.c (1658): (ajp13w) Tomcat is down or refused connection. No response has been sent to the client (yet) [Mon Dec 17 13:48:38.390 2007] [4340:3340] [error] jk_ajp_common.c (966): (ajp13w) can't receive the response message from tomcat, network problems or tomcat (127.0.0.1:8009) is down (errno=60) [Mon Dec 17 13:48:38.390 2007] [4340:3340] [error] jk_ajp_common.c (1658): (ajp13w) Tomcat is down or refused connection. No response has been sent to the client (yet) [Mon Dec 17 13:48:47.015 2007] [4340:5744] [error] jk_ajp_common.c (966): (ajp13w) can't receive the response message from tomcat, network problems or tomcat (127.0.0.1:8009) is down (errno=60) [Mon Dec 17 13:48:47.015 2007] [4340:5744] [error] jk_ajp_common.c (1658): (ajp13w) Tomcat is down or refused connection. No response has been sent to the client (yet) [Mon Dec 17 13:48:49.015 2007] [4340:5744] [error] jk_ajp_common.c (2097): (ajp13w) Connecting to tomcat failed. Tomcat is probably not started or is listening on the wrong port - Original Message - From: Rainer Jung [EMAIL PROTECTED] To: Tomcat Users List users@tomcat.apache.org Sent: Friday, December 14, 2007 9:20 AM Subject: Re: Tomcat and IIS losing their link Travis Haagen wrote: Hello, I've got IIS 6.0 linked with Tomcat 5.5.25 (with JDK 1.5.0_14-b03 and AJP dll) via the ISAPI connector (JK 1.2.25) on Windows 2003 Server, and during high traffic periods, the connection between IIS and Tomcat is being severed. This has been occurring in the mid-afternoon every day since this server went live and to get it running again, I've had to restart Tomcat. The peak connections to IIS during one of the days was just under 100, so this isn't really all that high of traffic. The isapi_redirect.log isn't much help. It has only been logging errors after the connection to Tomcat has been broken. Tomcat itself hasn't logged anything of relevance and I don't actually think that Tomcat is affected when the link is broken, but as I mentioned earlier, I have to restart Tomcat to get it to reconnect with IIS. Nevertheless at least i would be curious to know, what the exact contents of the isapi redirector log files are. We don't need all of it, but the first 100 lines or so could be interesting. What's you log level? Have a look at the other thread running on this list the last few days (ISAPI JK2 ran better than JK, how can that be?). To get a slightly better idea what's happening when, you could also introduce a load balancer and a status worker. Most likely it would help to take Java Thread Dumps and look inside the JVM, what your request threads are doing, when IIS gets stuck. Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat and IIS losing their link
Thanks for the reply Per... Tomcat and IIS are running on the same machine, with no external load balancer or firewall between IIS and Tomcat. When the connection is lost, IIS displays either service is unavailable or a 503 error message. I only have maxThreads=400, because I didn't think it would hurt anything. I was running with far fewer maxThreads and had the same problem. The machine is a quad-core CPU with 4 gigs of ram and the load is very light. I'll look into your other suggestions. Also, I noticed in my original post that the redirectPort was wrong and I fixed that :) Any other ideas out there? -Travis - Original Message - From: Per Johnsson [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Thursday, December 13, 2007 11:52 PM Subject: FW: Tomcat and IIS losing their link (Tomcat maillist bounced my mail so I send it to you directly./per) Hi! Some questions: - Is the tomcat and IIS running on the same machine? - What's the result when the connection is lost? Service unavaible from IIS? - Is there any loadbalancer, firewall between the IIS and tomcat? - You have maxThreads=400, are there 400 concurrant requests ever during high load? We had some similar issues and we didn't find any good solution for it but a lot of small adjustments made it better like: - Lower the maxthreads (we had very cpu consuming requests at some times) we went from like 300 to 20 I beleive. - Installing java 1.6 which is much more efficient. - Using the latest APR and Isapi dll. /Per Jonsson -Original Message- From: Travis Haagen [mailto:[EMAIL PROTECTED] Sent: den 14 december 2007 05:08 To: Tomcat Posts Subject: Tomcat and IIS losing their link Hello, I've got IIS 6.0 linked with Tomcat 5.5.25 (with JDK 1.5.0_14-b03 and AJP dll) via the ISAPI connector (JK 1.2.25) on Windows 2003 Server, and during high traffic periods, the connection between IIS and Tomcat is being severed. This has been occurring in the mid-afternoon every day since this server went live and to get it running again, I've had to restart Tomcat. The peak connections to IIS during one of the days was just under 100, so this isn't really all that high of traffic. The isapi_redirect.log isn't much help. It has only been logging errors after the connection to Tomcat has been broken. Tomcat itself hasn't logged anything of relevance and I don't actually think that Tomcat is affected when the link is broken, but as I mentioned earlier, I have to restart Tomcat to get it to reconnect with IIS. Most of my configuration settings came from this website: http://wiki.jboss.org/wiki/Wiki.jsp?page=OptimalMod_jk1.2Configuration Here are the relevant workers.properties settings: worker.ajp13w.type=ajp13 worker.ajp13w.host=localhost worker.ajp13w.port=8009 worker.ajp13w.lbfactor=1 worker.ajp13w.connect_timeout=1 worker.ajp13w.prepost_timeout=1 worker.ajp13w.socket_timeout=10 worker.ajp13w.connection_pool_timeout=600 worker.ajp13w.connection_pool_size=350 Here are the relevant server.xml settings: Connector port=8443 maxHttpHeaderSize=8192 maxThreads=400 minSpareThreads=50 maxSpareThreads=150 enableLookups=false disableUploadTimeout=true acceptCount=100 scheme=https secure=true SSLEngine=on SSLPassword=removed SSLCertificateFile=removed SSLCertificateKeyFile=removed / Connector port=8009 redirectPort=8444 protocol=AJP/1.3 connectionTimeout=60 enableLookups=false maxThreads=400 minSpareThreads=50 maxSpareThreads=150 / I just configured IIS 6.0 to run in IIS 5.0 isolation mode and that's the end of my ideas for fixing this issue. The only other ISAPI filters that I'm running are ASP.NET_1.1.4322.2307 and ISAPI_Rewrite3 (for URL rewriting). IIS web site connections is set to 'unlimited', connection timeout is 120 seconds, HTTP Keep-Alives is enabled. Until the crazy thing stops working, everything is fine. Any help will be greatly appreciated. Thanks, Travis - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat Process Memory Leak?
Any ideas about what could be causing this? I realize it could be an application-related leak, but in that case wouldn't we see it running out of heap? Server-side application leaks are really hard to figure out, because they usually only happen in a high-traffic production environment and are difficult to produce on development machines. Basically, you should look at bug reports (e.g., bugzilla) for all of the 3rd party libraries that your application uses, or at least those libraries that have been added/upgraded since this problem started occuring. For each component, search for 'memory leak' and see where that gets you. Good luck, Travis - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Tomcat and IIS losing their link
Hello, I've got IIS 6.0 linked with Tomcat 5.5.25 (with JDK 1.5.0_14-b03 and AJP dll) via the ISAPI connector (JK 1.2.25) on Windows 2003 Server, and during high traffic periods, the connection between IIS and Tomcat is being severed. This has been occurring in the mid-afternoon every day since this server went live and to get it running again, I've had to restart Tomcat. The peak connections to IIS during one of the days was just under 100, so this isn't really all that high of traffic. The isapi_redirect.log isn't much help. It has only been logging errors after the connection to Tomcat has been broken. Tomcat itself hasn't logged anything of relevance and I don't actually think that Tomcat is affected when the link is broken, but as I mentioned earlier, I have to restart Tomcat to get it to reconnect with IIS. Most of my configuration settings came from this website: http://wiki.jboss.org/wiki/Wiki.jsp?page=OptimalMod_jk1.2Configuration Here are the relevant workers.properties settings: worker.ajp13w.type=ajp13 worker.ajp13w.host=localhost worker.ajp13w.port=8009 worker.ajp13w.lbfactor=1 worker.ajp13w.connect_timeout=1 worker.ajp13w.prepost_timeout=1 worker.ajp13w.socket_timeout=10 worker.ajp13w.connection_pool_timeout=600 worker.ajp13w.connection_pool_size=350 Here are the relevant server.xml settings: Connector port=8443 maxHttpHeaderSize=8192 maxThreads=400 minSpareThreads=50 maxSpareThreads=150 enableLookups=false disableUploadTimeout=true acceptCount=100 scheme=https secure=true SSLEngine=on SSLPassword=removed SSLCertificateFile=removed SSLCertificateKeyFile=removed / Connector port=8009 redirectPort=8444 protocol=AJP/1.3 connectionTimeout=60 enableLookups=false maxThreads=400 minSpareThreads=50 maxSpareThreads=150 / I just configured IIS 6.0 to run in IIS 5.0 isolation mode and that's the end of my ideas for fixing this issue. The only other ISAPI filters that I'm running are ASP.NET_1.1.4322.2307 and ISAPI_Rewrite3 (for URL rewriting). IIS web site connections is set to 'unlimited', connection timeout is 120 seconds, HTTP Keep-Alives is enabled. Until the crazy thing stops working, everything is fine. Any help will be greatly appreciated. Thanks, Travis
