Konstantin,
On Fri, Jan 24, 2014 at 2:06 PM, Konstantin Preißer <kpreis...@apache.org>wrote: > Hi all, > > for my Java Servlet web applications which run on Tomcat (currently > 8.0.0-RC 10) on various Windows Server OSes (currently Windows Server 2012 > R2), I use the ISAPI Redirector to forward requests from IIS to Tomcat over > AJP. I use IIS as primary web server because I also host other websites > that use different technologies like ASP.Net and PHP (and because IIS > allows to run web applications as different processes as different user > accounts, and because I can configure the SSL settings over IIS, and so on). > > The ISAPI Redirector has its job done well in the past and currently I'm > still using it. Note that I'm only using it to forward requests from a > single IIS instance to a single Tomcat instance, but not for load-balancing > or other features. > > > However, over the time I found some issues which seem to result from the > changes that Win Server, IIS and other components have experienced over > time, which I wanted to list here and see how these could be changed. > > A possibility that I see is to use an ASP.Net (C#) based redirector > instead of an ISAPI based redirector as that will have a number of > advantages - see below. > > > ==> You raise good points. I have run into similar issues and thus created my own project outside the Apache foundation three years ago (BonCode). It is a C# based AJP connector. It can currently be used with Tomcat, JBOSS, Jetty. From support requests I am surmising that is currently bundled with software from a few manufacturers including: EMC, CSC, Siemens and others instead of ISAPI redirector. Thus, I do encourage the update of the current IIS connection mechanism to a more up-to-date method. Using a managed code mechanism is the way to go in my opinion. In the long run SPDY may also be of interest for the same purpose. The more choices the better. The following are differences already in existence with BonCode and in response to your extensive writing, only read on if you are curious:: ====> > 1. > > The ISAPI Redirector seems to be quite complicate to configure. You have > to: > 1) place the ISAPI redirector DLL in some arbitrary path (the docs suggest > to place them in your Tomcat\bin directory) > ==> not needed > 2) create a virtual directory in your IIS web application which points to > this path > ==> not needed > 3) change the handler settings for the virtual directory to allow to > execute ISAPI dlls > ==> not needed > 4) add the ISAPI redirector DLL to the list of CGI and ISAPI restrictions > in IIS > ==> not needed > 5) add the ISAPI redirector DLL to your web app as ISAPI filter > ==> not needed > 6) create some registry entries at HKLM\Software\Apache Software > Foundation\Jakarta Isapi Redirector\1.0 to specify the path of the virtual > directory, path to configuration files etc. > ==> not needed > 7) create configuration files (uriworkersmap.properties, > worker.properties) and but them in some arbitrary path (the docs suggest to > place them in your Tomcat\conf directory) > > ==> not needed, configurations are done via the IIS UI and/or an xml config file > I see a few problems here. > First, you have to place the ISAPI redirector DLL in some external > arbitrary path. This can introduce additional maintenance issues as you > always have to remember this when e.g. moving the server. Because the docs > suggest to place them in your Tomcat\bin directory, you might delete that > file by mistake when you delete your Tomcat installation and create a new > one. > The same is true for the config files - if you place them in your Tomcat > directory, you might delete them when you change your Tomcat config. > > Normally, these files do not belong to Tomcat, but to the ISAPI > redirector, so I would expect to place them somewhere in your IIS web > application. > > E.g, ASP.Net web applications have a "web.config" file in their root > directory for configuration, and a "bin" directory where .Net assemblies > can be placed. If you were using an ASP.Net based redirector for example > (implemented as a managed module), you can place the binary into the "bin" > directory of your IIS webapp and configure it by adding it to the > web.config file. This would also mean that you don't have to create a > virtual directory any more. > > I also got problems with the system-wide registry keys that you need to > set up for the ISAPI redirector, as they don't allow separate configs for > different IIS webapps. I once tried to create a "isapi_redirect.properties" > with the configuration (instead of using the registry) like it is described > on the Tomcat Connectors IIS reference page [1], but I didn't have success, > so I reverted back to the registry settings. > > Note also that Microsoft has deprecated ISAPI filters and extensions in > favor of native http modules [2]. > > ==> not needed > > 2. > > When using the ISAPI redirector with IIS 7, I got a few problems with > response buffering - it seemed that IIS buffered the complete response body > before starting to send it to the client. However, when I tried this again > on IIS 8.5 today, then IIS only buffered a few MB before sending it to the > client, so I think this is not a problem any more. > Note that you can manually specify the amount that IIS should buffer, by > adding the following in web.config (in <configuration> , > <system.webServer>): > > <handlers> > <remove name="ISAPI-dll" /> > <add name="ISAPI-dll" path="*.dll" verb="*" > modules="IsapiModule" resourceType="File" requireAccess="Execute" > allowPathInfo="true" responseBufferLimit="1" /> > </handlers> > > ==> flexible buffering works with IIS, default is to mimic ISAPI but byte and time based buffering / streaming options to client are available. > > 3. > > Chunked encoding is disabled by default. > Personally I think every web server should use chunked encoding if the > client specifies connection: keep-alive and the content-length is unknown, > so that the server doesn't have to close the connection to signal EOF. It > seems that this is a bit difficult in an ISAPI extension as this feature > was being considered experimental until a few years ago. > > However, for example if you code a managed HTTP module using C#, you can > just write bytes to the response without having to think about chunked > encoding, as IIS will do automatically - it is pretty much like writing to > the response in a Java Servlet. However I don't know how this would be done > in native HTTP modules for IIS. > > ==> chunked encoding is default behavior when content size is not known > > 4. > > By default, IIS applies a limit to POST requests that specify a > Content-Length and its value exceeds a specific size. This means IIS will > reject such POST requests before they reach Tomcat (which might allow such > a POST request). You can probably change the POST limit in IIS to allow > bigger request bodys. > E.g. In a managed module, since .Net 4.5 you can call > Request.GetBufferlessInputStream(true) [3] to retrieve a bufferless > InputStream to read the request body, and disable the request length limit. > > ==> might be an interesting consideration. However, bypassing input limits may make this less manageable. > > 5. > > Sometimes, the ISAPI redirector stops to work - a request times out and > IIS is not able to shutdown the worker process (w3wp.exe) that uses the > ISAPI redirector. > Normally, this does not happen when no configuration is changed on the > server. > > However, in my case, I have multiple IIS web applications that use the > ISAPI redirector (using the same AppPool), and one web application that > uses another app pool and uses PHP via FastCGI. > ==> BonCode can have multiple instances even inside the same IIS site, all hitting different backends. Will not conflict. You can use all IIS management options. > Now, everything works. However, if I now open the IIS manager and go to > the FastCGI settings, then double-click on the php-cgi.exe entry and > change e.g. the maximum number of instances, then the ISAPI redirector > suddenly stops working... This means IIS will not answer a request that > goes to this web application (or even to another app with the same apppool > - I haven't tested this). There is no CPU usage. > Sending a request to a webapp with another apppool (the one with PHP) > still works. > > I have no idea why this happens (and how changing settings for a FastCGI > application can impact the ISAPI redirector), but the only way to get > things working again is to kill the w3wp.exe process so that IIS starts a > new one - then everything will work again. > > I think with managed or native IIS modules, such strange issues shouldn't > happen. ;-) > > ==> True > > 6. > > As far as I can see, the ISAPI redirector uses blocking I/O when > forwarding requests to Tomcat. > > This means when a slow client sends a request to IIS which gets forwarded > to Tomcat, and Tomcat starts to send the response, in the IIS worker > process at least 1 Thread will be dedicated to this request as long as it > is not finished. This means if 500 slow clients concurrently send requests > to IIS (which get forwarded), IIS has to create 500 threads in the > w3wp.exe. This can be a problem in terms of scalability, if lots of clients > send concurrent requests to your server. > > ==> Not most elegant but BonCode will time out the connection based on IIS AppPool setting and reuse. You can also do a similar thing on the Tomcat side. The nature of AJP does not allow multiplexing a connection, so it needs to be dedicated to a client. > The recently released Servlet 3.1 spec allows to use non-blocking I/O when > reading the request body or writing the response body, so that Tomcat > doesn't have to dedicate a thread for the lifetime of the request, but only > take one from a Therad pool when further data can be sent to (or read from) > the client. IIS however would still need to dedicate a thread for the > request. Another problem is that at least IIS 8 only seems to create > additional threads very slowly (1 Thread per second) when its thread pool > is exhausted, which can happen if a few slow clients concurrently send > requests handled by the ISAPI redirector. So some of the clients would have > to wait very long until a thread is created which can handle their request. > ==> Interesting idea, would need to understand better how this applies to the connector and communication via AJP. Currently BonCode creates and maintains a larger base thread-pool, and times them out slowly so the under load behavior is more graceful with less create/destroy overhead. > > Now, e.g. .Net 4.5 introduces a new async programming model that allows > you to write code in blocking/synchronous style, but actually behave as > async code (using non-blocking I/O if possible). If you use such coding for > a managed module to forward requests (and use an async managed module), > then IIS only needs to take a thread when there's actually code to run for > sending additionally data. This means that most of the time the request > doesn't need a thread, so it should be much more scalable than the blocking > model. > > I once wrote a SPDY redirector for IIS based on C# (ASP.Net) that forwards > request over an SPDY connection, and after switching it to Async I/O, for > 170 concurrent requests the w3wp.exe only had ~ 35 threads, instead of ~195 > threads with blocking I/O [4]. > > SPDY allows to multiplex requests on a single TCP connection so this > combination should allow for a high number of concurrent requests without > needing a huge number of threads. > The SPDY redirector already works with Jetty's SPDY/3 implementation, but > for Tomcat it seems that it does not yet support SPDY/3 (Costin Manolache > once wrote that he wanted to switch Tomcat's SPDY implementation to SPDY/3, > but I didn't see activity in this arey so I don't know what the state of > SPDY/3 for Tomcat is. > > For AJP, however, a TCP connection can only server one request at a time, > so while an async AJP redirector would not require one thread per request, > it still would require one TCP connection per (concurrent) request. > > > 7. > > The ISAPI redirector handles requests to directories like WEB-INF by > itself (rejecting it with 404, or just disconnecting the client). However, > if it is just forwarding requests, then my point of view would be that it > should not reject requests by itself, but instead let this do Tomcat (or > whatever AJP server is behind), because Tomcat needs to also do this if you > directly connect to it over HTTP. > I can see a reason for rejecting requests to WEB-INF at IIS level if IIS > it is configured to serve static resources directly, and only forward > requests to Tomcat. However, personally I never to this due to the security > implications, and the performance benefit for IIS serving static resources > directly instead of letting Tomcat serve them was rather not noticeable for > me. > > ==> Wouldn't this present an easy exploit path especially if IIS and Tomcat use the same physical path and handlers are misconfigured, e.g. when using servlets. I do think the WEB-INF blocking is the safer option. > > 8. > > In the past there have been reports of problems with the ISAPI redirector > when using IIS features like "web garden" (when one app pool has multiple > worker processes) or apppool recycling. I can remember (but I do not have > references for this atm) that Mladen Turk once said, one should disable web > garden (so only 1 process is used per app pool) and disable the apppool > recycling to avoid problems with the ISAPI redirector. > I think such problems shouldn't happen when using a managed or native IIS > module. > ==> No issues reported with this scenario. Connector libraries and configurations can be shared across servers. > > > 9. > > The 32-bit version of the ISAPI redirector does not work on 64-bit > versions of Windows. Even when you set the Apppool to allow to execute > 32-bit applications (this means that the 32-bit version of w3wp.exe will be > executed), the 32-bit version of ISAPI redirector does not work. I do not > know why this is so, but e.g. as .Net is platform-independent, you can use > the same assembly (dll) for 32-bit as well as for 64-bit apppools. > > ==> Same set of files due to managed code will work for 32/64 bit > > 10. > > Sometimes I see in the Windows event log erros that IIS rejected a request > due to bad characters, but the URL that is logged is actually a good one > (e.g. http://myserver.com/myurl?param=value). If I request this URL, it > is correctly server, and no new entry is created in the event log. If I > look in the IIS logs for this URL and the logged time, I only find "200 OK" > entries so it seems this URL was correctly served. > > My guess is that it was actually a different URL that failed, but the > ISAPI redirector somehow had overwritten some structure containing the URL > so that IIS would log this one instead. However this is only very rarely so > I cannot verify what actually happens there. > > (Fin) > > > > Now, I think a future-proof redirector can be one that uses SPDY (or a > similar protocol) to redirects the requests instead of AJP, because > 1) it allows multiple concurrent requests per TCP connection, which in > conjunction with Async I/O should allow for high scalability, and > 2) it allows to forward WebSocket requests. > However, there are some issues with SPDY/3 (NPN and header compression) > that put a few barriers for writing a simple request redirector. > > Some time ago, I have written such an SPDY redirector based in C# on a > managed IIS module [4]. It already works well with Jetty (but Jetty does > not yet support WebSocket over SPDY/3). However, as Tomcat seems not yet to > support SPDY/3, it cannot be used with Tomcat. Another problem is that > SPDY/3 is still a draft of the SPDY spec which probably change a few times > in the future (Google is already working on SPDY/4). For a real usable SPDY > redirector, one would probably have to wait until SPDY is final to have a > common basis for using it with different SPDY servers. > > > Before the SPDY redirector, I also wrote a very simple AJP redirector in > C# (but still with blocking I/O etc.) for which I only needed ~ 1 day so it > should be very easy to write a AJP based replacement for the current ISAPI > redirector. > > Personally, I would like to get rid of the ISAPI based redirector for > Tomcat in the near future due to the above mentioned issues. My preference > would be to continue implementing the C#-based AJP redirector (but it will > probably need a rewrite), or if Tomcat already supports SPDY/3 in that > time, use the SPDY redirector. > > However, I'm wondering about the ISAPI redirector which is available at > the Tomcat website. From the docs one can see that the ISAPI redirector is > already a bit old as they still talk about Windows NT 4.0 and Windows 98 > (and PWS). What do you think about this? Should the ISAPI redirector still > stay? Is SPDY the right way for forwarding requests to Tomcat in the future? > > Thanks! > > > Regards, > Konstantin Preißer > ==> There is much to be gained by a fresh look at IIS and Tomcat interoperability and adjust it for todays problem set. -Bilal
