Very interesting! Yes, managed code is the path to follow.
First idea non-blocking IO (from C# client side): use the new async/await for the communication. But force to use the new .NET framework and Visual Studio. And await is a wait on the current threads: http://msdn.microsoft.com/en-us/library/hh750082.aspx Maybe, a "node.js" approach, with a callback: http://stackoverflow.com/questions/16894907/creating-asynchronous-methods-with-task-factory-and-callback and only .NET 4.0: http://msdn.microsoft.com/en-us/library/dd537612(v=vs.100).aspx I don't still see the value of await: it blocked the current thread. I guess it is better to use a callback Angel "Java" Lopez @ajlopez On Fri, Feb 14, 2014 at 9:26 PM, Bilal S <bilal.so...@gmail.com> wrote: > Konstantin, > > > > On Fri, Jan 24, 2014 at 2:06 PM, Konstantin Preißer <kpreis...@apache.org > >wrote: > > > Hi all, > > > > for my Java Servlet web applications which run on Tomcat (currently > > 8.0.0-RC 10) on various Windows Server OSes (currently Windows Server > 2012 > > R2), I use the ISAPI Redirector to forward requests from IIS to Tomcat > over > > AJP. I use IIS as primary web server because I also host other websites > > that use different technologies like ASP.Net and PHP (and because IIS > > allows to run web applications as different processes as different user > > accounts, and because I can configure the SSL settings over IIS, and so > on). > > > > The ISAPI Redirector has its job done well in the past and currently I'm > > still using it. Note that I'm only using it to forward requests from a > > single IIS instance to a single Tomcat instance, but not for > load-balancing > > or other features. > > > > > > However, over the time I found some issues which seem to result from the > > changes that Win Server, IIS and other components have experienced over > > time, which I wanted to list here and see how these could be changed. > > > > A possibility that I see is to use an ASP.Net (C#) based redirector > > instead of an ISAPI based redirector as that will have a number of > > advantages - see below. > > > > > > > ==> > You raise good points. I have run into similar issues and thus created my > own project outside the Apache foundation three years ago (BonCode). It is > a C# based AJP connector. It can currently be used with Tomcat, JBOSS, > Jetty. From support requests I am surmising that is currently bundled with > software from a few manufacturers including: EMC, CSC, Siemens and others > instead of ISAPI redirector. > > Thus, I do encourage the update of the current IIS connection mechanism to > a more up-to-date method. Using a managed code mechanism is the way to go > in my opinion. > In the long run SPDY may also be of interest for the same purpose. The more > choices the better. > > The following are differences already in existence with BonCode and in > response to your extensive writing, only read on if you are curious:: > ====> > > > > 1. > > > > The ISAPI Redirector seems to be quite complicate to configure. You have > > to: > > 1) place the ISAPI redirector DLL in some arbitrary path (the docs > suggest > > to place them in your Tomcat\bin directory) > > > ==> not needed > > > 2) create a virtual directory in your IIS web application which points to > > this path > > > ==> not needed > > > 3) change the handler settings for the virtual directory to allow to > > execute ISAPI dlls > > > ==> not needed > > > 4) add the ISAPI redirector DLL to the list of CGI and ISAPI restrictions > > in IIS > > > ==> not needed > > > 5) add the ISAPI redirector DLL to your web app as ISAPI filter > > > ==> not needed > > > 6) create some registry entries at HKLM\Software\Apache Software > > Foundation\Jakarta Isapi Redirector\1.0 to specify the path of the > virtual > > directory, path to configuration files etc. > > > ==> not needed > > > 7) create configuration files (uriworkersmap.properties, > > worker.properties) and but them in some arbitrary path (the docs suggest > to > > place them in your Tomcat\conf directory) > > > > ==> not needed, configurations are done via the IIS UI and/or an xml > config file > > > > I see a few problems here. > > First, you have to place the ISAPI redirector DLL in some external > > arbitrary path. This can introduce additional maintenance issues as you > > always have to remember this when e.g. moving the server. Because the > docs > > suggest to place them in your Tomcat\bin directory, you might delete that > > file by mistake when you delete your Tomcat installation and create a new > > one. > > The same is true for the config files - if you place them in your Tomcat > > directory, you might delete them when you change your Tomcat config. > > > > Normally, these files do not belong to Tomcat, but to the ISAPI > > redirector, so I would expect to place them somewhere in your IIS web > > application. > > > > E.g, ASP.Net web applications have a "web.config" file in their root > > directory for configuration, and a "bin" directory where .Net assemblies > > can be placed. If you were using an ASP.Net based redirector for example > > (implemented as a managed module), you can place the binary into the > "bin" > > directory of your IIS webapp and configure it by adding it to the > > web.config file. This would also mean that you don't have to create a > > virtual directory any more. > > > > I also got problems with the system-wide registry keys that you need to > > set up for the ISAPI redirector, as they don't allow separate configs for > > different IIS webapps. I once tried to create a > "isapi_redirect.properties" > > with the configuration (instead of using the registry) like it is > described > > on the Tomcat Connectors IIS reference page [1], but I didn't have > success, > > so I reverted back to the registry settings. > > > > Note also that Microsoft has deprecated ISAPI filters and extensions in > > favor of native http modules [2]. > > > > ==> not needed > > > > > > 2. > > > > When using the ISAPI redirector with IIS 7, I got a few problems with > > response buffering - it seemed that IIS buffered the complete response > body > > before starting to send it to the client. However, when I tried this > again > > on IIS 8.5 today, then IIS only buffered a few MB before sending it to > the > > client, so I think this is not a problem any more. > > Note that you can manually specify the amount that IIS should buffer, by > > adding the following in web.config (in <configuration> , > > <system.webServer>): > > > > <handlers> > > <remove name="ISAPI-dll" /> > > <add name="ISAPI-dll" path="*.dll" verb="*" > > modules="IsapiModule" resourceType="File" requireAccess="Execute" > > allowPathInfo="true" responseBufferLimit="1" /> > > </handlers> > > > > ==> flexible buffering works with IIS, default is to mimic ISAPI but byte > and time based buffering / streaming options to client are available. > > > > > 3. > > > > Chunked encoding is disabled by default. > > Personally I think every web server should use chunked encoding if the > > client specifies connection: keep-alive and the content-length is > unknown, > > so that the server doesn't have to close the connection to signal EOF. It > > seems that this is a bit difficult in an ISAPI extension as this feature > > was being considered experimental until a few years ago. > > > > However, for example if you code a managed HTTP module using C#, you can > > just write bytes to the response without having to think about chunked > > encoding, as IIS will do automatically - it is pretty much like writing > to > > the response in a Java Servlet. However I don't know how this would be > done > > in native HTTP modules for IIS. > > > > ==> chunked encoding is default behavior when content size is not known > > > > > > > 4. > > > > By default, IIS applies a limit to POST requests that specify a > > Content-Length and its value exceeds a specific size. This means IIS will > > reject such POST requests before they reach Tomcat (which might allow > such > > a POST request). You can probably change the POST limit in IIS to allow > > bigger request bodys. > > E.g. In a managed module, since .Net 4.5 you can call > > Request.GetBufferlessInputStream(true) [3] to retrieve a bufferless > > InputStream to read the request body, and disable the request length > limit. > > > > ==> might be an interesting consideration. However, bypassing input > limits > may make this less manageable. > > > > > 5. > > > > Sometimes, the ISAPI redirector stops to work - a request times out and > > IIS is not able to shutdown the worker process (w3wp.exe) that uses the > > ISAPI redirector. > > Normally, this does not happen when no configuration is changed on the > > server. > > > > However, in my case, I have multiple IIS web applications that use the > > ISAPI redirector (using the same AppPool), and one web application that > > uses another app pool and uses PHP via FastCGI. > > > > ==> BonCode can have multiple instances even inside the same IIS site, all > hitting different backends. Will not conflict. You can use all IIS > management options. > > > > > Now, everything works. However, if I now open the IIS manager and go to > > the FastCGI settings, then double-click on the php-cgi.exe entry and > > change e.g. the maximum number of instances, then the ISAPI redirector > > suddenly stops working... This means IIS will not answer a request that > > goes to this web application (or even to another app with the same > apppool > > - I haven't tested this). There is no CPU usage. > > Sending a request to a webapp with another apppool (the one with PHP) > > still works. > > > > I have no idea why this happens (and how changing settings for a FastCGI > > application can impact the ISAPI redirector), but the only way to get > > things working again is to kill the w3wp.exe process so that IIS starts a > > new one - then everything will work again. > > > > I think with managed or native IIS modules, such strange issues shouldn't > > happen. ;-) > > > > ==> True > > > > > 6. > > > > As far as I can see, the ISAPI redirector uses blocking I/O when > > forwarding requests to Tomcat. > > > > This means when a slow client sends a request to IIS which gets forwarded > > to Tomcat, and Tomcat starts to send the response, in the IIS worker > > process at least 1 Thread will be dedicated to this request as long as it > > is not finished. This means if 500 slow clients concurrently send > requests > > to IIS (which get forwarded), IIS has to create 500 threads in the > > w3wp.exe. This can be a problem in terms of scalability, if lots of > clients > > send concurrent requests to your server. > > > > ==> Not most elegant but BonCode will time out the connection based on > IIS > AppPool setting and reuse. You can also do a similar thing on the Tomcat > side. The nature of AJP does not allow multiplexing a connection, so it > needs to be dedicated to a client. > > > > The recently released Servlet 3.1 spec allows to use non-blocking I/O > when > > reading the request body or writing the response body, so that Tomcat > > doesn't have to dedicate a thread for the lifetime of the request, but > only > > take one from a Therad pool when further data can be sent to (or read > from) > > the client. IIS however would still need to dedicate a thread for the > > request. Another problem is that at least IIS 8 only seems to create > > additional threads very slowly (1 Thread per second) when its thread pool > > is exhausted, which can happen if a few slow clients concurrently send > > requests handled by the ISAPI redirector. So some of the clients would > have > > to wait very long until a thread is created which can handle their > request. > > > > ==> Interesting idea, would need to understand better how this applies to > the connector and communication via AJP. Currently BonCode creates and > maintains a larger base thread-pool, and times them out slowly so the under > load behavior is more graceful with less create/destroy overhead. > > > > > > Now, e.g. .Net 4.5 introduces a new async programming model that allows > > you to write code in blocking/synchronous style, but actually behave as > > async code (using non-blocking I/O if possible). If you use such coding > for > > a managed module to forward requests (and use an async managed module), > > then IIS only needs to take a thread when there's actually code to run > for > > sending additionally data. This means that most of the time the request > > doesn't need a thread, so it should be much more scalable than the > blocking > > model. > > > > I once wrote a SPDY redirector for IIS based on C# (ASP.Net) that > forwards > > request over an SPDY connection, and after switching it to Async I/O, for > > 170 concurrent requests the w3wp.exe only had ~ 35 threads, instead of > ~195 > > threads with blocking I/O [4]. > > > > SPDY allows to multiplex requests on a single TCP connection so this > > combination should allow for a high number of concurrent requests without > > needing a huge number of threads. > > The SPDY redirector already works with Jetty's SPDY/3 implementation, but > > for Tomcat it seems that it does not yet support SPDY/3 (Costin Manolache > > once wrote that he wanted to switch Tomcat's SPDY implementation to > SPDY/3, > > but I didn't see activity in this arey so I don't know what the state of > > SPDY/3 for Tomcat is. > > > > For AJP, however, a TCP connection can only server one request at a time, > > so while an async AJP redirector would not require one thread per > request, > > it still would require one TCP connection per (concurrent) request. > > > > > > 7. > > > > The ISAPI redirector handles requests to directories like WEB-INF by > > itself (rejecting it with 404, or just disconnecting the client). > However, > > if it is just forwarding requests, then my point of view would be that it > > should not reject requests by itself, but instead let this do Tomcat (or > > whatever AJP server is behind), because Tomcat needs to also do this if > you > > directly connect to it over HTTP. > > I can see a reason for rejecting requests to WEB-INF at IIS level if IIS > > it is configured to serve static resources directly, and only forward > > requests to Tomcat. However, personally I never to this due to the > security > > implications, and the performance benefit for IIS serving static > resources > > directly instead of letting Tomcat serve them was rather not noticeable > for > > me. > > > > ==> Wouldn't this present an easy exploit path especially if IIS and > Tomcat use the same physical path and handlers are misconfigured, e.g. when > using servlets. I do think the WEB-INF blocking is the safer option. > > > > > 8. > > > > In the past there have been reports of problems with the ISAPI redirector > > when using IIS features like "web garden" (when one app pool has multiple > > worker processes) or apppool recycling. I can remember (but I do not have > > references for this atm) that Mladen Turk once said, one should disable > web > > garden (so only 1 process is used per app pool) and disable the apppool > > recycling to avoid problems with the ISAPI redirector. > > I think such problems shouldn't happen when using a managed or native IIS > > module. > > > ==> No issues reported with this scenario. Connector libraries and > configurations can be shared across servers. > > > > > > > > 9. > > > > The 32-bit version of the ISAPI redirector does not work on 64-bit > > versions of Windows. Even when you set the Apppool to allow to execute > > 32-bit applications (this means that the 32-bit version of w3wp.exe will > be > > executed), the 32-bit version of ISAPI redirector does not work. I do not > > know why this is so, but e.g. as .Net is platform-independent, you can > use > > the same assembly (dll) for 32-bit as well as for 64-bit apppools. > > > > ==> Same set of files due to managed code will work for 32/64 bit > > > > > > 10. > > > > Sometimes I see in the Windows event log erros that IIS rejected a > request > > due to bad characters, but the URL that is logged is actually a good one > > (e.g. http://myserver.com/myurl?param=value). If I request this URL, it > > is correctly server, and no new entry is created in the event log. If I > > look in the IIS logs for this URL and the logged time, I only find "200 > OK" > > entries so it seems this URL was correctly served. > > > > My guess is that it was actually a different URL that failed, but the > > ISAPI redirector somehow had overwritten some structure containing the > URL > > so that IIS would log this one instead. However this is only very rarely > so > > I cannot verify what actually happens there. > > > > (Fin) > > > > > > > > Now, I think a future-proof redirector can be one that uses SPDY (or a > > similar protocol) to redirects the requests instead of AJP, because > > 1) it allows multiple concurrent requests per TCP connection, which in > > conjunction with Async I/O should allow for high scalability, and > > 2) it allows to forward WebSocket requests. > > However, there are some issues with SPDY/3 (NPN and header compression) > > that put a few barriers for writing a simple request redirector. > > > > Some time ago, I have written such an SPDY redirector based in C# on a > > managed IIS module [4]. It already works well with Jetty (but Jetty does > > not yet support WebSocket over SPDY/3). However, as Tomcat seems not yet > to > > support SPDY/3, it cannot be used with Tomcat. Another problem is that > > SPDY/3 is still a draft of the SPDY spec which probably change a few > times > > in the future (Google is already working on SPDY/4). For a real usable > SPDY > > redirector, one would probably have to wait until SPDY is final to have a > > common basis for using it with different SPDY servers. > > > > > > Before the SPDY redirector, I also wrote a very simple AJP redirector in > > C# (but still with blocking I/O etc.) for which I only needed ~ 1 day so > it > > should be very easy to write a AJP based replacement for the current > ISAPI > > redirector. > > > > Personally, I would like to get rid of the ISAPI based redirector for > > Tomcat in the near future due to the above mentioned issues. My > preference > > would be to continue implementing the C#-based AJP redirector (but it > will > > probably need a rewrite), or if Tomcat already supports SPDY/3 in that > > time, use the SPDY redirector. > > > > However, I'm wondering about the ISAPI redirector which is available at > > the Tomcat website. From the docs one can see that the ISAPI redirector > is > > already a bit old as they still talk about Windows NT 4.0 and Windows 98 > > (and PWS). What do you think about this? Should the ISAPI redirector > still > > stay? Is SPDY the right way for forwarding requests to Tomcat in the > future? > > > > Thanks! > > > > > > Regards, > > Konstantin Preißer > > > > ==> There is much to be gained by a fresh look at IIS and Tomcat > interoperability and adjust it for todays problem set. > > -Bilal >
