Hi all,
for my Java Servlet web applications which run on Tomcat (currently 8.0.0-RC
10) on various Windows Server OSes (currently Windows Server 2012 R2), I use
the ISAPI Redirector to forward requests from IIS to Tomcat over AJP. I use IIS
as primary web server because I also host other websites that use different
technologies like ASP.Net and PHP (and because IIS allows to run web
applications as different processes as different user accounts, and because I
can configure the SSL settings over IIS, and so on).
The ISAPI Redirector has its job done well in the past and currently I'm still
using it. Note that I'm only using it to forward requests from a single IIS
instance to a single Tomcat instance, but not for load-balancing or other
features.
However, over the time I found some issues which seem to result from the
changes that Win Server, IIS and other components have experienced over time,
which I wanted to list here and see how these could be changed.
A possibility that I see is to use an ASP.Net (C#) based redirector instead of
an ISAPI based redirector as that will have a number of advantages - see below.
1.
The ISAPI Redirector seems to be quite complicate to configure. You have to:
1) place the ISAPI redirector DLL in some arbitrary path (the docs suggest to
place them in your Tomcat\bin directory)
2) create a virtual directory in your IIS web application which points to this
path
3) change the handler settings for the virtual directory to allow to execute
ISAPI dlls
4) add the ISAPI redirector DLL to the list of CGI and ISAPI restrictions in IIS
5) add the ISAPI redirector DLL to your web app as ISAPI filter
6) create some registry entries at HKLM\Software\Apache Software
Foundation\Jakarta Isapi Redirector\1.0 to specify the path of the virtual
directory, path to configuration files etc.
7) create configuration files (uriworkersmap.properties, worker.properties) and
but them in some arbitrary path (the docs suggest to place them in your
Tomcat\conf directory)
I see a few problems here.
First, you have to place the ISAPI redirector DLL in some external arbitrary
path. This can introduce additional maintenance issues as you always have to
remember this when e.g. moving the server. Because the docs suggest to place
them in your Tomcat\bin directory, you might delete that file by mistake when
you delete your Tomcat installation and create a new one.
The same is true for the config files - if you place them in your Tomcat
directory, you might delete them when you change your Tomcat config.
Normally, these files do not belong to Tomcat, but to the ISAPI redirector, so
I would expect to place them somewhere in your IIS web application.
E.g, ASP.Net web applications have a "web.config" file in their root directory
for configuration, and a "bin" directory where .Net assemblies can be placed.
If you were using an ASP.Net based redirector for example (implemented as a
managed module), you can place the binary into the "bin" directory of your IIS
webapp and configure it by adding it to the web.config file. This would also
mean that you don't have to create a virtual directory any more.
I also got problems with the system-wide registry keys that you need to set up
for the ISAPI redirector, as they don't allow separate configs for different
IIS webapps. I once tried to create a "isapi_redirect.properties" with the
configuration (instead of using the registry) like it is described on the
Tomcat Connectors IIS reference page [1], but I didn't have success, so I
reverted back to the registry settings.
Note also that Microsoft has deprecated ISAPI filters and extensions in favor
of native http modules [2].
2.
When using the ISAPI redirector with IIS 7, I got a few problems with response
buffering - it seemed that IIS buffered the complete response body before
starting to send it to the client. However, when I tried this again on IIS 8.5
today, then IIS only buffered a few MB before sending it to the client, so I
think this is not a problem any more.
Note that you can manually specify the amount that IIS should buffer, by adding
the following in web.config (in <configuration> , <system.webServer>):
<handlers>
<remove name="ISAPI-dll" />
<add name="ISAPI-dll" path="*.dll" verb="*" modules="IsapiModule"
resourceType="File" requireAccess="Execute" allowPathInfo="true"
responseBufferLimit="1" />
</handlers>
3.
Chunked encoding is disabled by default.
Personally I think every web server should use chunked encoding if the client
specifies connection: keep-alive and the content-length is unknown, so that the
server doesn't have to close the connection to signal EOF. It seems that this
is a bit difficult in an ISAPI extension as this feature was being considered
experimental until a few years ago.
However, for example if you code a managed HTTP module using C#, you can just
write bytes to the response without having to think about chunked encoding, as
IIS will do automatically - it is pretty much like writing to the response in
a Java Servlet. However I don't know how this would be done in native HTTP
modules for IIS.
4.
By default, IIS applies a limit to POST requests that specify a Content-Length
and its value exceeds a specific size. This means IIS will reject such POST
requests before they reach Tomcat (which might allow such a POST request). You
can probably change the POST limit in IIS to allow bigger request bodys.
E.g. In a managed module, since .Net 4.5 you can call
Request.GetBufferlessInputStream(true) [3] to retrieve a bufferless InputStream
to read the request body, and disable the request length limit.
5.
Sometimes, the ISAPI redirector stops to work - a request times out and IIS is
not able to shutdown the worker process (w3wp.exe) that uses the ISAPI
redirector.
Normally, this does not happen when no configuration is changed on the server.
However, in my case, I have multiple IIS web applications that use the ISAPI
redirector (using the same AppPool), and one web application that uses another
app pool and uses PHP via FastCGI.
Now, everything works. However, if I now open the IIS manager and go to the
FastCGI settings, then double-click on the php-cgi.exe entry and change e.g.
the maximum number of instances, then the ISAPI redirector suddenly stops
working... This means IIS will not answer a request that goes to this web
application (or even to another app with the same apppool - I haven’t tested
this). There is no CPU usage.
Sending a request to a webapp with another apppool (the one with PHP) still
works.
I have no idea why this happens (and how changing settings for a FastCGI
application can impact the ISAPI redirector), but the only way to get things
working again is to kill the w3wp.exe process so that IIS starts a new one -
then everything will work again.
I think with managed or native IIS modules, such strange issues shouldn't
happen. ;-)
6.
As far as I can see, the ISAPI redirector uses blocking I/O when forwarding
requests to Tomcat.
This means when a slow client sends a request to IIS which gets forwarded to
Tomcat, and Tomcat starts to send the response, in the IIS worker process at
least 1 Thread will be dedicated to this request as long as it is not finished.
This means if 500 slow clients concurrently send requests to IIS (which get
forwarded), IIS has to create 500 threads in the w3wp.exe. This can be a
problem in terms of scalability, if lots of clients send concurrent requests to
your server.
The recently released Servlet 3.1 spec allows to use non-blocking I/O when
reading the request body or writing the response body, so that Tomcat doesn't
have to dedicate a thread for the lifetime of the request, but only take one
from a Therad pool when further data can be sent to (or read from) the client.
IIS however would still need to dedicate a thread for the request. Another
problem is that at least IIS 8 only seems to create additional threads very
slowly (1 Thread per second) when its thread pool is exhausted, which can
happen if a few slow clients concurrently send requests handled by the ISAPI
redirector. So some of the clients would have to wait very long until a thread
is created which can handle their request.
Now, e.g. .Net 4.5 introduces a new async programming model that allows you to
write code in blocking/synchronous style, but actually behave as async code
(using non-blocking I/O if possible). If you use such coding for a managed
module to forward requests (and use an async managed module), then IIS only
needs to take a thread when there's actually code to run for sending
additionally data. This means that most of the time the request doesn't need a
thread, so it should be much more scalable than the blocking model.
I once wrote a SPDY redirector for IIS based on C# (ASP.Net) that forwards
request over an SPDY connection, and after switching it to Async I/O, for 170
concurrent requests the w3wp.exe only had ~ 35 threads, instead of ~195 threads
with blocking I/O [4].
SPDY allows to multiplex requests on a single TCP connection so this
combination should allow for a high number of concurrent requests without
needing a huge number of threads.
The SPDY redirector already works with Jetty's SPDY/3 implementation, but for
Tomcat it seems that it does not yet support SPDY/3 (Costin Manolache once
wrote that he wanted to switch Tomcat's SPDY implementation to SPDY/3, but I
didn't see activity in this arey so I don't know what the state of SPDY/3 for
Tomcat is.
For AJP, however, a TCP connection can only server one request at a time, so
while an async AJP redirector would not require one thread per request, it
still would require one TCP connection per (concurrent) request.
7.
The ISAPI redirector handles requests to directories like WEB-INF by itself
(rejecting it with 404, or just disconnecting the client). However, if it is
just forwarding requests, then my point of view would be that it should not
reject requests by itself, but instead let this do Tomcat (or whatever AJP
server is behind), because Tomcat needs to also do this if you directly connect
to it over HTTP.
I can see a reason for rejecting requests to WEB-INF at IIS level if IIS it is
configured to serve static resources directly, and only forward requests to
Tomcat. However, personally I never to this due to the security implications,
and the performance benefit for IIS serving static resources directly instead
of letting Tomcat serve them was rather not noticeable for me.
8.
In the past there have been reports of problems with the ISAPI redirector when
using IIS features like "web garden" (when one app pool has multiple worker
processes) or apppool recycling. I can remember (but I do not have references
for this atm) that Mladen Turk once said, one should disable web garden (so
only 1 process is used per app pool) and disable the apppool recycling to avoid
problems with the ISAPI redirector.
I think such problems shouldn't happen when using a managed or native IIS
module.
9.
The 32-bit version of the ISAPI redirector does not work on 64-bit versions of
Windows. Even when you set the Apppool to allow to execute 32-bit applications
(this means that the 32-bit version of w3wp.exe will be executed), the 32-bit
version of ISAPI redirector does not work. I do not know why this is so, but
e.g. as .Net is platform-independent, you can use the same assembly (dll) for
32-bit as well as for 64-bit apppools.
10.
Sometimes I see in the Windows event log erros that IIS rejected a request due
to bad characters, but the URL that is logged is actually a good one (e.g.
http://myserver.com/myurl?param=value). If I request this URL, it is correctly
server, and no new entry is created in the event log. If I look in the IIS logs
for this URL and the logged time, I only find "200 OK" entries so it seems this
URL was correctly served.
My guess is that it was actually a different URL that failed, but the ISAPI
redirector somehow had overwritten some structure containing the URL so that
IIS would log this one instead. However this is only very rarely so I cannot
verify what actually happens there.
(Fin)
Now, I think a future-proof redirector can be one that uses SPDY (or a similar
protocol) to redirects the requests instead of AJP, because
1) it allows multiple concurrent requests per TCP connection, which in
conjunction with Async I/O should allow for high scalability, and
2) it allows to forward WebSocket requests.
However, there are some issues with SPDY/3 (NPN and header compression) that
put a few barriers for writing a simple request redirector.
Some time ago, I have written such an SPDY redirector based in C# on a managed
IIS module [4]. It already works well with Jetty (but Jetty does not yet
support WebSocket over SPDY/3). However, as Tomcat seems not yet to support
SPDY/3, it cannot be used with Tomcat. Another problem is that SPDY/3 is still
a draft of the SPDY spec which probably change a few times in the future
(Google is already working on SPDY/4). For a real usable SPDY redirector, one
would probably have to wait until SPDY is final to have a common basis for
using it with different SPDY servers.
Before the SPDY redirector, I also wrote a very simple AJP redirector in C#
(but still with blocking I/O etc.) for which I only needed ~ 1 day so it should
be very easy to write a AJP based replacement for the current ISAPI redirector.
Personally, I would like to get rid of the ISAPI based redirector for Tomcat in
the near future due to the above mentioned issues. My preference would be to
continue implementing the C#-based AJP redirector (but it will probably need a
rewrite), or if Tomcat already supports SPDY/3 in that time, use the SPDY
redirector.
However, I'm wondering about the ISAPI redirector which is available at the
Tomcat website. From the docs one can see that the ISAPI redirector is already
a bit old as they still talk about Windows NT 4.0 and Windows 98 (and PWS).
What do you think about this? Should the ISAPI redirector still stay? Is SPDY
the right way for forwarding requests to Tomcat in the future?
Thanks!
Regards,
Konstantin Preißer
[1] http://tomcat.apache.org/connectors-doc/reference/iis.html
[2] http://forums.iis.net/t/1189728.aspx
[3] http://msdn.microsoft.com/de-de/library/hh195568.aspx
[4] http://markmail.org/message/ojknieujkcchxqla
[5] http://markmail.org/message/livcbkoa4b7bl7yq
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
