-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
All,
So, I have some data from last night. It's about what you'd expect,
except that the NIO+sendfile connector test failed most of the time: the
client got something like "apr_connect: Connection reset by peer" when
it tried to connect to the server. I say "something like" because when I
woke up this morning, the error message was still in the scrollback
buffer in my xterm, but by the time the tests had completed, it had
fallen-off the end :(
catalina.out contains this message:
May 19, 2009 3:01:25 AM org.apache.tomcat.util.net.NioEndpoint$Acceptor run
SEVERE: Socket accept failed
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
at
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
at java.lang.Thread.run(Thread.java:619)
followed by 44224 more of the same message until May 19, 2009 3:17:39 AM
when the last message is printed.
My system reports:
$ ulimit -n (the incantation on my *NIX to get the max fds per process)
1024
lsof doesn't show anything out of the ordinary (although it's been hours
since the test ran) and the NIO-sendfile test ran immediately afterward
apparently without any problems.
Oddly enough, one of the files I served (the last one, possibly
coincidentally), appears to be still open:
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
java 1430 chris 77r REG 3,3 33554432 8751657
/home/chris/app/connector-test/8785/webapps/ROOT/32MiB.bin
Re-running the test now (without an intervening restart) appears to be
working just fine, although it looks like it's holding-on to file
descriptors longer than absolutely necessary:
java 1430 chris 68r REG 3,3 16777216 8751652
/home/chris/app/connector-test/8785/webapps/ROOT/16MiB.bin
java 1430 chris 69r REG 3,3 16777216 8751652
/home/chris/app/connector-test/8785/webapps/ROOT/16MiB.bin
java 1430 chris 70r REG 3,3 16777216 8751652
/home/chris/app/connector-test/8785/webapps/ROOT/16MiB.bin
java 1430 chris 71r REG 3,3 16777216 8751652
/home/chris/app/connector-test/8785/webapps/ROOT/16MiB.bin
java 1430 chris 72r REG 3,3 16777216 8751652
/home/chris/app/connector-test/8785/webapps/ROOT/16MiB.bin
java 1430 chris 73r REG 3,3 16777216 8751652
/home/chris/app/connector-test/8785/webapps/ROOT/16MiB.bin
java 1430 chris 75r REG 3,3 16777216 8751652
/home/chris/app/connector-test/8785/webapps/ROOT/16MiB.bin
java 1430 chris 77r REG 3,3 33554432 8751657
/home/chris/app/connector-test/8785/webapps/ROOT/32MiB.bin
java 1430 chris 88r REG 3,3 16777216 8751652
/home/chris/app/connector-test/8785/webapps/ROOT/16MiB.bin
java 1430 chris 94r REG 3,3 16777216 8751652
/home/chris/app/connector-test/8785/webapps/ROOT/16MiB.bin
java 1430 chris 95r REG 3,3 16777216 8751652
/home/chris/app/connector-test/8785/webapps/ROOT/16MiB.bin
java 1430 chris 96r REG 3,3 16777216 8751652
/home/chris/app/connector-test/8785/webapps/ROOT/16MiB.bin
The number of these open appears to fluctuate between about 13 and 40.
Perhaps a GC is necessary for the fds to actually be closed.
[My test just completed against the 16MiB.bin file, and performance was
horrible: 50% of the transfer rate of the next-worst connector (NIO w/o
sendfile, coincidentally). I didn't isolate the server for this test,
but it seems particularly bad.]
I'd love for someone intimately familiar (Bill? Mladen?) with the NIO
connector to shed some light on what might be going on, here.
Anyhow, here is the data I collected last night. First, the setup:
$ uname -a
Linux chadis 2.6.14-gentoo-r5 #2 PREEMPT Sat Dec 17 16:30:55 EST 2005
i686 AMD Athlon(tm) XP 1700+ AuthenticAMD GNU/Linux
$ java -version
java version "1.6.0_13"
Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
Java HotSpot(TM) Client VM (build 11.3-b02, mixed mode, sharing)
Apache Tomcat 6.0.18 with tcnative 1.1.16, Apache httpd 2.2.11 compiled
locally with CFLAGS="-O2 -march=athlon-xp -fomit-frame-pointer",
All tests were performed using ApacheBench 2.3 with concurrency=1
(single request thread) on localhost.
Here are the configurations for each setup:
httpd: prefork MPM:
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 150
MaxRequestsPerChild 10000
Tomcat:
<!-- Regular non-APR Coyote Connector -->
<Connector port="8001"
protocol="org.apache.coyote.http11.Http11Protocol"
connectionTimeout="20000"
server="Coyote1.1non-APR"
/>
<!-- APR Connector -->
<Connector port="8002"
protocol="org.apache.coyote.http11.Http11AprProtocol"
useSendfile="true"
connectionTimeout="20000"
server="Coyote1.1APR"
/>
<!-- APR without sendfile -->
<Connector port="8003"
protocol="org.apache.coyote.http11.Http11AprProtocol"
useSendfile="false"
connectionTimeout="20000"
server="Coyote1.1APRw/osendfile"
/>
<!-- NIO Connector -->
<Connector port="8004"
protocol="org.apache.coyote.http11.Http11NioProtocol"
useSendfile="true"
connectionTimeout="20000"
server="Coyote1.1NIO"
/>
<!-- APR without sendfile -->
<Connector port="8005"
protocol="org.apache.coyote.http11.Http11NioProtocol"
useSendfile="false"
connectionTimeout="20000"
server="Coyote1.1NIOw/osendfile"
/>
After my tests (all Tomcat tests were performed on the same Tomcat, in
the same JVM, without restarts or anything like that), the JVM heap
looks like this:
$ ./jmap -heap 1430
Attaching to process ID 1430, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 11.3-b02
using thread-local object allocation.
Mark Sweep Compact GC
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 67108864 (64.0MB)
NewSize = 1048576 (1.0MB)
MaxNewSize = 4294901760 (4095.9375MB)
OldSize = 4194304 (4.0MB)
NewRatio = 12
SurvivorRatio = 8
PermSize = 12582912 (12.0MB)
MaxPermSize = 67108864 (64.0MB)
Heap Usage:
New Generation (Eden + 1 Survivor Space):
capacity = 1114112 (1.0625MB)
used = 1026048 (0.978515625MB)
free = 88064 (0.083984375MB)
92.09558823529412% used
Eden Space:
capacity = 1048576 (1.0MB)
used = 960520 (0.9160232543945312MB)
free = 88056 (0.08397674560546875MB)
91.60232543945312% used
- From Space:
capacity = 65536 (0.0625MB)
used = 65528 (0.06249237060546875MB)
free = 8 (7.62939453125E-6MB)
99.98779296875% used
To Space:
capacity = 65536 (0.0625MB)
used = 0 (0.0MB)
free = 65536 (0.0625MB)
0.0% used
tenured generation:
capacity = 13664256 (13.03125MB)
used = 9892512 (9.434234619140625MB)
free = 3771744 (3.597015380859375MB)
72.39700427158273% used
Perm Generation:
capacity = 12582912 (12.0MB)
used = 8750096 (8.344741821289062MB)
free = 3832816 (3.6552581787109375MB)
69.53951517740886% used
I have a vmstat log during the tests, but it doesn't show much except
varying CPU usage (between ~60% all the way down to 5% or so) but no
timestamps are displayed in the log, so I can't really correlate the data :(
Here are the transfer rates (KiB/sec) for each server setup and file
size (apologies for formatting... tabs are embedded so copy-and-paste
into your favorite spreadsheet):
File Size Apache httpd Coyote Coyote APR Coyote APR-sendfile
Coyote NIO
Coyote NIO-sendfile
4kiB 6211.93 6051.87 8187.10 8414.07 4084.77 4157.97
8kiB 11569.26 10390.15 14376.04 14699.94 7283.49
7333.03
16kiB 20907.48 17302.11 23127.55 23376.37
12916.64 13003.59
32kiB 37567.86 28954.23 36646.48 36868.86
22607.88 22722.98
64kiB 63730.95 44118.45 62149.78 51688.76 DNF
36014.79
128kiB 94360.32 58602.61 93244.56 65651.15 DNF
44190.42
256kiB 130820.33 73980.20 129496.26 80313.87 DNF
60165.13
512kiB 164934.15 67829.40 153265.08 72341.08 DNF
48075.80
1MiB 185482.52 74348.43 172728.69 78199.68 DNF
56424.22
2MiB 189090.15 77581.23 183614.90 81764.11 DNF
54911.99
4MiB 183388.01 79048.73 192016.43 83516.21 DNF
48691.96
8MiB 194678.37 80731.30 192675.70 85208.25 DNF
61161.66
16MiB 191319.39 81109.97 191728.56 84484.19 DNF
44094.85
32MiB 183452.71 80865.02 191444.06 84079.59 DNF
45538.00
These results confirm the assertions often made on this list: that
Tomcat using an APR connector with sendfile enabled performs on-par with
Apache httpd, while the other connector configurations do not perform
nearly as well (excepting NIO, for which I don't have any data... I'll
try to correct that, tonight). At least for single concurrency. I'll be
running a multi-concurrency test tonight to see how these perform when
there are actually multiple clients connecting.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkoS3AoACgkQ9CaO5/Lv0PAKZgCgnZ63IgJc8YlEeoHVO97b43PR
uEQAnidf81Rgn6K4q2w20aECHDuJ6Rs6
=volO
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
