-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 All,
So, I have some data from last night. It's about what you'd expect, except that the NIO+sendfile connector test failed most of the time: the client got something like "apr_connect: Connection reset by peer" when it tried to connect to the server. I say "something like" because when I woke up this morning, the error message was still in the scrollback buffer in my xterm, but by the time the tests had completed, it had fallen-off the end :( catalina.out contains this message: May 19, 2009 3:01:25 AM org.apache.tomcat.util.net.NioEndpoint$Acceptor run SEVERE: Socket accept failed java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145) at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198) at java.lang.Thread.run(Thread.java:619) followed by 44224 more of the same message until May 19, 2009 3:17:39 AM when the last message is printed. My system reports: $ ulimit -n (the incantation on my *NIX to get the max fds per process) 1024 lsof doesn't show anything out of the ordinary (although it's been hours since the test ran) and the NIO-sendfile test ran immediately afterward apparently without any problems. Oddly enough, one of the files I served (the last one, possibly coincidentally), appears to be still open: COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME java 1430 chris 77r REG 3,3 33554432 8751657 /home/chris/app/connector-test/8785/webapps/ROOT/32MiB.bin Re-running the test now (without an intervening restart) appears to be working just fine, although it looks like it's holding-on to file descriptors longer than absolutely necessary: java 1430 chris 68r REG 3,3 16777216 8751652 /home/chris/app/connector-test/8785/webapps/ROOT/16MiB.bin java 1430 chris 69r REG 3,3 16777216 8751652 /home/chris/app/connector-test/8785/webapps/ROOT/16MiB.bin java 1430 chris 70r REG 3,3 16777216 8751652 /home/chris/app/connector-test/8785/webapps/ROOT/16MiB.bin java 1430 chris 71r REG 3,3 16777216 8751652 /home/chris/app/connector-test/8785/webapps/ROOT/16MiB.bin java 1430 chris 72r REG 3,3 16777216 8751652 /home/chris/app/connector-test/8785/webapps/ROOT/16MiB.bin java 1430 chris 73r REG 3,3 16777216 8751652 /home/chris/app/connector-test/8785/webapps/ROOT/16MiB.bin java 1430 chris 75r REG 3,3 16777216 8751652 /home/chris/app/connector-test/8785/webapps/ROOT/16MiB.bin java 1430 chris 77r REG 3,3 33554432 8751657 /home/chris/app/connector-test/8785/webapps/ROOT/32MiB.bin java 1430 chris 88r REG 3,3 16777216 8751652 /home/chris/app/connector-test/8785/webapps/ROOT/16MiB.bin java 1430 chris 94r REG 3,3 16777216 8751652 /home/chris/app/connector-test/8785/webapps/ROOT/16MiB.bin java 1430 chris 95r REG 3,3 16777216 8751652 /home/chris/app/connector-test/8785/webapps/ROOT/16MiB.bin java 1430 chris 96r REG 3,3 16777216 8751652 /home/chris/app/connector-test/8785/webapps/ROOT/16MiB.bin The number of these open appears to fluctuate between about 13 and 40. Perhaps a GC is necessary for the fds to actually be closed. [My test just completed against the 16MiB.bin file, and performance was horrible: 50% of the transfer rate of the next-worst connector (NIO w/o sendfile, coincidentally). I didn't isolate the server for this test, but it seems particularly bad.] I'd love for someone intimately familiar (Bill? Mladen?) with the NIO connector to shed some light on what might be going on, here. Anyhow, here is the data I collected last night. First, the setup: $ uname -a Linux chadis 2.6.14-gentoo-r5 #2 PREEMPT Sat Dec 17 16:30:55 EST 2005 i686 AMD Athlon(tm) XP 1700+ AuthenticAMD GNU/Linux $ java -version java version "1.6.0_13" Java(TM) SE Runtime Environment (build 1.6.0_13-b03) Java HotSpot(TM) Client VM (build 11.3-b02, mixed mode, sharing) Apache Tomcat 6.0.18 with tcnative 1.1.16, Apache httpd 2.2.11 compiled locally with CFLAGS="-O2 -march=athlon-xp -fomit-frame-pointer", All tests were performed using ApacheBench 2.3 with concurrency=1 (single request thread) on localhost. Here are the configurations for each setup: httpd: prefork MPM: StartServers 5 MinSpareServers 5 MaxSpareServers 10 MaxClients 150 MaxRequestsPerChild 10000 Tomcat: <!-- Regular non-APR Coyote Connector --> <Connector port="8001" protocol="org.apache.coyote.http11.Http11Protocol" connectionTimeout="20000" server="Coyote1.1non-APR" /> <!-- APR Connector --> <Connector port="8002" protocol="org.apache.coyote.http11.Http11AprProtocol" useSendfile="true" connectionTimeout="20000" server="Coyote1.1APR" /> <!-- APR without sendfile --> <Connector port="8003" protocol="org.apache.coyote.http11.Http11AprProtocol" useSendfile="false" connectionTimeout="20000" server="Coyote1.1APRw/osendfile" /> <!-- NIO Connector --> <Connector port="8004" protocol="org.apache.coyote.http11.Http11NioProtocol" useSendfile="true" connectionTimeout="20000" server="Coyote1.1NIO" /> <!-- APR without sendfile --> <Connector port="8005" protocol="org.apache.coyote.http11.Http11NioProtocol" useSendfile="false" connectionTimeout="20000" server="Coyote1.1NIOw/osendfile" /> After my tests (all Tomcat tests were performed on the same Tomcat, in the same JVM, without restarts or anything like that), the JVM heap looks like this: $ ./jmap -heap 1430 Attaching to process ID 1430, please wait... Debugger attached successfully. Client compiler detected. JVM version is 11.3-b02 using thread-local object allocation. Mark Sweep Compact GC Heap Configuration: MinHeapFreeRatio = 40 MaxHeapFreeRatio = 70 MaxHeapSize = 67108864 (64.0MB) NewSize = 1048576 (1.0MB) MaxNewSize = 4294901760 (4095.9375MB) OldSize = 4194304 (4.0MB) NewRatio = 12 SurvivorRatio = 8 PermSize = 12582912 (12.0MB) MaxPermSize = 67108864 (64.0MB) Heap Usage: New Generation (Eden + 1 Survivor Space): capacity = 1114112 (1.0625MB) used = 1026048 (0.978515625MB) free = 88064 (0.083984375MB) 92.09558823529412% used Eden Space: capacity = 1048576 (1.0MB) used = 960520 (0.9160232543945312MB) free = 88056 (0.08397674560546875MB) 91.60232543945312% used - From Space: capacity = 65536 (0.0625MB) used = 65528 (0.06249237060546875MB) free = 8 (7.62939453125E-6MB) 99.98779296875% used To Space: capacity = 65536 (0.0625MB) used = 0 (0.0MB) free = 65536 (0.0625MB) 0.0% used tenured generation: capacity = 13664256 (13.03125MB) used = 9892512 (9.434234619140625MB) free = 3771744 (3.597015380859375MB) 72.39700427158273% used Perm Generation: capacity = 12582912 (12.0MB) used = 8750096 (8.344741821289062MB) free = 3832816 (3.6552581787109375MB) 69.53951517740886% used I have a vmstat log during the tests, but it doesn't show much except varying CPU usage (between ~60% all the way down to 5% or so) but no timestamps are displayed in the log, so I can't really correlate the data :( Here are the transfer rates (KiB/sec) for each server setup and file size (apologies for formatting... tabs are embedded so copy-and-paste into your favorite spreadsheet): File Size Apache httpd Coyote Coyote APR Coyote APR-sendfile Coyote NIO Coyote NIO-sendfile 4kiB 6211.93 6051.87 8187.10 8414.07 4084.77 4157.97 8kiB 11569.26 10390.15 14376.04 14699.94 7283.49 7333.03 16kiB 20907.48 17302.11 23127.55 23376.37 12916.64 13003.59 32kiB 37567.86 28954.23 36646.48 36868.86 22607.88 22722.98 64kiB 63730.95 44118.45 62149.78 51688.76 DNF 36014.79 128kiB 94360.32 58602.61 93244.56 65651.15 DNF 44190.42 256kiB 130820.33 73980.20 129496.26 80313.87 DNF 60165.13 512kiB 164934.15 67829.40 153265.08 72341.08 DNF 48075.80 1MiB 185482.52 74348.43 172728.69 78199.68 DNF 56424.22 2MiB 189090.15 77581.23 183614.90 81764.11 DNF 54911.99 4MiB 183388.01 79048.73 192016.43 83516.21 DNF 48691.96 8MiB 194678.37 80731.30 192675.70 85208.25 DNF 61161.66 16MiB 191319.39 81109.97 191728.56 84484.19 DNF 44094.85 32MiB 183452.71 80865.02 191444.06 84079.59 DNF 45538.00 These results confirm the assertions often made on this list: that Tomcat using an APR connector with sendfile enabled performs on-par with Apache httpd, while the other connector configurations do not perform nearly as well (excepting NIO, for which I don't have any data... I'll try to correct that, tonight). At least for single concurrency. I'll be running a multi-concurrency test tonight to see how these perform when there are actually multiple clients connecting. - -chris -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkoS3AoACgkQ9CaO5/Lv0PAKZgCgnZ63IgJc8YlEeoHVO97b43PR uEQAnidf81Rgn6K4q2w20aECHDuJ6Rs6 =volO -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org