Re: Apache 1.3.x and 2.0.x Performance Issue
Jason Kissinger wrote: Ian, Were you able to verify our test results? sorry .. I've been out of my office for a while. I'll have a look as soon as I get back in. --Ian -Jason Ian Holsman wrote: Jess M. Holle wrote: William A. Rowe, Jr. wrote: At 11:54 AM 8/29/2002, Jess M. Holle wrote: Jason Kissinger responded: Windows clients hitting Solaris servers does not exhibit this problem. Only Solaris clients hitting Windows servers. I'm unsure if other UN*X have this problem, Linux and Windows does not. And Solaris client hitting anything but Windows works fine. We have some HP/UX boxes that I could build wget on to test as clients, if that would be helpful. Do the Windows boxes have the Quality of Service (QoS) Network Drivers installed? Are they running any firewall software? Are there any unpatched XP boxes with the AFD.SYS fault lying around? I don't know the answers to any of these questions. I do know that this occurs on multiple networks, i.e. ours and our customers, so whatever it is is not at all unusual. As such it would be great if either Apache was fixed or the necessary fixes to the Windows and/or Solaris configurations when this occurs were fully documented :-) Has anyone gotten *good* (e.g. something on par with 8MB/sec) download performance with an Apache server on Windows and a Solaris client? [Is there a counter-example to prove that this poor performance is not always true of this combination -- without cranking the ack interval to unreal values, that is.] I'll try to get a simple test going from my laptop tomorrow. can you tell me what solaris version/update you are running. you are downloading a single large file right server: Windows 2000sp2, Apache 1.3.26 with mod_jk/1.1.0 and Tomcat 3.2.4 (reproducible with all other Apache and Tomcat versions I've tried) file: dd if=/dev/urandom of=8mbfile bs=1 count=800 where httpd.conf has IfModule mod__jk.c JkWorkersFiles C:/tomcat/conf/workers.properties JkMount /webapp/* ajp13 /IfModule client: SunOS 5.8 Generic_108528-15 sun4u sparc SUNW,Sun-Blade-1000 test1: wget http://w2k/8mbfile == 80KB/s test2: wget http://w2k/webapp/8mbfile == 1MB/s client: Linux 2.4.18-3smp test1: wget http://w2k/8mbfile == 8MB/s test2: wget http://w2k/webapp/8mbfile == 1MB/s
Re: Apache 1.3.x and 2.0.x Performance Issue
Clay Webster wrote: Jess, Were IIS and Tomcat on the same windows hardware as the Apache httpds? Are you seeing network issues (e.g.- rexmits) -- you might have them everywhere, but your Solaris may not be configured well (ndd,MTU,etc). Jason Kissinger ([EMAIL PROTECTED]) responded: It isn't network hardware issues (rexmits). All Windows server tests were against same Windows 2000 server, though it's also been reported against Windows NT. The Solaris clients are the out-of-the-box settings, with acceptible performance against both IIS (bleh!) and the Tomcat built-in webserver. We were thinking it was problem with Nagle algorithm not being disabled, but snooping for setsockopt via a strace tool for Windows shows that TCP_NODELAY is being set. I don't know Windows enough (at all!) to know if this is sufficient, or if a win32 api also needs to be called directly. It still appears to be deferring ACK even though TCP_NODELAY should be set. Changing the tcp_deferred_ack_interval on the Solaris client improves performance, but that isn't an acceptable work around, and probably hurts performance for apps that'd prefer to delay ACK, especially in a WAN. The default on our Solaris boxes seem to be 20. Another oddity is that when retrieving a resource through Tomcat and mod_jk (possibly mod_jserv also, but haven't verified that), the performance is better, despite the additional hops and overhead of retrieving resource through java process. I see similar performance for all server OS' with this config. wget 8mb file from server: -1.3.x Apache through mod_jk on Windows: o client on Solaris: 1MB/sec o client on Linux: 1MB/sec -1.3.x Apache on Windows: o client on Solaris: 80K/sec o client on Linux: 8MB/sec Our thought is that mod_jk/Tomcat has response buffer larger than TCP floating window, thus the server is never stuck with a partial tcp fragment awaiting an ACK. Windows clients hitting Solaris servers does not exhibit this problem. Only Solaris clients hitting Windows servers. I'm unsure if other UN*X have this problem, Linux and Windows does not. And Solaris client hitting anything but Windows works fine. We have some HP/UX boxes that I could build wget on to test as clients, if that would be helpful. In short there really is something screwy with the way Apache on Windows interacts with Solaris clients. -- Jess Holle ---BeginMessage--- It isn't network hardware issues (rexmits). All Windows server tests were against same Windows 2000 server, though it's also been reported against Windows NT. The Solaris clients are the out-of-the-box settings, with acceptible performance against both IIS (bleh!) and the Tomcat built-in webserver. We were thinking it was problem with Nagle algorithm not being disabled, but snooping for setsockopt via a strace tool for Windows shows that TCP_NODELAY is being set. I don't know Windows enough (at all!) to know if this is sufficient, or if a win32 api also needs to be called directly. It still appears to be deferring ACK even though TCP_NODELAY should be set. Changing the tcp_deferred_ack_interval on the Solaris client improves performance, but that isn't an acceptable work around, and probably hurts performance for apps that'd prefer to delay ACK, especially in a WAN. The default on our Solaris boxes seem to be 20. Another oddity is that when retrieving a resource through Tomcat and mod_jk (possibly mod_jserv also, but haven't verified that), the performance is better, despite the additional hops and overhead of retrieving resource through java process. I see similar performance for all server OS' with this config. wget 8mb file from server: -1.3.x Apache through mod_jk on Windows: o client on Solaris: 1MB/sec o client on Linux: 1MB/sec -1.3.x Apache on Windows: o client on Solaris: 80K/sec o client on Linux: 8MB/sec Our thought is that mod_jk/Tomcat has response buffer larger than TCP floating window, thus the server is never stuck with a partial tcp fragment awaiting an ACK. Windows clients hitting Solaris servers does not exhibit this problem. Only Solaris clients hitting Windows servers. I'm unsure if other UN*X have this problem, Linux and Windows does not. And Solaris client hitting anything but Windows works fine. We have some HP/UX boxes that I could build wget on to test as clients, if that would be helpful. Jess M. Holle wrote: Subject: RE: Apache 1.3.x and 2.0.x Performance Issue From: Clay Webster [EMAIL PROTECTED] Date: Tue, 27 Aug 2002 13:49:15 -0700 To: '[EMAIL PROTECTED]' [EMAIL PROTECTED] Jess, Were IIS and Tomcat on the same windows hardware as the Apache httpds? Are you seeing network issues (e.g.- rexmits) -- you might have them everywhere, but your Solaris may not be configured well (ndd,MTU,etc
Re: Apache 1.3.x and 2.0.x Performance Issue
At 11:54 AM 8/29/2002, Jess M. Holle wrote: Jason Kissinger ([EMAIL PROTECTED]) responded: Windows clients hitting Solaris servers does not exhibit this problem. Only Solaris clients hitting Windows servers. I'm unsure if other UN*X have this problem, Linux and Windows does not. And Solaris client hitting anything but Windows works fine. We have some HP/UX boxes that I could build wget on to test as clients, if that would be helpful. Do the Windows boxes have the Quality of Service (QoS) Network Drivers installed? Are they running any firewall software? Are there any unpatched XP boxes with the AFD.SYS fault lying around? In short there really is something screwy with the way Apache on Windows interacts with Solaris clients. Something's strange, that's for certain. Bill
Re: Apache 1.3.x and 2.0.x Performance Issue
William A. Rowe, Jr. wrote: At 11:54 AM 8/29/2002, Jess M. Holle wrote: Jason Kissinger ([EMAIL PROTECTED]) responded: Windows clients hitting Solaris servers does not exhibit this problem. Only Solaris clients hitting Windows servers. I'm unsure if other UN*X have this problem, Linux and Windows does not. And Solaris client hitting anything but Windows works fine. We have some HP/UX boxes that I could build wget on to test as clients, if that would be helpful. Do the Windows boxes have the Quality of Service (QoS) Network Drivers installed? Are they running any firewall software? Are there any unpatched XP boxes with the AFD.SYS fault lying around? I don't know the answers to any of these questions. I do know that this occurs on multiple networks, i.e. ours and our customers, so whatever it is is not at all unusual. As such it would be great if either Apache was fixed or the necessary fixes to the Windows and/or Solaris configurations when this occurs were fully documented :-) Has anyone gotten *good* (e.g. something on par with 8MB/sec) download performance with an Apache server on Windows and a Solaris client? [Is there a counter-example to prove that this poor performance is not always true of this combination -- without cranking the ack interval to unreal values, that is.] -- Jess Holle
Re: Apache 1.3.x and 2.0.x Performance Issue
Note: wget was used in all measurements. Dirk-Willem van Gulik wrote: It would be nice if the client used was somehting like 'ab' - which comes with apache ran at 1-100 concurrency; or something like fetch, curl or wget to make the client identical on all platforms. Dw On Tue, 27 Aug 2002, Jess M. Holle wrote: Ian Holsman wrote: Jess M. Holle wrote: Both Apache 1.3.x and 2.0.x suffer a severe perfomance issue when the server is on Windows and the client is Solaris (and perhaps others). Before you stop reading this as simply "we know Windows does not perform well", I should point out that this does not occur when the client is Windows or Linux, nor when the server is on the same Windows box but is IIS or Tomcat (standalone)! Some rough download speeds: * recent Apache 1.3.x on Windows: o client on Solaris (8): 80K/sec o client on Linux or Windows: 8MB/sec * recent Apache 2.0.x on Windows: o client on Solaris (8): 120K/sec o client on Linux or Windows: 8MB/sec * IIS on Windows o any client tried: 8-9 MB/sec * Tomcat (standalone) on Windows o any client tried: ~8MB/sec just out of interest does the same thing happen when we have a solaris server and a windows client? I'm not sure. I've not yet tried that (and actually another engineer ran all these tests). I also just noticed that I got some numbers slightly wrong: Apache 2.0.39 on Windows * client on Solaris (8): 649 K/s * client on Linux: 9.1 MB/s [I've appended this amended info to the bug report.] -- Jess Holle
Re: Apache 1.3.x and 2.0.x Performance Issue
My test cases were against a Windows 2000 server, not XP. Our customer is seeing this against a Windows NT 4.0 system. I'm sorry, I'm a UN*X hack so don't know much about Windows: How do I check if QoS Network Drivers are installed? There is no firewall in the picture. I did a search of my 2000 server and found afd.sys files in C:\WINNT\ServicePackFiles\i386, C:\WINNT\system32\dllcache and C:\WINNT\system32\drivers. How do I check if these are patched sufficiently? They are 122,672 bytes large and have a timestamp of 5/4/2001. -Jason * From:* William A. Rowe, Jr. [EMAIL PROTECTED] http://groups.yahoo.com/group/new-httpd/post?protectID=125212253105056135218149051077181241248144166046039109074 *Date:* Thu Aug 29, 2002 12:22 pm *Subject:* Re: Apache 1.3.x and 2.0.x Performance Issue At 11:54 AM 8/29/2002, Jess M. Holle wrote: Jason Kissinger ([EMAIL PROTECTED] http://groups.yahoo.com/group/new-httpd/post?protectID=045202020112099132172242163148114100018179139218183041) responded: Windows clients hitting Solaris servers does not exhibit this problem. Only Solaris clients hitting Windows servers. I'm unsure if other UN*X have this problem, Linux and Windows does not. And Solaris client hitting anything but Windows works fine. We have some HP/UX boxes that I could build wget on to test as clients, if that would be helpful. Do the Windows boxes have the Quality of Service (QoS) Network Drivers installed? Are they running any firewall software? Are there any unpatched XP boxes with the AFD.SYS fault lying around? In short there really is something screwy with the way Apache on Windows interacts with Solaris clients. Something's strange, that's for certain. Bill
Re: Apache 1.3.x and 2.0.x Performance Issue
Jess M. Holle wrote: William A. Rowe, Jr. wrote: At 11:54 AM 8/29/2002, Jess M. Holle wrote: Jason Kissinger ([EMAIL PROTECTED]) responded: Windows clients hitting Solaris servers does not exhibit this problem. Only Solaris clients hitting Windows servers. I'm unsure if other UN*X have this problem, Linux and Windows does not. And Solaris client hitting anything but Windows works fine. We have some HP/UX boxes that I could build wget on to test as clients, if that would be helpful. Do the Windows boxes have the Quality of Service (QoS) Network Drivers installed? Are they running any firewall software? Are there any unpatched XP boxes with the AFD.SYS fault lying around? I don't know the answers to any of these questions. I do know that this occurs on multiple networks, i.e. ours and our customers, so whatever it is is not at all unusual. As such it would be great if either Apache was fixed or the necessary fixes to the Windows and/or Solaris configurations when this occurs were fully documented :-) Has anyone gotten *good* (e.g. something on par with 8MB/sec) download performance with an Apache server on Windows and a Solaris client? [Is there a counter-example to prove that this poor performance is not always true of this combination -- without cranking the ack interval to unreal values, that is.] I'll try to get a simple test going from my laptop tomorrow. can you tell me what solaris version/update you are running. you are downloading a single large file right -- Jess Holle
Re: Apache 1.3.x and 2.0.x Performance Issue
Ian Holsman wrote: Jess M. Holle wrote: William A. Rowe, Jr. wrote: At 11:54 AM 8/29/2002, Jess M. Holle wrote: Jason Kissinger ([EMAIL PROTECTED]) responded: Windows clients hitting Solaris servers does not exhibit this problem. Only Solaris clients hitting Windows servers. I'm unsure if other UN*X have this problem, Linux and Windows does not. And Solaris client hitting anything but Windows works fine. We have some HP/UX boxes that I could build wget on to test as clients, if that would be helpful. Do the Windows boxes have the Quality of Service (QoS) Network Drivers installed? Are they running any firewall software? Are there any unpatched XP boxes with the AFD.SYS fault lying around? I don't know the answers to any of these questions. I do know that this occurs on multiple networks, i.e. ours and our customers, so whatever it is is not at all unusual. As such it would be great if either Apache was fixed or the necessary fixes to the Windows and/or Solaris configurations when this occurs were fully documented :-) Has anyone gotten *good* (e.g. something on par with 8MB/sec) download performance with an Apache server on Windows and a Solaris client? [Is there a counter-example to prove that this poor performance is not always true of this combination -- without cranking the ack interval to unreal values, that is.] I'll try to get a simple test going from my laptop tomorrow. can you tell me what solaris version/update you are running. you are downloading a single large file right server: Windows 2000sp2, Apache 1.3.26 with mod_jk/1.1.0 and Tomcat 3.2.4 (reproducible with all other Apache and Tomcat versions I've tried) file: dd if=/dev/urandom of=8mbfile bs=1 count=800 where httpd.conf has IfModule mod__jk.c JkWorkersFiles C:/tomcat/conf/workers.properties JkMount /webapp/* ajp13 /IfModule client: SunOS 5.8 Generic_108528-15 sun4u sparc SUNW,Sun-Blade-1000 test1: wget http://w2k/8mbfile == 80KB/s test2: wget http://w2k/webapp/8mbfile == 1MB/s client: Linux 2.4.18-3smp test1: wget http://w2k/8mbfile == 8MB/s test2: wget http://w2k/webapp/8mbfile == 1MB/s
Re: Apache 1.3.x and 2.0.x Performance Issue
At 04:07 PM 8/29/2002, you wrote: client: SunOS 5.8 Generic_108528-15 sun4u sparc SUNW,Sun-Blade-1000 test1: wget http://w2k/8mbfile == 80KB/s test2: wget http://w2k/webapp/8mbfile == 1MB/s client: Linux 2.4.18-3smp test1: wget http://w2k/8mbfile == 8MB/s test2: wget http://w2k/webapp/8mbfile == 1MB/s I'm really beginning to suspect a borked sendfile implementation. Suggest that you try disabling sendfile on solaris and repeat the test.
Re: Apache 1.3.x and 2.0.x Performance Issue
On Thu, Aug 29, 2002 at 05:46:11PM -0500, William A. Rowe, Jr. wrote: I'm really beginning to suspect a borked sendfile implementation. Suggest that you try disabling sendfile on solaris and repeat the test. Solaris 8 doesn't have sendfile (sendfilev). I think Solaris 9 as well as really recent 8 MUs do. Dale probably knows when it was added (he made the initial post on dev@apr about it). But, I'm not sure how a lack of sendfile would cause the client to see performance degregation. Remember Solaris is the client not the server. Was sendfile even used on 1.3 for Win32? -- justin
Re: Apache 1.3.x and 2.0.x Performance Issue
On Tue, 27 Aug 2002, Jess M. Holle wrote: * recent Apache 1.3.x on Windows: o client on Solaris (8): 80K/sec o client on Linux or Windows: 8MB/sec * recent Apache 2.0.x on Windows: o client on Solaris (8): 120K/sec o client on Linux or Windows: 8MB/sec * IIS on Windows o any client tried: 8-9 MB/sec * Tomcat (standalone) on Windows o any client tried: ~8MB/sec Oddly Apache performance was around 1MB/sec when the request was routed through it to mod_jk to Tomcat -- which is far worse than the 8MB/sec, but far better than 80-120K/sec. I'd also look very carefully at the TCP stack configuration. A good starting point is the spec.org site. Solaris is special. Also note that the above was done on a local network with propably no packed drop or out of order arrival. It is relatively easy to optimize a stack in that environment at the expense of it functioning well in an overload or WAN environment. Any stack should find an accecptable compromize across the board. I suspect that some of the above are not. Dw
Re: Apache 1.3.x and 2.0.x Performance Issue
It would be nice if the client used was somehting like 'ab' - which comes with apache ran at 1-100 concurrency; or something like fetch, curl or wget to make the client identical on all platforms. Dw On Tue, 27 Aug 2002, Jess M. Holle wrote: Ian Holsman wrote: Jess M. Holle wrote: Both Apache 1.3.x and 2.0.x suffer a severe perfomance issue when the server is on Windows and the client is Solaris (and perhaps others). Before you stop reading this as simply we know Windows does not perform well, I should point out that this does not occur when the client is Windows or Linux, nor when the server is on the same Windows box but is IIS or Tomcat (standalone)! Some rough download speeds: * recent Apache 1.3.x on Windows: o client on Solaris (8): 80K/sec o client on Linux or Windows: 8MB/sec * recent Apache 2.0.x on Windows: o client on Solaris (8): 120K/sec o client on Linux or Windows: 8MB/sec * IIS on Windows o any client tried: 8-9 MB/sec * Tomcat (standalone) on Windows o any client tried: ~8MB/sec just out of interest does the same thing happen when we have a solaris server and a windows client? I'm not sure. I've not yet tried that (and actually another engineer ran all these tests). I also just noticed that I got some numbers slightly wrong: Apache 2.0.39 on Windows * client on Solaris (8): 649 K/s * client on Linux: 9.1 MB/s [I've appended this amended info to the bug report.] -- Jess Holle
Re: Apache 1.3.x and 2.0.x Performance Issue
Jess M. Holle wrote: Both Apache 1.3.x and 2.0.x suffer a severe perfomance issue when the server is on Windows and the client is Solaris (and perhaps others). Before you stop reading this as simply we know Windows does not perform well, I should point out that this does not occur when the client is Windows or Linux, nor when the server is on the same Windows box but is IIS or Tomcat (standalone)! Some rough download speeds: * recent Apache 1.3.x on Windows: o client on Solaris (8): 80K/sec o client on Linux or Windows: 8MB/sec * recent Apache 2.0.x on Windows: o client on Solaris (8): 120K/sec o client on Linux or Windows: 8MB/sec * IIS on Windows o any client tried: 8-9 MB/sec * Tomcat (standalone) on Windows o any client tried: ~8MB/sec just out of interest does the same thing happen when we have a solaris server and a windows client? Oddly Apache performance was around 1MB/sec when the request was routed through it to mod_jk to Tomcat -- which is far worse than the 8MB/sec, but far better than 80-120K/sec. I have filed bugs (12097 and 12099) against Apache 1.3 and 2.0 on these issues as it is fairly clear that Apache is doing something wrong that both IIS and Tomcat manage not to do. -- Jess Holle
Re: Apache 1.3.x and 2.0.x Performance Issue
Ian Holsman wrote: Jess M. Holle wrote: Both Apache 1.3.x and 2.0.x suffer a severe perfomance issue when the server is on Windows and the client is Solaris (and perhaps others). Before you stop reading this as simply "we know Windows does not perform well", I should point out that this does not occur when the client is Windows or Linux, nor when the server is on the same Windows box but is IIS or Tomcat (standalone)! Some rough download speeds: * recent Apache 1.3.x on Windows: o client on Solaris (8): 80K/sec o client on Linux or Windows: 8MB/sec * recent Apache 2.0.x on Windows: o client on Solaris (8): 120K/sec o client on Linux or Windows: 8MB/sec * IIS on Windows o any client tried: 8-9 MB/sec * Tomcat (standalone) on Windows o any client tried: ~8MB/sec just out of interest does the same thing happen when we have a solaris server and a windows client? I'm not sure. I've not yet tried that (and actually another engineer ran all these tests). I also just noticed that I got some numbers slightly wrong: Apache 2.0.39 on Windows client on Solaris (8): 649 K/s client on Linux: 9.1 MB/s [I've appended this amended info to the bug report.] -- Jess Holle
RE: Apache 1.3.x and 2.0.x Performance Issue
Jess, Were IIS and Tomcat on the same windows hardware as the Apache httpds? Are you seeing network issues (e.g.- rexmits) -- you might have them everywhere, but your Solaris may not be configured well (ndd,MTU,etc). --cw -Original Message- From: Jess M. Holle [mailto:[EMAIL PROTECTED]] Sent: Tuesday, August 27, 2002 4:30 PM To: [EMAIL PROTECTED] Subject: Re: Apache 1.3.x and 2.0.x Performance Issue Ian Holsman wrote: Jess M. Holle wrote: Both Apache 1.3.x and 2.0.x suffer a severe perfomance issue when the server is on Windows and the client is Solaris (and perhaps others). Before you stop reading this as simply we know Windows does not perform well, I should point out that this does not occur when the client is Windows or Linux, nor when the server is on the same Windows box but is IIS or Tomcat (standalone)! Some rough download speeds: * recent Apache 1.3.x on Windows: o client on Solaris (8): 80K/sec o client on Linux or Windows: 8MB/sec * recent Apache 2.0.x on Windows: o client on Solaris (8): 120K/sec o client on Linux or Windows: 8MB/sec * IIS on Windows o any client tried: 8-9 MB/sec * Tomcat (standalone) on Windows o any client tried: ~8MB/sec just out of interest does the same thing happen when we have a solaris server and a windows client? I'm not sure. I've not yet tried that (and actually another engineer ran all these tests). I also just noticed that I got some numbers slightly wrong: Apache 2.0.39 on Windows client on Solaris (8): 649 K/s client on Linux: 9.1 MB/s [I've appended this amended info to the bug report.] -- Jess Holle
Re: Apache 1.3.x and 2.0.x Performance Issue
I can check into this (as I said another engineer ran the tests), but these servers were on the same network segments. Also, our customers are seeing roughly the same numbers (at least for Apache 1.3 -- they're not up to 2.0 yet). -- Jess Holle Clay Webster wrote: Jess, Were IIS and Tomcat on the same windows hardware as the Apache httpds? Are you seeing network issues (e.g.- rexmits) -- you might have them everywhere, but your Solaris may not be configured well (ndd,MTU,etc). --cw -Original Message- From: Jess M. Holle [mailto:[EMAIL PROTECTED]] Sent: Tuesday, August 27, 2002 4:30 PM To: [EMAIL PROTECTED] Subject: Re: Apache 1.3.x and 2.0.x Performance Issue Ian Holsman wrote: Jess M. Holle wrote: Both Apache 1.3.x and 2.0.x suffer a severe perfomance issue when the server is on Windows and the client is Solaris (and perhaps others). Before you stop reading this as simply we know Windows does not perform well, I should point out that this does not occur when the client is Windows or Linux, nor when the server is on the same Windows box but is IIS or Tomcat (standalone)! Some rough download speeds: * recent Apache 1.3.x on Windows: o client on Solaris (8): 80K/sec o client on Linux or Windows: 8MB/sec * recent Apache 2.0.x on Windows: o client on Solaris (8): 120K/sec o client on Linux or Windows: 8MB/sec * IIS on Windows o any client tried: 8-9 MB/sec * Tomcat (standalone) on Windows o any client tried: ~8MB/sec just out of interest does the same thing happen when we have a solaris server and a windows client? I'm not sure. I've not yet tried that (and actually another engineer ran all these tests). I also just noticed that I got some numbers slightly wrong: Apache 2.0.39 on Windows client on Solaris (8): 649 K/s client on Linux: 9.1 MB/s [I've appended this amended info to the bug report.] -- Jess Holle
