RE: Performance Problems.. Server hardware smoked by $500 box?
> The first step would be to try to quantify the performance > difference in serving the actual web pages. Find a single > page that you think is slow on the production system and that > can be accessed without having to be part of a session, and > quantify the performance difference for that page. Remember > you don't care about high load, just a single user request. > You could use apachebench ("ab", comes with apache... > something like "ab -c 1 -n 20 > http://server/path/to/page";) or any simple command line tool > that you can time (eg. repeatedly run "time GET http://server/path/to/page";). Do this from as near as possible to the box you are running the web browser that sees the slowness loading. I ran a couple of tests, and put the one you requested along with one of the more amusing ones to me, below. These request a page that uses PHP sessions (the sessions are kept over NFS) and a couple of database queries. All 3 servers use the same database, tho obviously the database server handles them locally along with handling the files locally vs over NFS like the other two would. I think to sidestep any more testing that may/may not give us more clues, what we're going to press forward with is trying to get another >133Mhz FSB setup in here since they're so cheap and just testing to see if we see similar results if we set it's harddrive up as a mirror of the production web machine. If by using the same OS installation, and only change the underlying hardware, I'd say that would more-than-likely tell us if this is strictly a hardware issue, assuming it performs equally well as the development machine while using the production machine's setup. John Straiton jks@ clickcom.com Clickcom, Inc 704-365-9970x101 Running "ab -c 1 -n 10" on these machines: >Production Web over NFS Benchmarking 209.198.22.161 (be patient).done Server Software:Apache/2.0 Server Hostname:209.198.22.161 Server Port:80 Document Length:67769 bytes Concurrency Level: 1 Time taken for tests: 4.918 seconds Complete requests: 10 Failed requests:0 Broken pipe errors: 0 Total transferred: 681350 bytes HTML transferred: 677690 bytes Requests per second:2.03 [#/sec] (mean) Time per request: 491.80 [ms] (mean) Time per request: 491.80 [ms] (mean, across all concurrent requests) Transfer rate: 138.54 [Kbytes/sec] received >Production DB serving pages locally Benchmarking 209.198.22.35 (be patient).done Server Software:Apache/1.3.27 Server Hostname:209.198.22.35 Server Port:80 Document Length:68585 bytes Concurrency Level: 1 Time taken for tests: 2.232 seconds Complete requests: 10 Failed requests:0 Broken pipe errors: 0 Total transferred: 689870 bytes HTML transferred: 685850 bytes Requests per second:4.48 [#/sec] (mean) Time per request: 223.20 [ms] (mean) Time per request: 223.20 [ms] (mean, across all concurrent requests) Transfer rate: 309.08 [Kbytes/sec] received > Development Server over NFS Benchmarking dev.clickcomworks.net (be patient).done Server Software:Apache/1.3.27 Server Hostname:dev.clickcomworks.net Server Port:8000 Document Length:68237 bytes Concurrency Level: 1 Time taken for tests: 1.799 seconds Complete requests: 10 Failed requests:0 Broken pipe errors: 0 Total transferred: 686380 bytes HTML transferred: 682370 bytes Requests per second:5.56 [#/sec] (mean) Time per request: 179.90 [ms] (mean) Time per request: 179.90 [ms] (mean, across all concurrent requests) Transfer rate: 381.53 [Kbytes/sec] received Now with "ab -c 10 -n 10", which is just beautiful >Production Web over NFS Benchmarking 209.198.22.161 (be patient).done Server Software:Apache/2.0 Server Hostname:209.198.22.161 Server Port:80 Document Length:67769 bytes Concurrency Level: 100 Time taken for tests: 6.424 seconds Complete requests: 10 Failed requests:0 Broken pipe errors: 0 Total transferred: 817620 bytes HTML transferred: 813228 bytes Requests per second:1.56 [#/sec] (mean) Time per request: 64240.00 [ms] (mean) Time per request: 642.40 [ms] (mean, across all concurrent requests) Transfer rate: 127.28 [Kbytes/sec] received >Production DB over NFS Benchmarking 209.198.22.35 (be patient).done Server Software:Apache/1.3.27 Server Hostname:209.198.22.35 Server Port:80 Document Length:68585 bytes Concurrency Level: 10 Time taken for tests: 34.430 seconds Complete requests: 100 Failed requests:0 Broken pipe errors: 0 Total transferred: 6898700 bytes HTML transferred: 6858500 bytes Requests per second:2.90 [#/sec] (mean) Time per request: 3443.00 [m
Re: Performance Problems.. Server hardware smoked by $500 box?
On Thu, Sep 11, 2003 at 05:30:59PM -0400, John Straiton wrote: > If 5.1-C has debugging on by default then , yes, I'd concur that we have > those features turned on. 5.1-CURRENT indeed has a number of debugging features enabled by default, which can cause significant performance loss under load. Kris pgp0.pgp Description: PGP signature
Re: Performance Problems.. Server hardware smoked by $500 box?
> > Post your kernel configs, or better yet, do a diff -u between the > > 5.0-R and the 5.1-C kernel configs. I bet dime to dollar you've > > got some debugging options enabled in the 5.1-C config. At the > > very least you haven't remove the debugging options from your > > malloc options. > > *frown* The 5.0 development machine is running GENERIC. Being that > we never customized the kernel, we don't have a backup copy of it. I > can say since I'd have been the one to change it, that the only > differences between GENERIC and what we put into service is that we > always remove the device listings for things we don't have any use > for, like ISA nic's and such. I almost never change any of the other > features. Removing unused drivers shouldn't make any difference. > If 5.1-C has debugging on by default then , yes, I'd concur that we > have those features turned on. However the production machine was > 4.8-R when we noticed the problem. From what it sounds, it should > have been faster due to that fact. The only thing that's been > constant in this situation is that the development machine hasn't > changed. Everything I've done has been trying to change variables on > the production machine to either match or surpass the development to > bring it up to it's speed. hrm... On your development box, do you have any boot time configuration options set, how about differing values in sysctl.conf? Hrm, let me re-read your original post. Apache: Are you sure you have your reverse DNS setup correctly? Are you doing host name resolution in Apache for your logs? If so, turn that off! bonnie: It's very possible that the CPU is making a difference here if bonnie is getting near 100% hit rates for the cache. Given that you're getting good throughput, I'd double check that you're machines are at 100Mbps-TX, full duplex in ifconfig. I bet one of them is half-duplex and that's the difference there. I'm betting there are some sysctl's that are different between these machines, along with possibly Apache and your DNS not being setup correctly. > Roadmap for the production machine so far: > > Upgrade apache/php to newest in ports. I haven't seen anyone claim that Apache2 is faster than Apache 1. If Apache2 is faster, could someone provide some evidence? > Add RAM This won't make a difference unless you're swapping our out of RAM. > Upgrade OS to 5.1 from 4.8, reinstall every package in pkg_info Downgrade you mean? :) 4.8 is going to be more battle proven than 5.1, so I'd recommend using 4.8 for that reason alone. If you do want to use 5.1 for the sake of helping 5.1 become more mature, very cool. > Update apache to 2.X from 1.3.X & reinstall php as is required See above comment on Apache. > Swap network interfaces between the two onboard ones This could make some difference, but 6MBps is more than plenty throughput for webserving via NFS. > Swap ethernet cables with the development machine This would impact network performance, something else is going on. > Swap ethernet ports with the development machine Again, this would impact network performance, and at 6MBps, that's well more than enough for you to get reasonable performance out of a 5.1 or 4.8 box. Something basic and simple is going on here. DNS: dig my.test.box.example.com. a dig d.c.b.a.in-addr.arpa. ptr apache: grep Hostname /usr/local/etc/apache/httpd.conf -sc -- Sean Chittenden ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: Performance Problems.. Server hardware smoked by $500 box?
> There's lots of tricky stuff that can be going wrong. > I spent some time in my last two jobs (anybody got > a new one in NJ?) on speeding up stuff like this > and the first thing I try to do is put some kind of > steady-state load on the boxen and monitor each box involved > with systat 1 -vmstat . There's one hell of a lot of > information there, and interactions are sometimes hard to > see. If the CPU is fully occupied, it could be the network > stack (which will NOT show up at interrupt level) and that > can depend on what interface chipset you're using as well. > Or it could be ... well, get the data first. If you'd like > to send me a few sample screens, I'll try to make suggestions > on what to check next. You want to have a series from each > of the three configurations you're using. And being able to > _watch_ what's happening on systat is worth a whole lot of > non-sequenced snapshots. I'll take a look at this. > Are you running firewall software on the production > machine? Are you trying to hack me? *chuckle* j/k > I don't know how the FreeBSD version will > affect performance, but it can't help. How about > the reports from top ? What do they say? What's > soaking up the processor? If you saw the two machines, rc.d/* and rc.conf are nearly identical. There aren't any services that run on one that aren't exactly the same on the other except the differences between the OS versions as best I can tell. In looking at "top -qSi" (or even normal) the only things that tend to show up at the top are httpd processes. The development machine has the ata irq up there whereas the production doesn't seem to have disk access in the top few entries. Just reinforcing that even with more to do, the development machine smokes this Dell iron. > Can you try running the back end box on a simple > disk without the RAID in the way? I don't recall > all the properties of RAID 5 right now, but in general > RAID trades disk transactions away to get disk > throughput. In your application, you probably need > transactions more than throughput. No, it has no other interfaces configured that I could get into without taking the machine down to hook up some cables. This thing boots of the hardware RAID and has a cage in the front for the drives (hotswap). But being that the RAID is one of the few things that IS the same between production and development (the raid serves files via NFS to both incarnations of the webserver in the same fashion) , I'd have to get an idea why this was something I'd want to delve further into. > Dumb quesion: have you tried swapping cables/ports > on the ethernet connections? Does one link support > jumbo frames and the other not? How about network > buffers: have you got enough configured, and how > many are tied up at a time? I'm not familiar with those configuration options but if it makes a difference, I basically use completely default setups between the xl0 and fxp0 drivers being used in these scenarios, with GENERIC being the kernel for all intents and purposes between both machines. Unless one driver is set up different than the other or has inheirent speed advantages, they should both be equivalent. Yes, all cabling/ports and even in the case of the production machine, interfaces have been swapped/changed. Another good idea that didn't pan out... Thanks for the help! John Straiton jks@ clickcom.com Clickcom, Inc 704-365-9970x101 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: Performance Problems.. Server hardware smoked by $500 box?
> Post your kernel configs, or better yet, do a diff -u between > the 5.0-R and the 5.1-C kernel configs. I bet dime to dollar > you've got some debugging options enabled in the 5.1-C > config. At the very least you haven't remove the debugging > options from your malloc options. *frown* The 5.0 development machine is running GENERIC. Being that we never customized the kernel, we don't have a backup copy of it. I can say since I'd have been the one to change it, that the only differences between GENERIC and what we put into service is that we always remove the device listings for things we don't have any use for, like ISA nic's and such. I almost never change any of the other features. If 5.1-C has debugging on by default then , yes, I'd concur that we have those features turned on. However the production machine was 4.8-R when we noticed the problem. From what it sounds, it should have been faster due to that fact. The only thing that's been constant in this situation is that the development machine hasn't changed. Everything I've done has been trying to change variables on the production machine to either match or surpass the development to bring it up to it's speed. Roadmap for the production machine so far: Upgrade apache/php to newest in ports. Add RAM Upgrade OS to 5.1 from 4.8, reinstall every package in pkg_info Update apache to 2.X from 1.3.X & reinstall php as is required Swap network interfaces between the two onboard ones Swap ethernet cables with the development machine Swap ethernet ports with the development machine John Straiton jks@ clickcom.com Clickcom, Inc 704-365-9970x101 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Performance Problems.. Server hardware smoked by $500 box?
John Straiton writes: > I'm pretty confused right now with trying to > determine the nature of a performance problem ... > on one of my servers. ... in pulling up websites > from the machine, my silly POS development > box has nearly double performance ... There's lots of tricky stuff that can be going wrong. I spent some time in my last two jobs (anybody got a new one in NJ?) on speeding up stuff like this and the first thing I try to do is put some kind of steady-state load on the boxen and monitor each box involved with systat 1 -vmstat . There's one hell of a lot of information there, and interactions are sometimes hard to see. If the CPU is fully occupied, it could be the network stack (which will NOT show up at interrupt level) and that can depend on what interface chipset you're using as well. Or it could be ... well, get the data first. If you'd like to send me a few sample screens, I'll try to make suggestions on what to check next. You want to have a series from each of the three configurations you're using. And being able to _watch_ what's happening on systat is worth a whole lot of non-sequenced snapshots. Are you running firewall software on the production machine? I don't know how the FreeBSD version will affect performance, but it can't help. How about the reports from top ? What do they say? What's soaking up the processor? Running on a 1GHz PIII two years ago, I was able to get a web proxy (not squid!) to serve 1500+ requests per second, with about 200 MBit/sec of ethernet traffic (inbound and out). (The product never made it into full-scale production, largely due to financial problems in the large, well-known corporation.) So the problem isn't horsepower, but something not using it well. Can you try running the back end box on a simple disk without the RAID in the way? I don't recall all the properties of RAID 5 right now, but in general RAID trades disk transactions away to get disk throughput. In your application, you probably need transactions more than throughput. Dumb question: have you tried swapping cables/ports on the ethernet connections? Does one link support jumbo frames and the other not? How about network buffers: have you got enough configured, and how many are tied up at a time? Performance is often a negative art: find the worst roadblock and remove it, then the next worst after that, and so forth. Mark Terribile __ Do you Yahoo!? Yahoo! SiteBuilder - Free, easy-to-use web site design software http://sitebuilder.yahoo.com ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Performance Problems.. Server hardware smoked by $500 box?
On Thu, 11 Sep 2003, John Straiton wrote: > Greets! > > I'm pretty confused right now with trying to determine the nature of a > performance problem I'm having on one of my servers. The server is a > webserver with a separate db/file server sitting behind it. The issue is > that in pulling up websites from the machine, my silly POS development > box has nearly double performance although one would think it shouldn't. you need to quantify the performance differences starting with replicating what you see, and work down from there. Do the differences in the numbers you posted mean anything? No idea. I wouldn't expect them to make any difference for a single user hitting the server, but it is possible they are related to some problem somewhere. The easiest way to tell, however, is to start with the problem you do see (ie. things are slow loading) and work down through the software stack from there. The first step would be to try to quantify the performance difference in serving the actual web pages. Find a single page that you think is slow on the production system and that can be accessed without having to be part of a session, and quantify the performance difference for that page. Remember you don't care about high load, just a single user request. You could use apachebench ("ab", comes with apache... something like "ab -c 1 -n 20 http://server/path/to/page";) or any simple command line tool that you can time (eg. repeatedly run "time GET http://server/path/to/page";). Do this from as near as possible to the box you are running the web browser that sees the slowness loading. Until you can reproduce and quantify a performance difference at this level, don't worry about digging deeper. Once you can, keep taking one step closer. Try requesting a page that doesn't hit the database. Try setting things up so NFS isn't being used. Try making the request from the same machine the web server is running on. etc. Your goal here is to eliminate as many components as possible while still being able to reproduce the high level problem. So, for example, if you can reproduce it on a page that doesn't hit the database... you can eliminate that from further consideration. Unless your application is extremely heavyweight and demanding on hardware, or there is some bug in one of the drivers or configuration, none of the hardware differences would normally have any effect on the symptoms you say you are seeing. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: Performance Problems.. Server hardware smoked by $500 box?
> > I'm pretty confused right now with trying to determine the > nature of a > > performance problem I'm having on one of my servers. The > server is a > > webserver with a separate db/file server sitting behind it. > The issue > > is that in pulling up websites from the machine, my silly POS > > development box has nearly double performance although one > would think > > it shouldn't. [...] > > Disclaimer: these are random, uninformed guesses... > > - Is it possible the server has too much RAM? I'll look into this. > - Are you using the same db server as the backend for your > development box? Thanks for your suggestions, unfortunately- yes, both scenerios in production (webserver + db server or db server acting as both) are slower than the development box serving off of the db server. There are no local services other than apache on the development machine. John Straiton jks@ clickcom.com Clickcom, Inc 704-365-9970x101 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Performance Problems.. Server hardware smoked by $500 box?
Chris Pressey wrote: [ ... ] - Is it possible the server has too much RAM? I don't remember where I heard that that can degrade performance, but I'm pretty sure it was on one of the freebsd lists a couple of months ago. One of the early Pentium Pro/P2 chipsets, either the 430VX or the 430FX?, was unable to perform L2 caching of main memory above 64MB or some such, but aside from this sort of hardware limitation, more memory is going to be better for FreeBSD. [ This isn't true of all operating systems, but Unix systems like FreeBSD use some variant of LRU page replacement algorithm for VM, probably in conjunction with a global page-fault frequency algorithm to help size process working sets, which does not suffer from Belady's anomaly. At one point, Windows used FIFO with working set as the VM paging algorithm, which does not maintain the stack-based invariant of resident pages and thus sometimes making more memory available under Windows results in worse performance due to software issues. The classic MacOS used to have a similarly brain-dead VM system, which is why Mac users have generally resisted the notion of enabling or using VM, although they seem to be coping with Mach and Darwin ] -- -Chuck ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Performance Problems.. Server hardware smoked by $500 box?
On Thu, 11 Sep 2003 09:42:41 -0400 "John Straiton" <[EMAIL PROTECTED]> wrote: > Greets! > > I'm pretty confused right now with trying to determine the nature of a > performance problem I'm having on one of my servers. The server is a > webserver with a separate db/file server sitting behind it. The issue > is that in pulling up websites from the machine, my silly POS > development box has nearly double performance although one would think > it shouldn't. [...] Disclaimer: these are random, uninformed guesses... - Is it possible the server has too much RAM? I don't remember where I heard that that can degrade performance, but I'm pretty sure it was on one of the freebsd lists a couple of months ago. - Are you using the same db server as the backend for your development box? i.e. Maybe the lag is between the two servers, and if they're running on the same machine in your case, that could be why you don't see as dramatic a slowdown. Again, these are just shots in the dark :) -Chris ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"