Re: Finding the Bottleneck
Hi Bertrand, Thanks for your insightful email! Just so you know, this server is an: AMD K6-2 500Mhz, 128M-133Mhz, 2 UDMA100 drives (IBM), 10M bandwidth. The server runs Apache, Qmail, vpopmail (for pop3). The webserver is not primary (doesn't have to have the fastest response time), as this is mainly for the mailing lists. The 2 hard disks are on 2 different IDE channels, as putting both disks on the same cable would drastically reduce performance of both disks. The way it is organized is that the mail spool/queue is on the 2nd disk, while the OS and programs are on disk 1. Logging is also performed on disk 1, so that writing to the mail log won't interfere with the mail queue (as they commonly both occur simultaneously). From MY understanding, the load average shows how many programs are running, and not really how stressed the CPU is. I'm not sure exactly sure how this works (please correct me if i'm wrong) but 1 program taking 80% CPU might have load average of 2, while 100 programs taking 0.5% each would take 50% CPU and have load average of 8. Is that correct thinking? The reason I'm saying that is because qmail spawns a program called qmail-remote for EACH email to be sent. So if 200 concurrent emails are being sent at one time, then 200 qmail-remotes are created. Granted... each of these takes up tiny amounts of ram and CPU time, but I suspect (if the above statements are correct) that the load average would be artificially inflated because of it. We don't use NFS on this server. NFS on linux, as you said, is pretty crummy and should be avoided if possible. We simply put the mail queue on a seperate hard disk. pop3 load is extremely minimal. Its mainly an outgoing mail server (mail list server). People essentially use the website to send mail to the mailing list, so load of the pop3 server won't be an issue. About the HK job market, do you have any official qualifications (eg. a degree, diploma, cert., etc.)? In HK bosses like to see that... even more than in some other countries like Australia (not sure bout US). BTW. was your mother headmistress of St. Paul before? Sincerely, Jason - Original Message - From: schemerz [EMAIL PROTECTED] To: Jason Lim [EMAIL PROTECTED] Sent: Wednesday, June 06, 2001 3:57 PM Subject: Re: Finding the Bottleneck On Wed, Jun 06, 2001 at 11:53:22AM +0800, Jason Lim wrote: Hi all, I was wondering if there is a way to find out what/where the bottleneck of a large mail server is. A client is running a huge mail server that we set up for them (running qmail), but performance seems to be limited somewhere. Qmail has already been optimized as far as it can go (big-todo patch, large concurrency patch, etc.). We're thinking one of the Hard Disks may be the main bottleneck (the mail queue is already on a seperate disk on a seperate IDE channel from other disks). Is there any way to find out how utilized the IDE channel/hard disk is, or how hard it is going? Seems that right now the only way we really know is by looking at the light on the server case (how technical ;-) ). Must be a better way... The bottleneck wouldn't be bandwidth... it is definately with the server. Perhaps the CPU or kernel is the bottleneck (load average: 4.84, 3.94, 3.88, going up to 5 or 6 during heavy mailing)? Is that normal for a large mail server? We haven't run such a large mail server before (anywhere between 500k to 1M per day so far, increasing each day), so ANY tips and pointers would be greatly appreciated. We've already been playing around with hdparm to see if we can tweak the disks, but doesn't seem to help much. Maybe some cache settings we can fiddle with? Maybe the mail queue disk could use a different file cache setting (each email being from 1K to 10K on average)? Thanks in advance! Sincerely, Jason Jason, I am a lurker on the list. I don't run linux anymore but recently a friend of mine encountered a similar load problem. Granted, he was running sendmail, but his main bottleneck wasn't the mta at all. I will explain his situation and see if you will find any similarites with yours... The discussion pertains to the mailserver at sunflower.com. When my friend took over the previous admin left the place in a mess. One of the machines he inherited was a dual 400 mhz penta 2 with about 256 megs of ram. It would be quite adequate for serving about 5000 accounts except... 1) The server was running on a box using the promise IDE raid controllers. IDE for a small server would work fine, but this box was getting hit alot. 2) This server was also the pop server. One of the reasons of design to run pop and sendmail on the same box was because nfs-exporting mail spools on linux was unsafe (nfs still sucks and WILL suck for ad infinitum, BUT linux was and still is probably the worst platform to employ nfs in a production system). So nfs exporting the mail spools to a seperate pop3 boxes were out of the question. Another local ISP,
Virus scanners
Hi all, Can any body suggest a decent virus scanner for Linux? I've heard that Sophos is good, but its a bit pricy... But anything that'll work with amavis would be a great help. Thx in advanced. Simon -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
On Wednesday 06 June 2001 10:51, Jason Lim wrote: Just so you know, this server is an: AMD K6-2 500Mhz, 128M-133Mhz, 2 UDMA100 drives (IBM), 10M bandwidth. How much swap is being used? If you have any serious amount of mail being delivered then having a mere 128M of RAM will seriously hurt performance! RAM is also cheap and easy to upgrade... mainly for the mailing lists. The 2 hard disks are on 2 different IDE channels, as putting both disks on the same cable would drastically reduce performance of both disks. In my tests so far I have not been able to show drastic performance difference. I have shown about a 20% performance benefit for using separate cables... The way it is organized is that the mail spool/queue is on the 2nd disk, while the OS and programs are on disk 1. Logging is also performed on disk 1, so that writing to the mail log won't interfere with the mail queue (as they commonly both occur simultaneously). Where is swap? Take note of the following paragraph from syslog.conf(5): You may prefix each entry with the minus ``-'' sign to omit syncing the file after every logging. Note that you might lose information if the system crashes right behind a write attempt. Nevertheless this might give you back some performance, especially if you run programs that use logging in a very verbose manner. Do that for all logs apart from kern.log! Then syslogd will hardly use any disk access. From MY understanding, the load average shows how many programs are running, and not really how stressed the CPU is. I'm not sure exactly sure how this works (please correct me if i'm wrong) but 1 program taking 80% CPU might have load average of 2, while 100 programs taking 0.5% each would take 50% CPU and have load average of 8. Is that correct thinking? Not. 1 program taking up all CPU time will give a load average of 1.00. 1 program being blocked on disk IO (EG reading from a floppy disk) will give a load average of 1.00. Two programs blocked on disk IO to different disks and a third program that's doing a lot of CPU usage will result in load average of 3.00 while the machine is running as efficiently as it can. Load average isn't a very good way of measuring system use! We don't use NFS on this server. NFS on linux, as you said, is pretty crummy and should be avoided if possible. We simply put the mail queue on a seperate hard disk. Actually if you have the latest patches then NFS should be quite solid. Now firstly the OS and the syslog will not use the disk much at all if you have enough RAM that the machine doesn't swap and has some spare memory for caching. Boost the machine to 256M. Don't bother with DDR RAM as it won't gain you anything, get 384 or 512M if you can afford it. Next the most important thing for local mail delivery is to have the queue on a separate disk to the spool. Queue and spool writes are independant and the data is immidiately sync'd. Having them on separate disks can provide serious performance benefits. Also if your data is at all important to you then you should be using RAID. Software RAID-1 in the 2.4.x kernels and with the patch for 2.2.x kernels is very solid. I suggest getting 4 drives and running two RAID-1 sets, one for the OS and queue, the other for the spool. RAID-1 will improve read speed as the system will be able to execute two read requests from the RAID-1 at the same time. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Virus scanners
Is this for checking virus's on mail servers or on linux workstations? Are there virus's attacking linux machines? James On Wed, 6 Jun 2001, Dmitry Litovchenko wrote: Hi all, Can any body suggest a decent virus scanner for Linux? I've heard that Sophos is good, but its a bit pricy... But anything that'll work with amavis would be a great help. Thx in advanced. Simon Please visit www.avp.ru (you will need to look for English version on the title page if you're unfamiliar with Russian :) There is Antiviral Toolkit Pro - antivirus scanner fior Linux, FreeBSD and Windoze servers and workstations. I didn't try Linux version yet, but windoze version is popular in Russia and Ukraine, not legal copies, of course, most of us here cannot buy commercial software :) Maybe pricing may satisfact you... For example such an software is present (my mailer told me so) on freemail.ukr.net webserver, etc. This is good antiviral software to play with. There is demo for RedHat workstations... -- Sincerely, Dmitry -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
firewall question...
PB Hello all, Can anyone tell me if there is a good reason to PB allow connections to a local DNS port(53) from remote PB privledges ports( 1024)? Yes. Windows and possibly some other systems (little internet devices maybe) do not have this privileged port notion. Why do you care what port people send _from_? cheers, BM -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: firewall question...
Bulent Murtezaoglu wrote: PB Hello all, Can anyone tell me if there is a good reason to PB allow connections to a local DNS port(53) from remote PB privledges ports( 1024)? Yes. Windows and possibly some other systems (little internet devices maybe) do not have this privileged port notion. Why do you care what port people send _from_? Paranoia. Generally accepted practice when setting up a firewall is to be as restrictive as possible without breaking things, that includes restricting the originating ports. For example I want to give people access to port 80 but if someone is trying to connect to port 80 from port 25 their system is either broken or they are attempting to do something that you probably don't want them to do. There is no good reason to allow that connection. Thanks for the Windows info but I don't understand how can they not have the notion of privledged ports? Aren't privledged ports just generally accepted port assignments? And I'm not sure that Windows is a *good* reason! :-) Pete -- http://www.elbnet.com ELB Internet Services, Inc. Web Design, Computer Consulting, Internet Hosting -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
Hi Russell, Here is the result of top: 05:51:18 up 5 days, 22:38, 1 user, load average: 6.60, 7.40, 6.51 119 processes: 106 sleeping, 11 running, 2 zombie, 0 stopped CPU states: 16.4% user, 18.3% system, 0.0% nice, 65.3% idle Mem:128236K total, 124348K used, 3888K free,72392K buffers Swap: 289160K total,0K used, 289160K free, 9356K cached And of qmail-qstat: sh-2.05# qmail-qstat messages in queue: 108903 messages in queue but not yet preprocessed: 19537 Swap is on Disk 1, because mail queue/spool is on Disk 2. I also already added the - in front of most entries except the emergency or critical ones (if I didn't do it, the load was way higher just writing the log files). Concerning the mail queue and spool being on the same disk, the reason is that there is virtually no emails incoming, 99.999% outgoing. About running software raid... I've heard that the CPU usage is increased dramatically if you use any form of software raid. Is that true? Actually... i doubt the customer would be willing to pay us to implement this for him on a hardware level. Good raid cards with good amounts of ram don't come cheap last time I checked... :-/ Sincerely, Jason - Original Message - From: Russell Coker [EMAIL PROTECTED] To: Jason Lim [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Wednesday, June 06, 2001 8:05 PM Subject: Re: Finding the Bottleneck On Wednesday 06 June 2001 10:51, Jason Lim wrote: Just so you know, this server is an: AMD K6-2 500Mhz, 128M-133Mhz, 2 UDMA100 drives (IBM), 10M bandwidth. How much swap is being used? If you have any serious amount of mail being delivered then having a mere 128M of RAM will seriously hurt performance! RAM is also cheap and easy to upgrade... mainly for the mailing lists. The 2 hard disks are on 2 different IDE channels, as putting both disks on the same cable would drastically reduce performance of both disks. In my tests so far I have not been able to show drastic performance difference. I have shown about a 20% performance benefit for using separate cables... The way it is organized is that the mail spool/queue is on the 2nd disk, while the OS and programs are on disk 1. Logging is also performed on disk 1, so that writing to the mail log won't interfere with the mail queue (as they commonly both occur simultaneously). Where is swap? Take note of the following paragraph from syslog.conf(5): You may prefix each entry with the minus ``-'' sign to omit syncing the file after every logging. Note that you might lose information if the system crashes right behind a write attempt. Nevertheless this might give you back some performance, especially if you run programs that use logging in a very verbose manner. Do that for all logs apart from kern.log! Then syslogd will hardly use any disk access. From MY understanding, the load average shows how many programs are running, and not really how stressed the CPU is. I'm not sure exactly sure how this works (please correct me if i'm wrong) but 1 program taking 80% CPU might have load average of 2, while 100 programs taking 0.5% each would take 50% CPU and have load average of 8. Is that correct thinking? Not. 1 program taking up all CPU time will give a load average of 1.00. 1 program being blocked on disk IO (EG reading from a floppy disk) will give a load average of 1.00. Two programs blocked on disk IO to different disks and a third program that's doing a lot of CPU usage will result in load average of 3.00 while the machine is running as efficiently as it can. Load average isn't a very good way of measuring system use! We don't use NFS on this server. NFS on linux, as you said, is pretty crummy and should be avoided if possible. We simply put the mail queue on a seperate hard disk. Actually if you have the latest patches then NFS should be quite solid. Now firstly the OS and the syslog will not use the disk much at all if you have enough RAM that the machine doesn't swap and has some spare memory for caching. Boost the machine to 256M. Don't bother with DDR RAM as it won't gain you anything, get 384 or 512M if you can afford it. Next the most important thing for local mail delivery is to have the queue on a separate disk to the spool. Queue and spool writes are independant and the data is immidiately sync'd. Having them on separate disks can provide serious performance benefits. Also if your data is at all important to you then you should be using RAID. Software RAID-1 in the 2.4.x kernels and with the patch for 2.2.x kernels is very solid. I suggest getting 4 drives and running two RAID-1 sets, one for the OS and queue, the other for the spool. RAID-1 will improve read speed as the system will be able to execute two read requests from the RAID-1 at the same time. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal
