Re: Finding the Bottleneck
On Thursday 14 June 2001 15:31, Marc Haber wrote: > <[EMAIL PROTECTED]> wrote: > >One significant benefit of ReiserFS is that it is journalled and does > > not require a lengthy fsck operation after a power failure. > > However, if the journal gets corrupted, you are in for serious > trouble, because Hans Reiser didn't plan this to happen and didn't > write any fsck code. I have seen this happening, with severe data That is offensive. There has been a reiserfsck program for quite some time (long before it was included in standard kernels or in common use). There have been problems with the fsck program, but they are getting fixed. > loss. AFAIK, there is now a "third party ReiserFS fsck", but I never > actually tried it. I have not heard of any such program, I am on the ReiserFS list, and I am known for testing out various ReiserFS things. So if there was such a program then I am sure I would have heard of it. I expect that you are getting confused with a bad-blocks program that has been written. Currently ReiserFS has no support for detecting bad blocks on hard drives, this is because most hardware that is in use will never develop bad sectors. As an example I recently had a read error on a 1.5 year old IDE hard drive. I ran "cat /dev/zero > /dev/hdc" and the problem was gone for good. For most people there is no need to have a file system handle bad blocks. I have been using Linux since 1993, currently I run 9 Linux machines (the number varies), and I have never had a bad block on a hard drive on a Linux machine with one exception - a client received a bad shipment of hard drives that died under Linux, no bad block detection would save them, and incidentally they were running Ext2. > Additionally, ReiserFS has some issues that shouldn't be present in a > modern FS (for example, it is not 64 bit clean, as I am told). You mean there are issues with the Alpha and UltraSPARC ports? Personally I don't care as SPARC is slow and expensive, and I don't do the things that Alpha is good at (floating point). Files larger than 2G in size work well for me on ReiserFS. Apparently kernel 2.4.5 has a bug that limits ReiserFS to 4G (not sure if it's a kernel bug or a ReiserFS bug). Kernel 2.4.4 apparently allows larger files. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page
Re: Finding the Bottleneck
On Thursday 14 June 2001 15:31, Marc Haber wrote: > <[EMAIL PROTECTED]> wrote: > >One significant benefit of ReiserFS is that it is journalled and does > > not require a lengthy fsck operation after a power failure. > > However, if the journal gets corrupted, you are in for serious > trouble, because Hans Reiser didn't plan this to happen and didn't > write any fsck code. I have seen this happening, with severe data That is offensive. There has been a reiserfsck program for quite some time (long before it was included in standard kernels or in common use). There have been problems with the fsck program, but they are getting fixed. > loss. AFAIK, there is now a "third party ReiserFS fsck", but I never > actually tried it. I have not heard of any such program, I am on the ReiserFS list, and I am known for testing out various ReiserFS things. So if there was such a program then I am sure I would have heard of it. I expect that you are getting confused with a bad-blocks program that has been written. Currently ReiserFS has no support for detecting bad blocks on hard drives, this is because most hardware that is in use will never develop bad sectors. As an example I recently had a read error on a 1.5 year old IDE hard drive. I ran "cat /dev/zero > /dev/hdc" and the problem was gone for good. For most people there is no need to have a file system handle bad blocks. I have been using Linux since 1993, currently I run 9 Linux machines (the number varies), and I have never had a bad block on a hard drive on a Linux machine with one exception - a client received a bad shipment of hard drives that died under Linux, no bad block detection would save them, and incidentally they were running Ext2. > Additionally, ReiserFS has some issues that shouldn't be present in a > modern FS (for example, it is not 64 bit clean, as I am told). You mean there are issues with the Alpha and UltraSPARC ports? Personally I don't care as SPARC is slow and expensive, and I don't do the things that Alpha is good at (floating point). Files larger than 2G in size work well for me on ReiserFS. Apparently kernel 2.4.5 has a bug that limits ReiserFS to 4G (not sure if it's a kernel bug or a ReiserFS bug). Kernel 2.4.4 apparently allows larger files. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
On Fri, Jun 08, 2001 at 09:59:50PM +0800, Jason Lim wrote: > I have also thought about that... but if you have a look at Qmail's > website (http://www.qmail.org) then you'll see that a number of > extremely large mail companies (hotmail for one) uses qmail for... get > this... outgoing mail. They could've easily chosen sendmail, postfix > (maybe it wasn't around when they designed their systems), zmailer, > etc. but those chose qmail. > > Why would that be? probably because qmail was the only high-performance MTA readily available at the time (not quite true - zmailer was around but it's not well known and is considerably harder to configure and tune than qmail) however for certain types of mailing lists, qmail's one-smtp-connection-per-recipient can be a huge bottleneck in delivery performance. if you send personalised messages to each recipient, then it makes no difference - and qmail's easy VERP support makes bounce-handling a breeze. OTOH, if you run a "normal" mailing list where a single message is BCC-ed to thousands (or hundreds of thousands, or millions) of recipients then an MTA that delivers to multiple recipients at the same domain within one smtp session is MUCH faster. e.g. say you have a mailing list with 200,000 subscribers. of these, say 5,000 are hotmail.com addresses. with qmail, delivering the 5,000 messages to hotmail.com users would take 5,000 separate successful smtp sessions - with all the latency and overhead establishing a connection each time. that's bad enough by itself, but hotmail's mail servers are often overloaded and it can sometimes take several attempts to even get a successful connection. with postfix (or sendmail or exim - but i wouldn't seriously recommend either of those as a high-performance MTA), delivering the same 5,000 messages to hotmail would take 100 successful smtp sessions with default settings. it could take more (or less), depending on what value you set for $smtp_destination_recipient_limit (the postfix default is 50) - the limit is there to work around the fact that many mail servers reject messages with too many recipientsit's probably safe to increase it to a few hundred before you notice any servers rejecting messages. of course, you can still do personalised one-message-per-recipient style lists with postfix if you want to. the difference is that with postfix it is an option whereas with qmail it is mandatory. hotmail's only an extreme example. yahoo.com is similar, and so are many other "free mail" providers. with large mailing lists, it's not uncommon to get dozens of recipients even to "normal" domains - e.g company domains and ISP domains. the delays are cumulative. it would not be at all unusual to see a 10:1 or even 20:1 overall difference in the number of smtp sessions required to deliver a large mailing list. on a more general note: i've used most of the available MTAs heavily over the last 8 years or so, and i've at least dabbled with all of them. until postfix came along i used qmail as my MTA of choice. now i use postfix exclusively (except on a handful of "legacy" systems where it's not worth the bother of converting them to postfix). postfix has all the advantages of qmail, without the disadvantages. the only situation where i would now choose qmail over postfix is if there was an absolute requirement to use ezmlm or some other mailing list manager or mail-related tool that only works with qmail. craig -- craig sanders <[EMAIL PROTECTED]> Fabricati Diem, PVNC. -- motto of the Ankh-Morpork City Watch
Re: Finding the Bottleneck
On Fri, Jun 08, 2001 at 09:59:50PM +0800, Jason Lim wrote: > I have also thought about that... but if you have a look at Qmail's > website (http://www.qmail.org) then you'll see that a number of > extremely large mail companies (hotmail for one) uses qmail for... get > this... outgoing mail. They could've easily chosen sendmail, postfix > (maybe it wasn't around when they designed their systems), zmailer, > etc. but those chose qmail. > > Why would that be? probably because qmail was the only high-performance MTA readily available at the time (not quite true - zmailer was around but it's not well known and is considerably harder to configure and tune than qmail) however for certain types of mailing lists, qmail's one-smtp-connection-per-recipient can be a huge bottleneck in delivery performance. if you send personalised messages to each recipient, then it makes no difference - and qmail's easy VERP support makes bounce-handling a breeze. OTOH, if you run a "normal" mailing list where a single message is BCC-ed to thousands (or hundreds of thousands, or millions) of recipients then an MTA that delivers to multiple recipients at the same domain within one smtp session is MUCH faster. e.g. say you have a mailing list with 200,000 subscribers. of these, say 5,000 are hotmail.com addresses. with qmail, delivering the 5,000 messages to hotmail.com users would take 5,000 separate successful smtp sessions - with all the latency and overhead establishing a connection each time. that's bad enough by itself, but hotmail's mail servers are often overloaded and it can sometimes take several attempts to even get a successful connection. with postfix (or sendmail or exim - but i wouldn't seriously recommend either of those as a high-performance MTA), delivering the same 5,000 messages to hotmail would take 100 successful smtp sessions with default settings. it could take more (or less), depending on what value you set for $smtp_destination_recipient_limit (the postfix default is 50) - the limit is there to work around the fact that many mail servers reject messages with too many recipientsit's probably safe to increase it to a few hundred before you notice any servers rejecting messages. of course, you can still do personalised one-message-per-recipient style lists with postfix if you want to. the difference is that with postfix it is an option whereas with qmail it is mandatory. hotmail's only an extreme example. yahoo.com is similar, and so are many other "free mail" providers. with large mailing lists, it's not uncommon to get dozens of recipients even to "normal" domains - e.g company domains and ISP domains. the delays are cumulative. it would not be at all unusual to see a 10:1 or even 20:1 overall difference in the number of smtp sessions required to deliver a large mailing list. on a more general note: i've used most of the available MTAs heavily over the last 8 years or so, and i've at least dabbled with all of them. until postfix came along i used qmail as my MTA of choice. now i use postfix exclusively (except on a handful of "legacy" systems where it's not worth the bother of converting them to postfix). postfix has all the advantages of qmail, without the disadvantages. the only situation where i would now choose qmail over postfix is if there was an absolute requirement to use ezmlm or some other mailing list manager or mail-related tool that only works with qmail. craig -- craig sanders <[EMAIL PROTECTED]> Fabricati Diem, PVNC. -- motto of the Ankh-Morpork City Watch -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
On Mon, 11 Jun 2001 17:14:46 +0200, Russell Coker <[EMAIL PROTECTED]> wrote: >One significant benefit of ReiserFS is that it is journalled and does not >require a lengthy fsck operation after a power failure. However, if the journal gets corrupted, you are in for serious trouble, because Hans Reiser didn't plan this to happen and didn't write any fsck code. I have seen this happening, with severe data loss. AFAIK, there is now a "third party ReiserFS fsck", but I never actually tried it. Additionally, ReiserFS has some issues that shouldn't be present in a modern FS (for example, it is not 64 bit clean, as I am told). >I would not recommend that you change to ReiserFS at this time. ACK. ACK. ACK: Greetings Marc -- -- !! No courtesy copies, please !! - Marc Haber | " Questions are the | Mailadresse im Header Karlsruhe, Germany | Beginning of Wisdom " | Fon: *49 721 966 32 15 Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fax: *49 721 966 31 29
Re: Finding the Bottleneck
On Mon, 11 Jun 2001 17:14:46 +0200, Russell Coker <[EMAIL PROTECTED]> wrote: >One significant benefit of ReiserFS is that it is journalled and does not >require a lengthy fsck operation after a power failure. However, if the journal gets corrupted, you are in for serious trouble, because Hans Reiser didn't plan this to happen and didn't write any fsck code. I have seen this happening, with severe data loss. AFAIK, there is now a "third party ReiserFS fsck", but I never actually tried it. Additionally, ReiserFS has some issues that shouldn't be present in a modern FS (for example, it is not 64 bit clean, as I am told). >I would not recommend that you change to ReiserFS at this time. ACK. ACK. ACK: Greetings Marc -- -- !! No courtesy copies, please !! - Marc Haber | " Questions are the | Mailadresse im Header Karlsruhe, Germany | Beginning of Wisdom " | Fon: *49 721 966 32 15 Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fax: *49 721 966 31 29 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
On Monday 11 June 2001 10:51, Jason Lim wrote: > Too bad this is a production system or I would try it. I've never tried > reiserFS (neither has anyone else here) so we might test it along with > a 2.4 kernel later. I hear 2.4 has intergrated reiserFS support? 2.4 has integrated ReiserFS support. ReiserFS primarily boosts performance when you have large numbers of small files in a directory (as a heavily loaded outbound mail server does). However when all data is being sync'd all the time (as a mail queue is) then ReiserFS does not perform so well and may deliver less performance than Ext2. One significant benefit of ReiserFS is that it is journalled and does not require a lengthy fsck operation after a power failure. I would not recommend that you change to ReiserFS at this time. All my most important data is on ReiserFS and I am quite confidant in the combination of the reliability of ReiserFS (which is not 100% but is getting close) and the quality of my backups. However changing the file system type on a server is a serious operation, I suggest that you play with ReiserFS on some less important machines and then consider using it on servers once you've got a feel for it. > From: "Alson van der Meulen" <[EMAIL PROTECTED]> > > > not help (even hindered) by reiserFS. Don't flame me if I'm wrong > > > as I haven't done huge amounts of research into this, but this is > > > just what I've heard. > > > > I know at least for news servers, reiserfs can be quite faster, > > so I guess it might be faster for mail too... > > > > There's really one way to be sure: test it ;) -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page
Re: Finding the Bottleneck
On Monday 11 June 2001 10:51, Jason Lim wrote: > Too bad this is a production system or I would try it. I've never tried > reiserFS (neither has anyone else here) so we might test it along with > a 2.4 kernel later. I hear 2.4 has intergrated reiserFS support? 2.4 has integrated ReiserFS support. ReiserFS primarily boosts performance when you have large numbers of small files in a directory (as a heavily loaded outbound mail server does). However when all data is being sync'd all the time (as a mail queue is) then ReiserFS does not perform so well and may deliver less performance than Ext2. One significant benefit of ReiserFS is that it is journalled and does not require a lengthy fsck operation after a power failure. I would not recommend that you change to ReiserFS at this time. All my most important data is on ReiserFS and I am quite confidant in the combination of the reliability of ReiserFS (which is not 100% but is getting close) and the quality of my backups. However changing the file system type on a server is a serious operation, I suggest that you play with ReiserFS on some less important machines and then consider using it on servers once you've got a feel for it. > From: "Alson van der Meulen" <[EMAIL PROTECTED]> > > > not help (even hindered) by reiserFS. Don't flame me if I'm wrong > > > as I haven't done huge amounts of research into this, but this is > > > just what I've heard. > > > > I know at least for news servers, reiserfs can be quite faster, > > so I guess it might be faster for mail too... > > > > There's really one way to be sure: test it ;) -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck (nearly there!)
Thought I'd mention one more thing that would be pretty important to know. qmail is now sending up to 400-450 concurrent outgoing emails (not all the time obviously, but easily goes up to that). Previously the maximum it would go to would be around 50... max 100, especially when it had 20K odd messages not preprocessed yet. Sincerely, Jason - Original Message - From: "Jason Lim" <[EMAIL PROTECTED]> To: "Russell Coker" <[EMAIL PROTECTED]>; Sent: Monday, June 11, 2001 11:59 PM Subject: Re: Finding the Bottleneck (nearly there!) > Hi, > > Something VERY interested has occurred. > > I kept playing around with the /var/qmail/queue directory, to see how I > could optimize it. > > I also saw in some qmail-* manpage that mess & pid directories, and todo & > intd directories have to be on the same drive (or was that partition? > nevermind) > > So since mess has the content of the emails on it, then in theory would be > the most "hard" on the disk (larger files than any other directories), I > left mess and pid on disk 2, and kept todo and intd onto disk 1. > > sh-2.05# ls -al > total 36 > drwxr-x---9 qmailq qmail4096 Jun 10 21:12 . > drwxr-xr-x7 root root 4096 Jun 10 21:11 .. > drwx--2 qmails qmail4096 Jun 11 23:46 bounce > drwx-- 25 qmails qmail4096 Jun 10 21:11 info > drwx-- 25 qmailq qmail4096 Jun 10 21:11 intd > drwx-- 25 qmails qmail4096 Jun 10 21:11 local > drwxr-x---2 qmailq qmail4096 Jun 10 21:11 lock > lrwxrwxrwx1 qmailq qmail 15 Jun 10 21:12 mess -> > /mnt/disk2/mess > lrwxrwxrwx1 qmailq qmail 14 Jun 10 21:12 pid -> > /mnt/disk2/pid > drwx-- 25 qmails qmail4096 Jun 10 21:11 remote > drwxr-x--- 25 qmailq qmail4096 Jun 10 21:11 todo > > > Surprise suprise... HUGE PERFORMANCE INCREASE!!! > > I mean double or even triple the thoughtput! > > sh-2.05# qmail-qstat > messages in queue: 28617 > messages in queue but not yet preprocessed: 0 > > NO unprocessed messages (compared to having 20-50K) and only 28K messages > in queue (compared to 500K). > > The mail volume has not changed since before, so besides playing with > hdparm a bit previously, nothing else has been changed. > > I have NO idea why putting the entire queue on disk 2, compared to just > putting mess and pid on disk 2, would have SUCH a huge difference. It > baffles me. > > Anyway... I have only observed this huge performance increase for a day, > so I will monitor this for another day and see if it keeps this up. I'll > post the findings tomorrow. > > Who could've guessed? It makes SOME sense... but double to triple the > performance? > > Sincerely, > Jason > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > >
Re: Finding the Bottleneck (nearly there!)
Hi, Something VERY interested has occurred. I kept playing around with the /var/qmail/queue directory, to see how I could optimize it. I also saw in some qmail-* manpage that mess & pid directories, and todo & intd directories have to be on the same drive (or was that partition? nevermind) So since mess has the content of the emails on it, then in theory would be the most "hard" on the disk (larger files than any other directories), I left mess and pid on disk 2, and kept todo and intd onto disk 1. sh-2.05# ls -al total 36 drwxr-x---9 qmailq qmail4096 Jun 10 21:12 . drwxr-xr-x7 root root 4096 Jun 10 21:11 .. drwx--2 qmails qmail4096 Jun 11 23:46 bounce drwx-- 25 qmails qmail4096 Jun 10 21:11 info drwx-- 25 qmailq qmail4096 Jun 10 21:11 intd drwx-- 25 qmails qmail4096 Jun 10 21:11 local drwxr-x---2 qmailq qmail4096 Jun 10 21:11 lock lrwxrwxrwx1 qmailq qmail 15 Jun 10 21:12 mess -> /mnt/disk2/mess lrwxrwxrwx1 qmailq qmail 14 Jun 10 21:12 pid -> /mnt/disk2/pid drwx-- 25 qmails qmail4096 Jun 10 21:11 remote drwxr-x--- 25 qmailq qmail4096 Jun 10 21:11 todo Surprise suprise... HUGE PERFORMANCE INCREASE!!! I mean double or even triple the thoughtput! sh-2.05# qmail-qstat messages in queue: 28617 messages in queue but not yet preprocessed: 0 NO unprocessed messages (compared to having 20-50K) and only 28K messages in queue (compared to 500K). The mail volume has not changed since before, so besides playing with hdparm a bit previously, nothing else has been changed. I have NO idea why putting the entire queue on disk 2, compared to just putting mess and pid on disk 2, would have SUCH a huge difference. It baffles me. Anyway... I have only observed this huge performance increase for a day, so I will monitor this for another day and see if it keeps this up. I'll post the findings tomorrow. Who could've guessed? It makes SOME sense... but double to triple the performance? Sincerely, Jason
Re: Finding the Bottleneck
Hi, Yeah.. they still make 15Gb drives. I think they still make 10Gbs but I'm not sure. Concerning the drives, since most of these are dedicated servers, they main way we can differenciate (sp.? plz correct me if wrong) is to provide different amounts of ram, cpu mhz, disk space, bandwidth, etc. Actually 15Gb is the smallest drive we offer. Most are in the range of 30-40Gbs. We will probably be increasing these numbers soon to reflect the market prices. BTW we also thought about buying 30Gb drives and partitioning them so that only 15Gb is usable, but with the advent of partition resizing proggies (and they are pretty stable now too), we can't do that. Sincerely, Jason - Original Message - From: "Russell Coker" <[EMAIL PROTECTED]> To: "Jason Lim" <[EMAIL PROTECTED]> Sent: Monday, June 11, 2001 7:03 PM Subject: Re: Finding the Bottleneck On Monday 11 June 2001 10:52, you wrote: > Hi, > > These 15G drives are new, or they wouldn't be ata100 ;-) I didn't know that they still made such small drives. The major stores in Amsterdam haven't sold such drives for over a year. > Only reason we don't get 45 or 50G drives all the time is that not all > customers need that amount of space. They pay for what they get. When you say "they pay for what they get" are you saying that they are trying to save money by buying tiny drives, or is this some sort of encouragement for the customers to pay you more management fees? If the former then you should compare the prices and see how little the difference is. If the latter then try and sell them on the backup/management angle... > > Sincerely, > Jason > > - Original Message - > From: "Russell Coker" <[EMAIL PROTECTED]> > To: "Rich Puhek" <[EMAIL PROTECTED]> > Cc: "Jason Lim" <[EMAIL PROTECTED]>; > Sent: Sunday, June 10, 2001 1:04 AM > Subject: Re: Finding the Bottleneck > > On Saturday 09 June 2001 01:11, Rich Puhek wrote: > > Memory memory memory! True, memory is not currently a limiting > > factor, but it likely could be if he were running BIND locally. As > > for making sure that the server is not authoratative for other > > domains, that will help keep other DNS demands to a minimum. > > From memory (sic) a caching name server for an ISP with 500,000 > customers that has typically >10,000 customers online at busy times > will grow to about 200M of RAM. Extrapolating from that I expect that > 20M of RAM should be adequate for a caching name server for the type of > load we are discussing. > > If the machine is upgraded to a decent amount of RAM (128M is nothing > by today's standards and upgrading RAM is the cheapest upgrade > possible) then the amount of RAM for a caching name server should not > be an issue. > > > Other than that, yea, some kind of RAID solution would be cool for > > him. I'd also look at making sure /var/log is on a seperate drive > > from /var/spool/mail. I saw an email that indicated that /swap was > > seperate from /var/spool, but nothing about where the log files were > > located. Not synching after evey write will help obviously, but I > > recall seeing quite a benefit from seperate drive for /var/log and > > /var/spool. > > My understanding of the discussion was that there was one drive for > /var/spool (which is for the queue and /var/spool/mail) and another > drive for everything else. > > That should be fine, but getting some drives that are less than 3 years > old would be a good idea... > > -- > http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark > http://www.coker.com.au/postal/ Postal SMTP/POP benchmark > http://www.coker.com.au/projects.html Projects I am working on > http://www.coker.com.au/~russell/ My home page > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact > [EMAIL PROTECTED] -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page
Re: Finding the Bottleneck
Hi, I in no way pretend to know a lot about the kernel and the specific ways it handles free memory and caches, but i just look at it from a "logical" point of view. Hopefully I'm not too far off course in the assumptions i make! Hope Rik can clarify this not just for me but for everyone thats been following this thread. Since he is the expert on this, he's the authority! Sincerely, Jason - Original Message - From: "Russell Coker" <[EMAIL PROTECTED]> To: "Jason Lim" <[EMAIL PROTECTED]> Cc: ; <[EMAIL PROTECTED]> Sent: Monday, June 11, 2001 6:43 PM Subject: Re: Finding the Bottleneck On Saturday 09 June 2001 20:04, Jason Lim wrote: > I'm not exactly sure how the Linux kernel would handle this. > > Right now, the swap is untouched. If the server needed more ram, > wouldn't it be swapping something... anything? I mean, it currently has > 0kb in swap, and still has free memory. That is a really good point. What you say makes a lot of sense. However the Linux kernel policies on when to free cache and when to swap are always being tweaked and are very complex. I have CC'd this message to Rik. Rik wrote most of the code in question and is the expert in this area. Rik, as a general rule if a machine has 0 swap in use then can it be assumed that the gain from adding more RAM will be minimal or non-existant? Or is my previous assumption correct in that it could still be able to productively use more RAM for cache? As a more specific issue, assuming that there is memory which is not being touched (there always is some) will that memory ever be paged out to allow for more caching? I believe that Jason is using kernel 2.2.19, but I would like to have this issue clarified for both 2.2.19 and 2.4.5 kernels if possible. > Anyway... as for the raid solution, is there anything I should look out > for BEFORE i start implementing it? Like any particular disk or ext2 > settings that would benefit the mail queue in any way? Don't want to > get everything set up, only to find I missed something critical that > you already thought of! There is an option to mke2fs to tune it for RAID-0 or RAID-5. I'm not sure if it provides much benefit though, and it does not help RAID-1 (which is the RAID level you are most likely to use). I suggest that you firstly run zcav from my bonnie++ suite on your new hard drives. Then allocate partitions for the most speed critical data in the fastest parts of the drives. Then use RAID-1 on those partitions. Good luck! -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page
Re: Finding the Bottleneck
Hi, More buffers makes sense... but i wonder what KIND of buffers those are. Only if they are disk buffers would the performance be increased. Sincerely, Jason - Original Message - From: "Marcin Owsiany" <[EMAIL PROTECTED]> To: Sent: Monday, June 11, 2001 5:37 PM Subject: Re: Finding the Bottleneck > On Mon, Jun 11, 2001 at 04:49:21PM +0800, Jason Lim wrote: > > Hi, > > > > AFAIK, even if there was a gig of ram in there, it would not allocate any > > (or maybe just a little) to free memory, and would throw any free memory > > into buffers anyway. > > > > So 68M of buffers tells me it has ample free memory, it or wouldn't > > allocate so much there anyway, right? > > Right, it probably would not allocate any more memory for the > processes themselves, but my point is that "the bigger buffers, > the better performance". I guess that 68 MB buffers isn't that > much for such a heavily loaded machine. > > Marcin > > PS: No need to CC to me. > -- > Marcin Owsiany <[EMAIL PROTECTED]> > http://student.uci.agh.edu.pl/~porridge/ > GnuPG: 1024D/60F41216 FE67 DA2D 0ACA FC5E 3F75 D6F6 3A0D 8AA0 60F4 1216 > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > >
Re: Finding the Bottleneck
On Mon, 11 Jun 2001, Russell Coker wrote: > Rik, as a general rule if a machine has 0 swap in use then can it be > assumed that the gain from adding more RAM will be minimal or > non-existant? Or is my previous assumption correct in that it could > still be able to productively use more RAM for cache? No. You cannot make any assumptions on that little data. ;) It would be useful to see some output from top and vmstat and get some information on what the machine is doing, how much memory it has, etc... > As a more specific issue, assuming that there is memory which is not > being touched (there always is some) will that memory ever be paged > out to allow for more caching? The faster you evict cache pages from memory, the less time programs have to touch it and the more will seem to be untouched. This is fairly unrelated to system load. regards, Rik -- Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose... http://www.surriel.com/ http://distro.conectiva.com/ Send all your spam to [EMAIL PROTECTED] (spam digging piggy)
Re: Finding the Bottleneck (nearly there!)
Thought I'd mention one more thing that would be pretty important to know. qmail is now sending up to 400-450 concurrent outgoing emails (not all the time obviously, but easily goes up to that). Previously the maximum it would go to would be around 50... max 100, especially when it had 20K odd messages not preprocessed yet. Sincerely, Jason - Original Message - From: "Jason Lim" <[EMAIL PROTECTED]> To: "Russell Coker" <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]> Sent: Monday, June 11, 2001 11:59 PM Subject: Re: Finding the Bottleneck (nearly there!) > Hi, > > Something VERY interested has occurred. > > I kept playing around with the /var/qmail/queue directory, to see how I > could optimize it. > > I also saw in some qmail-* manpage that mess & pid directories, and todo & > intd directories have to be on the same drive (or was that partition? > nevermind) > > So since mess has the content of the emails on it, then in theory would be > the most "hard" on the disk (larger files than any other directories), I > left mess and pid on disk 2, and kept todo and intd onto disk 1. > > sh-2.05# ls -al > total 36 > drwxr-x---9 qmailq qmail4096 Jun 10 21:12 . > drwxr-xr-x7 root root 4096 Jun 10 21:11 .. > drwx--2 qmails qmail4096 Jun 11 23:46 bounce > drwx-- 25 qmails qmail4096 Jun 10 21:11 info > drwx-- 25 qmailq qmail4096 Jun 10 21:11 intd > drwx-- 25 qmails qmail4096 Jun 10 21:11 local > drwxr-x---2 qmailq qmail4096 Jun 10 21:11 lock > lrwxrwxrwx1 qmailq qmail 15 Jun 10 21:12 mess -> > /mnt/disk2/mess > lrwxrwxrwx1 qmailq qmail 14 Jun 10 21:12 pid -> > /mnt/disk2/pid > drwx-- 25 qmails qmail4096 Jun 10 21:11 remote > drwxr-x--- 25 qmailq qmail4096 Jun 10 21:11 todo > > > Surprise suprise... HUGE PERFORMANCE INCREASE!!! > > I mean double or even triple the thoughtput! > > sh-2.05# qmail-qstat > messages in queue: 28617 > messages in queue but not yet preprocessed: 0 > > NO unprocessed messages (compared to having 20-50K) and only 28K messages > in queue (compared to 500K). > > The mail volume has not changed since before, so besides playing with > hdparm a bit previously, nothing else has been changed. > > I have NO idea why putting the entire queue on disk 2, compared to just > putting mess and pid on disk 2, would have SUCH a huge difference. It > baffles me. > > Anyway... I have only observed this huge performance increase for a day, > so I will monitor this for another day and see if it keeps this up. I'll > post the findings tomorrow. > > Who could've guessed? It makes SOME sense... but double to triple the > performance? > > Sincerely, > Jason > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > > -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck (nearly there!)
Hi, Something VERY interested has occurred. I kept playing around with the /var/qmail/queue directory, to see how I could optimize it. I also saw in some qmail-* manpage that mess & pid directories, and todo & intd directories have to be on the same drive (or was that partition? nevermind) So since mess has the content of the emails on it, then in theory would be the most "hard" on the disk (larger files than any other directories), I left mess and pid on disk 2, and kept todo and intd onto disk 1. sh-2.05# ls -al total 36 drwxr-x---9 qmailq qmail4096 Jun 10 21:12 . drwxr-xr-x7 root root 4096 Jun 10 21:11 .. drwx--2 qmails qmail4096 Jun 11 23:46 bounce drwx-- 25 qmails qmail4096 Jun 10 21:11 info drwx-- 25 qmailq qmail4096 Jun 10 21:11 intd drwx-- 25 qmails qmail4096 Jun 10 21:11 local drwxr-x---2 qmailq qmail4096 Jun 10 21:11 lock lrwxrwxrwx1 qmailq qmail 15 Jun 10 21:12 mess -> /mnt/disk2/mess lrwxrwxrwx1 qmailq qmail 14 Jun 10 21:12 pid -> /mnt/disk2/pid drwx-- 25 qmails qmail4096 Jun 10 21:11 remote drwxr-x--- 25 qmailq qmail4096 Jun 10 21:11 todo Surprise suprise... HUGE PERFORMANCE INCREASE!!! I mean double or even triple the thoughtput! sh-2.05# qmail-qstat messages in queue: 28617 messages in queue but not yet preprocessed: 0 NO unprocessed messages (compared to having 20-50K) and only 28K messages in queue (compared to 500K). The mail volume has not changed since before, so besides playing with hdparm a bit previously, nothing else has been changed. I have NO idea why putting the entire queue on disk 2, compared to just putting mess and pid on disk 2, would have SUCH a huge difference. It baffles me. Anyway... I have only observed this huge performance increase for a day, so I will monitor this for another day and see if it keeps this up. I'll post the findings tomorrow. Who could've guessed? It makes SOME sense... but double to triple the performance? Sincerely, Jason -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
Hi, Yeah.. they still make 15Gb drives. I think they still make 10Gbs but I'm not sure. Concerning the drives, since most of these are dedicated servers, they main way we can differenciate (sp.? plz correct me if wrong) is to provide different amounts of ram, cpu mhz, disk space, bandwidth, etc. Actually 15Gb is the smallest drive we offer. Most are in the range of 30-40Gbs. We will probably be increasing these numbers soon to reflect the market prices. BTW we also thought about buying 30Gb drives and partitioning them so that only 15Gb is usable, but with the advent of partition resizing proggies (and they are pretty stable now too), we can't do that. Sincerely, Jason - Original Message - From: "Russell Coker" <[EMAIL PROTECTED]> To: "Jason Lim" <[EMAIL PROTECTED]> Sent: Monday, June 11, 2001 7:03 PM Subject: Re: Finding the Bottleneck On Monday 11 June 2001 10:52, you wrote: > Hi, > > These 15G drives are new, or they wouldn't be ata100 ;-) I didn't know that they still made such small drives. The major stores in Amsterdam haven't sold such drives for over a year. > Only reason we don't get 45 or 50G drives all the time is that not all > customers need that amount of space. They pay for what they get. When you say "they pay for what they get" are you saying that they are trying to save money by buying tiny drives, or is this some sort of encouragement for the customers to pay you more management fees? If the former then you should compare the prices and see how little the difference is. If the latter then try and sell them on the backup/management angle... > > Sincerely, > Jason > > - Original Message - > From: "Russell Coker" <[EMAIL PROTECTED]> > To: "Rich Puhek" <[EMAIL PROTECTED]> > Cc: "Jason Lim" <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]> > Sent: Sunday, June 10, 2001 1:04 AM > Subject: Re: Finding the Bottleneck > > On Saturday 09 June 2001 01:11, Rich Puhek wrote: > > Memory memory memory! True, memory is not currently a limiting > > factor, but it likely could be if he were running BIND locally. As > > for making sure that the server is not authoratative for other > > domains, that will help keep other DNS demands to a minimum. > > From memory (sic) a caching name server for an ISP with 500,000 > customers that has typically >10,000 customers online at busy times > will grow to about 200M of RAM. Extrapolating from that I expect that > 20M of RAM should be adequate for a caching name server for the type of > load we are discussing. > > If the machine is upgraded to a decent amount of RAM (128M is nothing > by today's standards and upgrading RAM is the cheapest upgrade > possible) then the amount of RAM for a caching name server should not > be an issue. > > > Other than that, yea, some kind of RAID solution would be cool for > > him. I'd also look at making sure /var/log is on a seperate drive > > from /var/spool/mail. I saw an email that indicated that /swap was > > seperate from /var/spool, but nothing about where the log files were > > located. Not synching after evey write will help obviously, but I > > recall seeing quite a benefit from seperate drive for /var/log and > > /var/spool. > > My understanding of the discussion was that there was one drive for > /var/spool (which is for the queue and /var/spool/mail) and another > drive for everything else. > > That should be fine, but getting some drives that are less than 3 years > old would be a good idea... > > -- > http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark > http://www.coker.com.au/postal/ Postal SMTP/POP benchmark > http://www.coker.com.au/projects.html Projects I am working on > http://www.coker.com.au/~russell/ My home page > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact > [EMAIL PROTECTED] -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
Hi, I in no way pretend to know a lot about the kernel and the specific ways it handles free memory and caches, but i just look at it from a "logical" point of view. Hopefully I'm not too far off course in the assumptions i make! Hope Rik can clarify this not just for me but for everyone thats been following this thread. Since he is the expert on this, he's the authority! Sincerely, Jason - Original Message - From: "Russell Coker" <[EMAIL PROTECTED]> To: "Jason Lim" <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]> Sent: Monday, June 11, 2001 6:43 PM Subject: Re: Finding the Bottleneck On Saturday 09 June 2001 20:04, Jason Lim wrote: > I'm not exactly sure how the Linux kernel would handle this. > > Right now, the swap is untouched. If the server needed more ram, > wouldn't it be swapping something... anything? I mean, it currently has > 0kb in swap, and still has free memory. That is a really good point. What you say makes a lot of sense. However the Linux kernel policies on when to free cache and when to swap are always being tweaked and are very complex. I have CC'd this message to Rik. Rik wrote most of the code in question and is the expert in this area. Rik, as a general rule if a machine has 0 swap in use then can it be assumed that the gain from adding more RAM will be minimal or non-existant? Or is my previous assumption correct in that it could still be able to productively use more RAM for cache? As a more specific issue, assuming that there is memory which is not being touched (there always is some) will that memory ever be paged out to allow for more caching? I believe that Jason is using kernel 2.2.19, but I would like to have this issue clarified for both 2.2.19 and 2.4.5 kernels if possible. > Anyway... as for the raid solution, is there anything I should look out > for BEFORE i start implementing it? Like any particular disk or ext2 > settings that would benefit the mail queue in any way? Don't want to > get everything set up, only to find I missed something critical that > you already thought of! There is an option to mke2fs to tune it for RAID-0 or RAID-5. I'm not sure if it provides much benefit though, and it does not help RAID-1 (which is the RAID level you are most likely to use). I suggest that you firstly run zcav from my bonnie++ suite on your new hard drives. Then allocate partitions for the most speed critical data in the fastest parts of the drives. Then use RAID-1 on those partitions. Good luck! -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
Hi, More buffers makes sense... but i wonder what KIND of buffers those are. Only if they are disk buffers would the performance be increased. Sincerely, Jason - Original Message - From: "Marcin Owsiany" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Monday, June 11, 2001 5:37 PM Subject: Re: Finding the Bottleneck > On Mon, Jun 11, 2001 at 04:49:21PM +0800, Jason Lim wrote: > > Hi, > > > > AFAIK, even if there was a gig of ram in there, it would not allocate any > > (or maybe just a little) to free memory, and would throw any free memory > > into buffers anyway. > > > > So 68M of buffers tells me it has ample free memory, it or wouldn't > > allocate so much there anyway, right? > > Right, it probably would not allocate any more memory for the > processes themselves, but my point is that "the bigger buffers, > the better performance". I guess that 68 MB buffers isn't that > much for such a heavily loaded machine. > > Marcin > > PS: No need to CC to me. > -- > Marcin Owsiany <[EMAIL PROTECTED]> > http://student.uci.agh.edu.pl/~porridge/ > GnuPG: 1024D/60F41216 FE67 DA2D 0ACA FC5E 3F75 D6F6 3A0D 8AA0 60F4 1216 > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > > -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
On Mon, 11 Jun 2001, Russell Coker wrote: > Rik, as a general rule if a machine has 0 swap in use then can it be > assumed that the gain from adding more RAM will be minimal or > non-existant? Or is my previous assumption correct in that it could > still be able to productively use more RAM for cache? No. You cannot make any assumptions on that little data. ;) It would be useful to see some output from top and vmstat and get some information on what the machine is doing, how much memory it has, etc... > As a more specific issue, assuming that there is memory which is not > being touched (there always is some) will that memory ever be paged > out to allow for more caching? The faster you evict cache pages from memory, the less time programs have to touch it and the more will seem to be untouched. This is fairly unrelated to system load. regards, Rik -- Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose... http://www.surriel.com/ http://distro.conectiva.com/ Send all your spam to [EMAIL PROTECTED] (spam digging piggy) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
On Saturday 09 June 2001 20:04, Jason Lim wrote: > I'm not exactly sure how the Linux kernel would handle this. > > Right now, the swap is untouched. If the server needed more ram, > wouldn't it be swapping something... anything? I mean, it currently has > 0kb in swap, and still has free memory. That is a really good point. What you say makes a lot of sense. However the Linux kernel policies on when to free cache and when to swap are always being tweaked and are very complex. I have CC'd this message to Rik. Rik wrote most of the code in question and is the expert in this area. Rik, as a general rule if a machine has 0 swap in use then can it be assumed that the gain from adding more RAM will be minimal or non-existant? Or is my previous assumption correct in that it could still be able to productively use more RAM for cache? As a more specific issue, assuming that there is memory which is not being touched (there always is some) will that memory ever be paged out to allow for more caching? I believe that Jason is using kernel 2.2.19, but I would like to have this issue clarified for both 2.2.19 and 2.4.5 kernels if possible. > Anyway... as for the raid solution, is there anything I should look out > for BEFORE i start implementing it? Like any particular disk or ext2 > settings that would benefit the mail queue in any way? Don't want to > get everything set up, only to find I missed something critical that > you already thought of! There is an option to mke2fs to tune it for RAID-0 or RAID-5. I'm not sure if it provides much benefit though, and it does not help RAID-1 (which is the RAID level you are most likely to use). I suggest that you firstly run zcav from my bonnie++ suite on your new hard drives. Then allocate partitions for the most speed critical data in the fastest parts of the drives. Then use RAID-1 on those partitions. Good luck! -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page
Re: Finding the Bottleneck
On Mon, Jun 11, 2001 at 04:49:21PM +0800, Jason Lim wrote: > Hi, > > AFAIK, even if there was a gig of ram in there, it would not allocate any > (or maybe just a little) to free memory, and would throw any free memory > into buffers anyway. > > So 68M of buffers tells me it has ample free memory, it or wouldn't > allocate so much there anyway, right? Right, it probably would not allocate any more memory for the processes themselves, but my point is that "the bigger buffers, the better performance". I guess that 68 MB buffers isn't that much for such a heavily loaded machine. Marcin PS: No need to CC to me. -- Marcin Owsiany <[EMAIL PROTECTED]> http://student.uci.agh.edu.pl/~porridge/ GnuPG: 1024D/60F41216 FE67 DA2D 0ACA FC5E 3F75 D6F6 3A0D 8AA0 60F4 1216
Re: Finding the Bottleneck
On Mon, Jun 11, 2001 at 04:51:03PM +0800, Jason Lim wrote: > Hi, > > Too bad this is a production system or I would try it. I've never tried > reiserFS (neither has anyone else here) so we might test it along with a > 2.4 kernel later. I hear 2.4 has intergrated reiserFS support? Yup, it runs quite stable for me on several servers, 2.4 has integrated reiserfs support, though the 2.2.x reiserfs patches are quite stable too... Cheers, Alson -- ,---. > Name: Alson van der Meulen < > Personal: [EMAIL PROTECTED] < > School: [EMAIL PROTECTED]< `---' The drive ate the tape but that's OK, I brought my screwdriver. -
Re: Finding the Bottleneck
Hi, These 15G drives are new, or they wouldn't be ata100 ;-) Only reason we don't get 45 or 50G drives all the time is that not all customers need that amount of space. They pay for what they get. Sincerely, Jason - Original Message - From: "Russell Coker" <[EMAIL PROTECTED]> To: "Rich Puhek" <[EMAIL PROTECTED]> Cc: "Jason Lim" <[EMAIL PROTECTED]>; Sent: Sunday, June 10, 2001 1:04 AM Subject: Re: Finding the Bottleneck On Saturday 09 June 2001 01:11, Rich Puhek wrote: > Memory memory memory! True, memory is not currently a limiting factor, > but it likely could be if he were running BIND locally. As for making > sure that the server is not authoratative for other domains, that will > help keep other DNS demands to a minimum. >From memory (sic) a caching name server for an ISP with 500,000 customers that has typically >10,000 customers online at busy times will grow to about 200M of RAM. Extrapolating from that I expect that 20M of RAM should be adequate for a caching name server for the type of load we are discussing. If the machine is upgraded to a decent amount of RAM (128M is nothing by today's standards and upgrading RAM is the cheapest upgrade possible) then the amount of RAM for a caching name server should not be an issue. > Other than that, yea, some kind of RAID solution would be cool for him. > I'd also look at making sure /var/log is on a seperate drive from > /var/spool/mail. I saw an email that indicated that /swap was seperate > from /var/spool, but nothing about where the log files were located. > Not synching after evey write will help obviously, but I recall seeing > quite a benefit from seperate drive for /var/log and /var/spool. My understanding of the discussion was that there was one drive for /var/spool (which is for the queue and /var/spool/mail) and another drive for everything else. That should be fine, but getting some drives that are less than 3 years old would be a good idea... -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
Hi, Too bad this is a production system or I would try it. I've never tried reiserFS (neither has anyone else here) so we might test it along with a 2.4 kernel later. I hear 2.4 has intergrated reiserFS support? Sincerely, Jason - Original Message - From: "Alson van der Meulen" <[EMAIL PROTECTED]> To: Sent: Sunday, June 10, 2001 4:25 AM Subject: Re: Finding the Bottleneck > On Sun, Jun 10, 2001 at 04:14:10AM +0800, Jason Lim wrote: > > Hi, > > > > Actually, I thought they increased performance mainly if you were doing > > large file transfers and such, and that small random file transfers were > > not help (even hindered) by reiserFS. Don't flame me if I'm wrong as I > > haven't done huge amounts of research into this, but this is just what > > I've heard. > I know at least for news servers, reiserfs can be quite faster, > so I guess it might be faster for mail too... > > There's really one way to be sure: test it ;) > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > >
Re: Finding the Bottleneck
Hi, AFAIK, even if there was a gig of ram in there, it would not allocate any (or maybe just a little) to free memory, and would throw any free memory into buffers anyway. So 68M of buffers tells me it has ample free memory, it or wouldn't allocate so much there anyway, right? Sincerely, Jason - Original Message - From: "Marcin Owsiany" <[EMAIL PROTECTED]> To: Sent: Monday, June 11, 2001 7:10 AM Subject: Re: Finding the Bottleneck > On Sun, Jun 10, 2001 at 02:04:36AM +0800, Jason Lim wrote: > > I'm not exactly sure how the Linux kernel would handle this. > > > > Right now, the swap is untouched. If the server needed more ram, wouldn't > > it be swapping something... anything? I mean, it currently has 0kb in > > swap, and still has free memory. > > > > Here is a recent top: > > > > 101 processes: 97 sleeping, 3 running, 1 zombie, 0 stopped > > CPU states: 9.4% user, 14.0% system, 0.5% nice, 76.1% idle > > Mem:128236K total, 125492K used, 2744K free,69528K buffers > > Swap: 289160K total,0K used, 289160K free,10320K cached > > Remember that adding RAM means larger buffers/cache, and so > faster IO. Only 3 MB free memory means that Linux would really > like more RAM for larger buffers. > > Marcin > -- > Marcin Owsiany <[EMAIL PROTECTED]> > http://student.uci.agh.edu.pl/~porridge/ > GnuPG: 1024D/60F41216 FE67 DA2D 0ACA FC5E 3F75 D6F6 3A0D 8AA0 60F4 1216 > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > >
Re: Finding the Bottleneck
On Saturday 09 June 2001 20:04, Jason Lim wrote: > I'm not exactly sure how the Linux kernel would handle this. > > Right now, the swap is untouched. If the server needed more ram, > wouldn't it be swapping something... anything? I mean, it currently has > 0kb in swap, and still has free memory. That is a really good point. What you say makes a lot of sense. However the Linux kernel policies on when to free cache and when to swap are always being tweaked and are very complex. I have CC'd this message to Rik. Rik wrote most of the code in question and is the expert in this area. Rik, as a general rule if a machine has 0 swap in use then can it be assumed that the gain from adding more RAM will be minimal or non-existant? Or is my previous assumption correct in that it could still be able to productively use more RAM for cache? As a more specific issue, assuming that there is memory which is not being touched (there always is some) will that memory ever be paged out to allow for more caching? I believe that Jason is using kernel 2.2.19, but I would like to have this issue clarified for both 2.2.19 and 2.4.5 kernels if possible. > Anyway... as for the raid solution, is there anything I should look out > for BEFORE i start implementing it? Like any particular disk or ext2 > settings that would benefit the mail queue in any way? Don't want to > get everything set up, only to find I missed something critical that > you already thought of! There is an option to mke2fs to tune it for RAID-0 or RAID-5. I'm not sure if it provides much benefit though, and it does not help RAID-1 (which is the RAID level you are most likely to use). I suggest that you firstly run zcav from my bonnie++ suite on your new hard drives. Then allocate partitions for the most speed critical data in the fastest parts of the drives. Then use RAID-1 on those partitions. Good luck! -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
On Mon, Jun 11, 2001 at 04:49:21PM +0800, Jason Lim wrote: > Hi, > > AFAIK, even if there was a gig of ram in there, it would not allocate any > (or maybe just a little) to free memory, and would throw any free memory > into buffers anyway. > > So 68M of buffers tells me it has ample free memory, it or wouldn't > allocate so much there anyway, right? Right, it probably would not allocate any more memory for the processes themselves, but my point is that "the bigger buffers, the better performance". I guess that 68 MB buffers isn't that much for such a heavily loaded machine. Marcin PS: No need to CC to me. -- Marcin Owsiany <[EMAIL PROTECTED]> http://student.uci.agh.edu.pl/~porridge/ GnuPG: 1024D/60F41216 FE67 DA2D 0ACA FC5E 3F75 D6F6 3A0D 8AA0 60F4 1216 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
On Mon, Jun 11, 2001 at 04:51:03PM +0800, Jason Lim wrote: > Hi, > > Too bad this is a production system or I would try it. I've never tried > reiserFS (neither has anyone else here) so we might test it along with a > 2.4 kernel later. I hear 2.4 has intergrated reiserFS support? Yup, it runs quite stable for me on several servers, 2.4 has integrated reiserfs support, though the 2.2.x reiserfs patches are quite stable too... Cheers, Alson -- ,---. > Name: Alson van der Meulen < > Personal: [EMAIL PROTECTED] < > School: [EMAIL PROTECTED]< `---' The drive ate the tape but that's OK, I brought my screwdriver. - -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
Hi, These 15G drives are new, or they wouldn't be ata100 ;-) Only reason we don't get 45 or 50G drives all the time is that not all customers need that amount of space. They pay for what they get. Sincerely, Jason - Original Message - From: "Russell Coker" <[EMAIL PROTECTED]> To: "Rich Puhek" <[EMAIL PROTECTED]> Cc: "Jason Lim" <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]> Sent: Sunday, June 10, 2001 1:04 AM Subject: Re: Finding the Bottleneck On Saturday 09 June 2001 01:11, Rich Puhek wrote: > Memory memory memory! True, memory is not currently a limiting factor, > but it likely could be if he were running BIND locally. As for making > sure that the server is not authoratative for other domains, that will > help keep other DNS demands to a minimum. >From memory (sic) a caching name server for an ISP with 500,000 customers that has typically >10,000 customers online at busy times will grow to about 200M of RAM. Extrapolating from that I expect that 20M of RAM should be adequate for a caching name server for the type of load we are discussing. If the machine is upgraded to a decent amount of RAM (128M is nothing by today's standards and upgrading RAM is the cheapest upgrade possible) then the amount of RAM for a caching name server should not be an issue. > Other than that, yea, some kind of RAID solution would be cool for him. > I'd also look at making sure /var/log is on a seperate drive from > /var/spool/mail. I saw an email that indicated that /swap was seperate > from /var/spool, but nothing about where the log files were located. > Not synching after evey write will help obviously, but I recall seeing > quite a benefit from seperate drive for /var/log and /var/spool. My understanding of the discussion was that there was one drive for /var/spool (which is for the queue and /var/spool/mail) and another drive for everything else. That should be fine, but getting some drives that are less than 3 years old would be a good idea... -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
Hi, Too bad this is a production system or I would try it. I've never tried reiserFS (neither has anyone else here) so we might test it along with a 2.4 kernel later. I hear 2.4 has intergrated reiserFS support? Sincerely, Jason - Original Message - From: "Alson van der Meulen" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Sunday, June 10, 2001 4:25 AM Subject: Re: Finding the Bottleneck > On Sun, Jun 10, 2001 at 04:14:10AM +0800, Jason Lim wrote: > > Hi, > > > > Actually, I thought they increased performance mainly if you were doing > > large file transfers and such, and that small random file transfers were > > not help (even hindered) by reiserFS. Don't flame me if I'm wrong as I > > haven't done huge amounts of research into this, but this is just what > > I've heard. > I know at least for news servers, reiserfs can be quite faster, > so I guess it might be faster for mail too... > > There's really one way to be sure: test it ;) > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > > -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
Hi, AFAIK, even if there was a gig of ram in there, it would not allocate any (or maybe just a little) to free memory, and would throw any free memory into buffers anyway. So 68M of buffers tells me it has ample free memory, it or wouldn't allocate so much there anyway, right? Sincerely, Jason - Original Message - From: "Marcin Owsiany" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Monday, June 11, 2001 7:10 AM Subject: Re: Finding the Bottleneck > On Sun, Jun 10, 2001 at 02:04:36AM +0800, Jason Lim wrote: > > I'm not exactly sure how the Linux kernel would handle this. > > > > Right now, the swap is untouched. If the server needed more ram, wouldn't > > it be swapping something... anything? I mean, it currently has 0kb in > > swap, and still has free memory. > > > > Here is a recent top: > > > > 101 processes: 97 sleeping, 3 running, 1 zombie, 0 stopped > > CPU states: 9.4% user, 14.0% system, 0.5% nice, 76.1% idle > > Mem:128236K total, 125492K used, 2744K free,69528K buffers > > Swap: 289160K total,0K used, 289160K free,10320K cached > > Remember that adding RAM means larger buffers/cache, and so > faster IO. Only 3 MB free memory means that Linux would really > like more RAM for larger buffers. > > Marcin > -- > Marcin Owsiany <[EMAIL PROTECTED]> > http://student.uci.agh.edu.pl/~porridge/ > GnuPG: 1024D/60F41216 FE67 DA2D 0ACA FC5E 3F75 D6F6 3A0D 8AA0 60F4 1216 > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > > -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
On Sun, Jun 10, 2001 at 02:04:36AM +0800, Jason Lim wrote: > I'm not exactly sure how the Linux kernel would handle this. > > Right now, the swap is untouched. If the server needed more ram, wouldn't > it be swapping something... anything? I mean, it currently has 0kb in > swap, and still has free memory. > > Here is a recent top: > > 101 processes: 97 sleeping, 3 running, 1 zombie, 0 stopped > CPU states: 9.4% user, 14.0% system, 0.5% nice, 76.1% idle > Mem:128236K total, 125492K used, 2744K free,69528K buffers > Swap: 289160K total,0K used, 289160K free,10320K cached Remember that adding RAM means larger buffers/cache, and so faster IO. Only 3 MB free memory means that Linux would really like more RAM for larger buffers. Marcin -- Marcin Owsiany <[EMAIL PROTECTED]> http://student.uci.agh.edu.pl/~porridge/ GnuPG: 1024D/60F41216 FE67 DA2D 0ACA FC5E 3F75 D6F6 3A0D 8AA0 60F4 1216
Re: Finding the Bottleneck
On Sun, Jun 10, 2001 at 02:04:36AM +0800, Jason Lim wrote: > I'm not exactly sure how the Linux kernel would handle this. > > Right now, the swap is untouched. If the server needed more ram, wouldn't > it be swapping something... anything? I mean, it currently has 0kb in > swap, and still has free memory. > > Here is a recent top: > > 101 processes: 97 sleeping, 3 running, 1 zombie, 0 stopped > CPU states: 9.4% user, 14.0% system, 0.5% nice, 76.1% idle > Mem:128236K total, 125492K used, 2744K free,69528K buffers > Swap: 289160K total,0K used, 289160K free,10320K cached Remember that adding RAM means larger buffers/cache, and so faster IO. Only 3 MB free memory means that Linux would really like more RAM for larger buffers. Marcin -- Marcin Owsiany <[EMAIL PROTECTED]> http://student.uci.agh.edu.pl/~porridge/ GnuPG: 1024D/60F41216 FE67 DA2D 0ACA FC5E 3F75 D6F6 3A0D 8AA0 60F4 1216 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
On Sun, Jun 10, 2001 at 04:14:10AM +0800, Jason Lim wrote: > Hi, > > Actually, I thought they increased performance mainly if you were doing > large file transfers and such, and that small random file transfers were > not help (even hindered) by reiserFS. Don't flame me if I'm wrong as I > haven't done huge amounts of research into this, but this is just what > I've heard. I know at least for news servers, reiserfs can be quite faster, so I guess it might be faster for mail too... There's really one way to be sure: test it ;)
Re: Finding the Bottleneck
Hi, Actually, I thought they increased performance mainly if you were doing large file transfers and such, and that small random file transfers were not help (even hindered) by reiserFS. Don't flame me if I'm wrong as I haven't done huge amounts of research into this, but this is just what I've heard. Sincerely, Jason - Original Message - From: "Alson van der Meulen" <[EMAIL PROTECTED]> To: Sent: Sunday, June 10, 2001 2:32 AM Subject: Re: Finding the Bottleneck > On Sun, Jun 10, 2001 at 02:04:36AM +0800, Jason Lim wrote: > > I'm not exactly sure how the Linux kernel would handle this. > > > [...] > > > > Anyway... as for the raid solution, is there anything I should look out > > for BEFORE i start implementing it? Like any particular disk or ext2 > > settings that would benefit the mail queue in any way? Don't want to get > > everything set up, only to find I missed something critical that you > > already thought of! > Maybe reiserfs (or xfs) might help? I know they increase > filesystem/metadata performance in some cases... > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > >
Re: Finding the Bottleneck
On Sun, Jun 10, 2001 at 04:14:10AM +0800, Jason Lim wrote: > Hi, > > Actually, I thought they increased performance mainly if you were doing > large file transfers and such, and that small random file transfers were > not help (even hindered) by reiserFS. Don't flame me if I'm wrong as I > haven't done huge amounts of research into this, but this is just what > I've heard. I know at least for news servers, reiserfs can be quite faster, so I guess it might be faster for mail too... There's really one way to be sure: test it ;) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
On Sun, Jun 10, 2001 at 02:04:36AM +0800, Jason Lim wrote: > I'm not exactly sure how the Linux kernel would handle this. > [...] > > Anyway... as for the raid solution, is there anything I should look out > for BEFORE i start implementing it? Like any particular disk or ext2 > settings that would benefit the mail queue in any way? Don't want to get > everything set up, only to find I missed something critical that you > already thought of! Maybe reiserfs (or xfs) might help? I know they increase filesystem/metadata performance in some cases...
Re: Finding the Bottleneck
I'm not exactly sure how the Linux kernel would handle this. Right now, the swap is untouched. If the server needed more ram, wouldn't it be swapping something... anything? I mean, it currently has 0kb in swap, and still has free memory. Here is a recent top: 101 processes: 97 sleeping, 3 running, 1 zombie, 0 stopped CPU states: 9.4% user, 14.0% system, 0.5% nice, 76.1% idle Mem:128236K total, 125492K used, 2744K free,69528K buffers Swap: 289160K total,0K used, 289160K free,10320K cached PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 5361 qmails 4 0 2728 2728 368 R 5.6 2.1 68:58 qmail-send 11911 root 4 0 1052 1052 800 R 1.7 0.8 0:00 top 165 root 1 0 2640 2640 860 S 0.9 2.0 25:00 named 5367 qmailr17 0 464 464 324 S 0.9 0.3 6:58 qmail-rspawn 1178 root 0 0 832 832 708 S 0.3 0.6 4:30 syslogd 5365 qmaill 0 0 476 476 404 S 0.1 0.3 6:12 splogger 5368 qmailq 1 0 396 396 332 S 0.1 0.3 5:20 qmail-clean 11988 qmailr 1 0 512 512 432 S 0.1 0.3 0:00 qmail-remote 11993 qmailr 4 0 512 512 432 R 0.1 0.3 0:00 qmail-remote 11994 qmailr 4 0 512 512 432 S 0.1 0.3 0:00 qmail-remote 11996 qmailr 5 0 512 512 432 R 0.1 0.3 0:00 qmail-remote 11997 qmailr 8 0 512 512 432 S 0.1 0.3 0:00 qmail-remote 11998 qmailr 9 0 512 512 432 R 0.1 0.3 0:00 qmail-remote 11999 qmailr10 0 512 512 432 R 0.1 0.3 0:00 qmail-remote 12000 qmailr10 0 512 512 432 S 0.1 0.3 0:00 qmail-remote 1 root 0 0 532 532 472 S 0.0 0.4 0:07 init 2 root 0 0 00 0 SW0.0 0.0 0:07 kflushd I hope you can read the above because it won't be formatted right when I send it, but hopefully you get the idea. As far as I know, linux will allocate as much free ram to the buffers, rather than just leave it empty. So ~68M in buffers sort of tells me that it has plenty of memory. I mean, if you think more would really help, we could try more ram, but I doubt the bottleneck really is with the memory limit...? Anyway... as for the raid solution, is there anything I should look out for BEFORE i start implementing it? Like any particular disk or ext2 settings that would benefit the mail queue in any way? Don't want to get everything set up, only to find I missed something critical that you already thought of! Sincerely, Jason - Original Message - From: "Russell Coker" <[EMAIL PROTECTED]> To: "Jason Lim" <[EMAIL PROTECTED]>; "Rich Puhek" <[EMAIL PROTECTED]> Cc: Sent: Sunday, June 10, 2001 1:07 AM Subject: Re: Finding the Bottleneck On Saturday 09 June 2001 08:23, Jason Lim wrote: > Well... I'm not sure if you saw the "top" output I sent to the list a > while back, but the swap isn't touched at all. The 128M ram seems to be > sufficient at this time. I'm not sure that throwing more memory at it > would help much, would it? I think even if more ram is put in, it will > just use at buffers. er that MIGHT help, right? Would be an > easy solution if 256M would help get an extra 20% performance :-) More cache is very likely to help, and it requires little expense and little work to add another 128M of RAM to the machine. I'm not sure that you'll get 20% more performance, I'd expect maybe 10% - but it depends on the load patterns. For a cheap and easy way to add performance adding RAM is the best thing you can do IMHO. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
Hi, Actually, I thought they increased performance mainly if you were doing large file transfers and such, and that small random file transfers were not help (even hindered) by reiserFS. Don't flame me if I'm wrong as I haven't done huge amounts of research into this, but this is just what I've heard. Sincerely, Jason - Original Message - From: "Alson van der Meulen" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Sunday, June 10, 2001 2:32 AM Subject: Re: Finding the Bottleneck > On Sun, Jun 10, 2001 at 02:04:36AM +0800, Jason Lim wrote: > > I'm not exactly sure how the Linux kernel would handle this. > > > [...] > > > > Anyway... as for the raid solution, is there anything I should look out > > for BEFORE i start implementing it? Like any particular disk or ext2 > > settings that would benefit the mail queue in any way? Don't want to get > > everything set up, only to find I missed something critical that you > > already thought of! > Maybe reiserfs (or xfs) might help? I know they increase > filesystem/metadata performance in some cases... > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > > -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
On Saturday 09 June 2001 08:23, Jason Lim wrote: > Well... I'm not sure if you saw the "top" output I sent to the list a > while back, but the swap isn't touched at all. The 128M ram seems to be > sufficient at this time. I'm not sure that throwing more memory at it > would help much, would it? I think even if more ram is put in, it will > just use at buffers. er that MIGHT help, right? Would be an > easy solution if 256M would help get an extra 20% performance :-) More cache is very likely to help, and it requires little expense and little work to add another 128M of RAM to the machine. I'm not sure that you'll get 20% more performance, I'd expect maybe 10% - but it depends on the load patterns. For a cheap and easy way to add performance adding RAM is the best thing you can do IMHO. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page
Re: Finding the Bottleneck
On Saturday 09 June 2001 01:11, Rich Puhek wrote: > Memory memory memory! True, memory is not currently a limiting factor, > but it likely could be if he were running BIND locally. As for making > sure that the server is not authoratative for other domains, that will > help keep other DNS demands to a minimum. From memory (sic) a caching name server for an ISP with 500,000 customers that has typically >10,000 customers online at busy times will grow to about 200M of RAM. Extrapolating from that I expect that 20M of RAM should be adequate for a caching name server for the type of load we are discussing. If the machine is upgraded to a decent amount of RAM (128M is nothing by today's standards and upgrading RAM is the cheapest upgrade possible) then the amount of RAM for a caching name server should not be an issue. > Other than that, yea, some kind of RAID solution would be cool for him. > I'd also look at making sure /var/log is on a seperate drive from > /var/spool/mail. I saw an email that indicated that /swap was seperate > from /var/spool, but nothing about where the log files were located. > Not synching after evey write will help obviously, but I recall seeing > quite a benefit from seperate drive for /var/log and /var/spool. My understanding of the discussion was that there was one drive for /var/spool (which is for the queue and /var/spool/mail) and another drive for everything else. That should be fine, but getting some drives that are less than 3 years old would be a good idea... -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page
Re: Finding the Bottleneck
On Sun, Jun 10, 2001 at 02:04:36AM +0800, Jason Lim wrote: > I'm not exactly sure how the Linux kernel would handle this. > [...] > > Anyway... as for the raid solution, is there anything I should look out > for BEFORE i start implementing it? Like any particular disk or ext2 > settings that would benefit the mail queue in any way? Don't want to get > everything set up, only to find I missed something critical that you > already thought of! Maybe reiserfs (or xfs) might help? I know they increase filesystem/metadata performance in some cases... -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
I'm not exactly sure how the Linux kernel would handle this. Right now, the swap is untouched. If the server needed more ram, wouldn't it be swapping something... anything? I mean, it currently has 0kb in swap, and still has free memory. Here is a recent top: 101 processes: 97 sleeping, 3 running, 1 zombie, 0 stopped CPU states: 9.4% user, 14.0% system, 0.5% nice, 76.1% idle Mem:128236K total, 125492K used, 2744K free,69528K buffers Swap: 289160K total,0K used, 289160K free,10320K cached PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 5361 qmails 4 0 2728 2728 368 R 5.6 2.1 68:58 qmail-send 11911 root 4 0 1052 1052 800 R 1.7 0.8 0:00 top 165 root 1 0 2640 2640 860 S 0.9 2.0 25:00 named 5367 qmailr17 0 464 464 324 S 0.9 0.3 6:58 qmail-rspawn 1178 root 0 0 832 832 708 S 0.3 0.6 4:30 syslogd 5365 qmaill 0 0 476 476 404 S 0.1 0.3 6:12 splogger 5368 qmailq 1 0 396 396 332 S 0.1 0.3 5:20 qmail-clean 11988 qmailr 1 0 512 512 432 S 0.1 0.3 0:00 qmail-remote 11993 qmailr 4 0 512 512 432 R 0.1 0.3 0:00 qmail-remote 11994 qmailr 4 0 512 512 432 S 0.1 0.3 0:00 qmail-remote 11996 qmailr 5 0 512 512 432 R 0.1 0.3 0:00 qmail-remote 11997 qmailr 8 0 512 512 432 S 0.1 0.3 0:00 qmail-remote 11998 qmailr 9 0 512 512 432 R 0.1 0.3 0:00 qmail-remote 11999 qmailr10 0 512 512 432 R 0.1 0.3 0:00 qmail-remote 12000 qmailr10 0 512 512 432 S 0.1 0.3 0:00 qmail-remote 1 root 0 0 532 532 472 S 0.0 0.4 0:07 init 2 root 0 0 00 0 SW0.0 0.0 0:07 kflushd I hope you can read the above because it won't be formatted right when I send it, but hopefully you get the idea. As far as I know, linux will allocate as much free ram to the buffers, rather than just leave it empty. So ~68M in buffers sort of tells me that it has plenty of memory. I mean, if you think more would really help, we could try more ram, but I doubt the bottleneck really is with the memory limit...? Anyway... as for the raid solution, is there anything I should look out for BEFORE i start implementing it? Like any particular disk or ext2 settings that would benefit the mail queue in any way? Don't want to get everything set up, only to find I missed something critical that you already thought of! Sincerely, Jason - Original Message - From: "Russell Coker" <[EMAIL PROTECTED]> To: "Jason Lim" <[EMAIL PROTECTED]>; "Rich Puhek" <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Sunday, June 10, 2001 1:07 AM Subject: Re: Finding the Bottleneck On Saturday 09 June 2001 08:23, Jason Lim wrote: > Well... I'm not sure if you saw the "top" output I sent to the list a > while back, but the swap isn't touched at all. The 128M ram seems to be > sufficient at this time. I'm not sure that throwing more memory at it > would help much, would it? I think even if more ram is put in, it will > just use at buffers. er that MIGHT help, right? Would be an > easy solution if 256M would help get an extra 20% performance :-) More cache is very likely to help, and it requires little expense and little work to add another 128M of RAM to the machine. I'm not sure that you'll get 20% more performance, I'd expect maybe 10% - but it depends on the load patterns. For a cheap and easy way to add performance adding RAM is the best thing you can do IMHO. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
On Saturday 09 June 2001 08:23, Jason Lim wrote: > Well... I'm not sure if you saw the "top" output I sent to the list a > while back, but the swap isn't touched at all. The 128M ram seems to be > sufficient at this time. I'm not sure that throwing more memory at it > would help much, would it? I think even if more ram is put in, it will > just use at buffers. er that MIGHT help, right? Would be an > easy solution if 256M would help get an extra 20% performance :-) More cache is very likely to help, and it requires little expense and little work to add another 128M of RAM to the machine. I'm not sure that you'll get 20% more performance, I'd expect maybe 10% - but it depends on the load patterns. For a cheap and easy way to add performance adding RAM is the best thing you can do IMHO. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
On Saturday 09 June 2001 01:11, Rich Puhek wrote: > Memory memory memory! True, memory is not currently a limiting factor, > but it likely could be if he were running BIND locally. As for making > sure that the server is not authoratative for other domains, that will > help keep other DNS demands to a minimum. From memory (sic) a caching name server for an ISP with 500,000 customers that has typically >10,000 customers online at busy times will grow to about 200M of RAM. Extrapolating from that I expect that 20M of RAM should be adequate for a caching name server for the type of load we are discussing. If the machine is upgraded to a decent amount of RAM (128M is nothing by today's standards and upgrading RAM is the cheapest upgrade possible) then the amount of RAM for a caching name server should not be an issue. > Other than that, yea, some kind of RAID solution would be cool for him. > I'd also look at making sure /var/log is on a seperate drive from > /var/spool/mail. I saw an email that indicated that /swap was seperate > from /var/spool, but nothing about where the log files were located. > Not synching after evey write will help obviously, but I recall seeing > quite a benefit from seperate drive for /var/log and /var/spool. My understanding of the discussion was that there was one drive for /var/spool (which is for the queue and /var/spool/mail) and another drive for everything else. That should be fine, but getting some drives that are less than 3 years old would be a good idea... -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
Hi, Well... I'm not sure if you saw the "top" output I sent to the list a while back, but the swap isn't touched at all. The 128M ram seems to be sufficient at this time. I'm not sure that throwing more memory at it would help much, would it? I think even if more ram is put in, it will just use at buffers. er that MIGHT help, right? Would be an easy solution if 256M would help get an extra 20% performance :-) Concerning the dns server, if it was not run locally, then every single dns lookup would have to go across the network. Now that would be a LOT of requests that would otherwise have been answered nearly instantly if it was local. The mail queue is on its OWN disk (disk 2). Everything else is on disk 1. The mail queue disk ONLY has the mail queue and absolutely nothing else. It is there for that specific purpose (even though it hasn't helped that much). Also, log files in syslog already have "-" in front of the file name so they aren't synced every write. We've tried just about everything (except the raid setup) to get every last ounce of performance out of this... but it doesn't seem like its made a huge difference :-/ Sincerely, Jason - Original Message - From: "Rich Puhek" <[EMAIL PROTECTED]> To: "Russell Coker" <[EMAIL PROTECTED]> Cc: "Jason Lim" <[EMAIL PROTECTED]>; Sent: Saturday, June 09, 2001 7:11 AM Subject: Re: Finding the Bottleneck > Memory memory memory! True, memory is not currently a limiting factor, > but it likely could be if he were running BIND locally. As for making > sure that the server is not authoratative for other domains, that will > help keep other DNS demands to a minimum. > > The mail server will chew up a load of memory (or can anyhow.. his > doesn't seem too bad). A highly-utilized DNS server will also chew up a > load of memory. You do not want the DNS server to swap, so you need to > have enough memory to be sure it can cache enough information. > > As for speed, if you have a machine on a LAN set up as a caching-only > DNS server (that's what I was trying to say before), I'm thinking I'll > take the LAN latency hit over having the MTA competing for resources > with the DNS server. > > Other than that, yea, some kind of RAID solution would be cool for him. > I'd also look at making sure /var/log is on a seperate drive from > /var/spool/mail. I saw an email that indicated that /swap was seperate > from /var/spool, but nothing about where the log files were located. Not > synching after evey write will help obviously, but I recall seeing quite > a benefit from seperate drive for /var/log and /var/spool. > > --Rich > > > Russell Coker wrote: > > > > On Friday 08 June 2001 05:47, Rich Puhek wrote: > > > In addition to checking the disk usage, memory, and the other > > > suggestions that have come up on the list, have you looked at DNS? > > > Quite often you'll find that DNS lookups are severely limiting the > > > performance of something like a mailing list. Make sure that the mail > > > server itself isn't running a DNS server. Make sure you've got one or > > > > Why not? When DNS speed is important I ALWAYS install a local DNS. > > Requests to 127.0.0.1 have to be faster than any other requests... > > > > > two DNS servers in close proximity to the mail server. Make sure that > > > the DNS server process isn't swapping on the DNS servers (for the kind > > > > The output of "top" that he recently posted suggests that nothing is > > swapping. > > > > > with 128 MB of RAM as your DNS server. Also, if possible, I like to > > > have the DNS server I'm querying kept free from being the authoratative > > > server for any domains (not always practical in a real life situation, > > > I know). > > > > How does that help? > > > > If DNS caching is the issue then probably the only place to look for a > > solution is djb-dnscache. > > > > -- > > http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark > > http://www.coker.com.au/postal/ Postal SMTP/POP benchmark > > http://www.coker.com.au/projects.html Projects I am working on > > http://www.coker.com.au/~russell/ My home page > > > > -- > > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > > -- > > _ > > Rich Puhek > ETN Systems Inc. > _ > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > >
Re: Finding the Bottleneck
Hi, Well... I'm not sure if you saw the "top" output I sent to the list a while back, but the swap isn't touched at all. The 128M ram seems to be sufficient at this time. I'm not sure that throwing more memory at it would help much, would it? I think even if more ram is put in, it will just use at buffers. er that MIGHT help, right? Would be an easy solution if 256M would help get an extra 20% performance :-) Concerning the dns server, if it was not run locally, then every single dns lookup would have to go across the network. Now that would be a LOT of requests that would otherwise have been answered nearly instantly if it was local. The mail queue is on its OWN disk (disk 2). Everything else is on disk 1. The mail queue disk ONLY has the mail queue and absolutely nothing else. It is there for that specific purpose (even though it hasn't helped that much). Also, log files in syslog already have "-" in front of the file name so they aren't synced every write. We've tried just about everything (except the raid setup) to get every last ounce of performance out of this... but it doesn't seem like its made a huge difference :-/ Sincerely, Jason - Original Message - From: "Rich Puhek" <[EMAIL PROTECTED]> To: "Russell Coker" <[EMAIL PROTECTED]> Cc: "Jason Lim" <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]> Sent: Saturday, June 09, 2001 7:11 AM Subject: Re: Finding the Bottleneck > Memory memory memory! True, memory is not currently a limiting factor, > but it likely could be if he were running BIND locally. As for making > sure that the server is not authoratative for other domains, that will > help keep other DNS demands to a minimum. > > The mail server will chew up a load of memory (or can anyhow.. his > doesn't seem too bad). A highly-utilized DNS server will also chew up a > load of memory. You do not want the DNS server to swap, so you need to > have enough memory to be sure it can cache enough information. > > As for speed, if you have a machine on a LAN set up as a caching-only > DNS server (that's what I was trying to say before), I'm thinking I'll > take the LAN latency hit over having the MTA competing for resources > with the DNS server. > > Other than that, yea, some kind of RAID solution would be cool for him. > I'd also look at making sure /var/log is on a seperate drive from > /var/spool/mail. I saw an email that indicated that /swap was seperate > from /var/spool, but nothing about where the log files were located. Not > synching after evey write will help obviously, but I recall seeing quite > a benefit from seperate drive for /var/log and /var/spool. > > --Rich > > > Russell Coker wrote: > > > > On Friday 08 June 2001 05:47, Rich Puhek wrote: > > > In addition to checking the disk usage, memory, and the other > > > suggestions that have come up on the list, have you looked at DNS? > > > Quite often you'll find that DNS lookups are severely limiting the > > > performance of something like a mailing list. Make sure that the mail > > > server itself isn't running a DNS server. Make sure you've got one or > > > > Why not? When DNS speed is important I ALWAYS install a local DNS. > > Requests to 127.0.0.1 have to be faster than any other requests... > > > > > two DNS servers in close proximity to the mail server. Make sure that > > > the DNS server process isn't swapping on the DNS servers (for the kind > > > > The output of "top" that he recently posted suggests that nothing is > > swapping. > > > > > with 128 MB of RAM as your DNS server. Also, if possible, I like to > > > have the DNS server I'm querying kept free from being the authoratative > > > server for any domains (not always practical in a real life situation, > > > I know). > > > > How does that help? > > > > If DNS caching is the issue then probably the only place to look for a > > solution is djb-dnscache. > > > > -- > > http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark > > http://www.coker.com.au/postal/ Postal SMTP/POP benchmark > > http://www.coker.com.au/projects.html Projects I am working on > > http://www.coker.com.au/~russell/ My home page > > > > -- > > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > > -- > > _ > > Rich Puhek > ETN Systems Inc. > _ > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > > -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
Memory memory memory! True, memory is not currently a limiting factor, but it likely could be if he were running BIND locally. As for making sure that the server is not authoratative for other domains, that will help keep other DNS demands to a minimum. The mail server will chew up a load of memory (or can anyhow.. his doesn't seem too bad). A highly-utilized DNS server will also chew up a load of memory. You do not want the DNS server to swap, so you need to have enough memory to be sure it can cache enough information. As for speed, if you have a machine on a LAN set up as a caching-only DNS server (that's what I was trying to say before), I'm thinking I'll take the LAN latency hit over having the MTA competing for resources with the DNS server. Other than that, yea, some kind of RAID solution would be cool for him. I'd also look at making sure /var/log is on a seperate drive from /var/spool/mail. I saw an email that indicated that /swap was seperate from /var/spool, but nothing about where the log files were located. Not synching after evey write will help obviously, but I recall seeing quite a benefit from seperate drive for /var/log and /var/spool. --Rich Russell Coker wrote: > > On Friday 08 June 2001 05:47, Rich Puhek wrote: > > In addition to checking the disk usage, memory, and the other > > suggestions that have come up on the list, have you looked at DNS? > > Quite often you'll find that DNS lookups are severely limiting the > > performance of something like a mailing list. Make sure that the mail > > server itself isn't running a DNS server. Make sure you've got one or > > Why not? When DNS speed is important I ALWAYS install a local DNS. > Requests to 127.0.0.1 have to be faster than any other requests... > > > two DNS servers in close proximity to the mail server. Make sure that > > the DNS server process isn't swapping on the DNS servers (for the kind > > The output of "top" that he recently posted suggests that nothing is > swapping. > > > with 128 MB of RAM as your DNS server. Also, if possible, I like to > > have the DNS server I'm querying kept free from being the authoratative > > server for any domains (not always practical in a real life situation, > > I know). > > How does that help? > > If DNS caching is the issue then probably the only place to look for a > solution is djb-dnscache. > > -- > http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark > http://www.coker.com.au/postal/ Postal SMTP/POP benchmark > http://www.coker.com.au/projects.html Projects I am working on > http://www.coker.com.au/~russell/ My home page > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] -- _ Rich Puhek ETN Systems Inc. _
Re: Finding the Bottleneck
Memory memory memory! True, memory is not currently a limiting factor, but it likely could be if he were running BIND locally. As for making sure that the server is not authoratative for other domains, that will help keep other DNS demands to a minimum. The mail server will chew up a load of memory (or can anyhow.. his doesn't seem too bad). A highly-utilized DNS server will also chew up a load of memory. You do not want the DNS server to swap, so you need to have enough memory to be sure it can cache enough information. As for speed, if you have a machine on a LAN set up as a caching-only DNS server (that's what I was trying to say before), I'm thinking I'll take the LAN latency hit over having the MTA competing for resources with the DNS server. Other than that, yea, some kind of RAID solution would be cool for him. I'd also look at making sure /var/log is on a seperate drive from /var/spool/mail. I saw an email that indicated that /swap was seperate from /var/spool, but nothing about where the log files were located. Not synching after evey write will help obviously, but I recall seeing quite a benefit from seperate drive for /var/log and /var/spool. --Rich Russell Coker wrote: > > On Friday 08 June 2001 05:47, Rich Puhek wrote: > > In addition to checking the disk usage, memory, and the other > > suggestions that have come up on the list, have you looked at DNS? > > Quite often you'll find that DNS lookups are severely limiting the > > performance of something like a mailing list. Make sure that the mail > > server itself isn't running a DNS server. Make sure you've got one or > > Why not? When DNS speed is important I ALWAYS install a local DNS. > Requests to 127.0.0.1 have to be faster than any other requests... > > > two DNS servers in close proximity to the mail server. Make sure that > > the DNS server process isn't swapping on the DNS servers (for the kind > > The output of "top" that he recently posted suggests that nothing is > swapping. > > > with 128 MB of RAM as your DNS server. Also, if possible, I like to > > have the DNS server I'm querying kept free from being the authoratative > > server for any domains (not always practical in a real life situation, > > I know). > > How does that help? > > If DNS caching is the issue then probably the only place to look for a > solution is djb-dnscache. > > -- > http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark > http://www.coker.com.au/postal/ Postal SMTP/POP benchmark > http://www.coker.com.au/projects.html Projects I am working on > http://www.coker.com.au/~russell/ My home page > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] -- _ Rich Puhek ETN Systems Inc. _ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
It is strange that we each get such different results.
I haven't run these in totally sanitized environments (haven't had the
luxury or time to do so) so I can't say mine were really scientific, but i
just looked at the real results of each command.
I switch back to using 4Kb readahead as you suggested.
I couldn't set the -m setting any higher:
/dev/hdc:
Model=IBM-DTLA-307015, FwRev=TX2OA50C, SerialNo=YF0YFT68281
Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=40
BuffType=DualPortCache, BuffSize=1916kB, MaxMultSect=16, MultSect=16
CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=30003120
IORDY=on/off, tPIO={min:240,w/IORDY:120}, tDMA={min:120,rec:120}
PIO modes: pio0 pio1 pio2 pio3 pio4
DMA modes: mdma0 mdma1 mdma2 udma0 udma1 *udma2 udma3 udma4 udma5
The maxMultSect is 16 :-/
We haven't run nor tested 2.4 kernels yet... but sooner or later we'll get
there. We run a business and need things to be near 100% stable (or at
least try). Do you see any other way to tweak these disks?
- Original Message -
From: "Russell Coker" <[EMAIL PROTECTED]>
To: "Jason Lim" <[EMAIL PROTECTED]>;
Sent: Friday, June 08, 2001 10:49 PM
Subject: Re: Finding the Bottleneck
On Friday 08 June 2001 16:14, Jason Lim wrote:
> Today I played around with hdparm to see if I could tweak some
>
> Specifically, I set /sbin/hdparm -a4 -c3 -d1 -m16 -u1 /dev/hdc:
>
>-a Get/set sector count for filesystem read-ahead.
> This is used to improve performance in sequential
> reads of large files, by prefetching additional
> blocks in anticipation of them being needed by the
> running task. In the current kernel version
> (2.0.10) this has a default setting of 8 sectors
> (4KB). This value seems good for most purposes,
> but in a system where most file accesses are random
> seeks, a smaller setting might provide better per
> formance. Also, many IDE drives also have a sepa
> rate built-in read-ahead function, which alleviates
> the need for a filesystem read-ahead in many situa
> tions.
> (Since most the emails are small and randomly placed around, I thought
> maybe 2KB read-ahead might make more sense. Tell me if I'm wrong...
> because the performance jump may not be due to this setting)
What is the block size on the file system? What file system are you
using?
If you use Ext2 then I suggest using a 4K block size. It will make FSCK
much faster, and it will reduce fragmentation and file system overhead
and generally make things faster.
If you use a file system with a 4K block size then it probably makes
sense to have a 4K read-ahead (but I have never tested this option).
>-c Query/enable (E)IDE 32-bit I/O support. A numeric
>
> (Couldn't hurt to have it going 32 bit rather than 16 bit)
If you have DMA on then the 32bit IO flag is ignored...
>-d Disable/enable the "using_dma" flag for this drive.
> (this is a dma100 7200 drive so setting this couldn't hurt either.
> Didn't see much performance increase with this though)
This is where you expect to see some performance increase, particularly
when combined with the "-m" option.
>-m Get/set sector count for multiple sector I/O on the
> (I set it to 16... do you think 32 would make more sense?)
I suggest setting it to the maximum. Also why not use kernel 2.4.4 and
compile your kernel to enable DMA and multi-mode by default? The 2.4
series of kernels should increase performance for disk IO even if you use
the same settings, and it has better tuning options.
>-u Get/set interrupt-unmask flag for the drive. A
> (this seem to have the largest performance boost)
That's strange. Last time I played with that one I couldn't find any
benefit to it. I'll have to test again.
--
http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/ Postal SMTP/POP benchmark
http://www.coker.com.au/projects.html Projects I am working on
http://www.coker.com.au/~russell/ My home page
Re: Finding the Bottleneck
On Friday 08 June 2001 16:14, Jason Lim wrote: > Today I played around with hdparm to see if I could tweak some > > Specifically, I set /sbin/hdparm -a4 -c3 -d1 -m16 -u1 /dev/hdc: > >-a Get/set sector count for filesystem read-ahead. > This is used to improve performance in sequential > reads of large files, by prefetching additional > blocks in anticipation of them being needed by the > running task. In the current kernel version > (2.0.10) this has a default setting of 8 sectors > (4KB). This value seems good for most purposes, > but in a system where most file accesses are random > seeks, a smaller setting might provide better per > formance. Also, many IDE drives also have a sepa > rate built-in read-ahead function, which alleviates > the need for a filesystem read-ahead in many situa > tions. > (Since most the emails are small and randomly placed around, I thought > maybe 2KB read-ahead might make more sense. Tell me if I'm wrong... > because the performance jump may not be due to this setting) What is the block size on the file system? What file system are you using? If you use Ext2 then I suggest using a 4K block size. It will make FSCK much faster, and it will reduce fragmentation and file system overhead and generally make things faster. If you use a file system with a 4K block size then it probably makes sense to have a 4K read-ahead (but I have never tested this option). >-c Query/enable (E)IDE 32-bit I/O support. A numeric > > (Couldn't hurt to have it going 32 bit rather than 16 bit) If you have DMA on then the 32bit IO flag is ignored... >-d Disable/enable the "using_dma" flag for this drive. > (this is a dma100 7200 drive so setting this couldn't hurt either. > Didn't see much performance increase with this though) This is where you expect to see some performance increase, particularly when combined with the "-m" option. >-m Get/set sector count for multiple sector I/O on the > (I set it to 16... do you think 32 would make more sense?) I suggest setting it to the maximum. Also why not use kernel 2.4.4 and compile your kernel to enable DMA and multi-mode by default? The 2.4 series of kernels should increase performance for disk IO even if you use the same settings, and it has better tuning options. >-u Get/set interrupt-unmask flag for the drive. A > (this seem to have the largest performance boost) That's strange. Last time I played with that one I couldn't find any benefit to it. I'll have to test again. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page
Re: Finding the Bottleneck
On Friday 08 June 2001 16:26, Jason Lim wrote: > This statement makes me wonder: > "Also even the slowest part of a 45G drive will be twice as fast as the > fastest part of a 15G drive." > > Are you sure? I never heard this before... might be a 1% difference There is a huge difference. I have tested many drives, I have not tested the IBM 15G drives, but the IBM 10G and 13G drives are quite slow, I typically see speeds of around 10MB/s compared to >30MB/s for the fast part of a 45G drive! One of the main ways that modern drives are getting better performance is by having more sectors per track. If you have twice the number of sectors per track and the same rotational speed then the maximum bandwidth is doubled. > there, but twice as fast? The 15G runs at 7200rpms, and I assume the > 45G also does unless its a 1rpm unit (but I don't think there are > any 1000rpm units for ata interfaces, only scsi, right?) The cache on > the 15G is 2Mb, which I assume is the same on the 45G disk. Of course it will depend on other factors too, the version of the kernel (newer kernels give better results), the motherboard, etc. Download my Bonnie++ package and try the zcav program that is included... > "That an average of over 11 messages per second. Not bad with such > tiny hardware." > > Well, it doesn't seem to come to raw hardware performance here. From > "top" output, the 500mhz CPU isn't loaded and swap is not touched at > all (totally in ram). Many small-mid range servers don't run Raid > configurations. However, it does appear that a Raid setup will help > greatly in this case, even with the lower end hardware specs, don't you > think? Definately. You should be able to get 4 times the performance by getting new drives and using RAID-1. > As soon as you clarify the first point, I think we'll be looking far > more closely at Raid solutions (either software or cheap hardware > which would be better if we go with software the bottleneck might > then be shifted to the CPU?!) RAID-1 involves writes going to two drives at the same time, which is exactly double the amount of CPU power for the system calls for writing the blocks of data. The writing of the data should take less CPU power than the file system overhead, and that should be less than the application uses to decide what to write. Almost nothing multiplied by two isn't much... For reads RAID-1 takes no more CPU power than a single drive, as you only read from a single drive. RAID-5 however is quite different. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page
Re: Finding the Bottleneck
I have also thought about that... but if you have a look at Qmail's website (http://www.qmail.org) then you'll see that a number of extremely large mail companies (hotmail for one) uses qmail for... get this... outgoing mail. They could've easily chosen sendmail, postfix (maybe it wasn't around when they designed their systems), zmailer, etc. but those chose qmail. Why would that be? I'm not sure about exim (haven't used it before) but qmail can open (after patching) over 500 or even 1000 concurrent outgoing connections. I've patched and recompiled qmail to send 500 concurrent outgoing emails, which I thought would be plenty. Unfortunately, I haven't had the opportunity to ever see it go that high because (as Russell is suggesting) the hard disks just cannot feed mail fast enough into the system. Most I've ever seen it go was approximately 250 concurrent emails. And who said bandwidth is always the limiting factor? If only there was some simple way to spread the mail queue over 2 disks naturally (not by raid). In the /var/qmail/queue directory, sh-2.05# ls -al total 44 drwxr-xr-x 11 qmailq qmail4096 Jun 8 09:04 . drwxr-xr-x 13 root root 4096 Jun 8 15:24 .. drwx--2 qmails qmail4096 Jun 8 21:54 bounce drwx-- 25 qmails qmail4096 Jun 8 09:04 info drwx-- 25 qmailq qmail4096 Jun 8 09:04 intd drwx-- 25 qmails qmail4096 Jun 8 09:04 local drwxr-x---2 qmailq qmail4096 Jun 8 09:04 lock drwxr-x--- 25 qmailq qmail4096 Jun 8 09:04 mess drwx--2 qmailq qmail4096 Jun 8 21:54 pid drwx-- 25 qmails qmail4096 Jun 8 09:04 remote drwxr-x--- 25 qmailq qmail4096 Jun 8 09:04 todo However, as Russell also mentioned in a previous email, qmail uses inode numbers extensively for the queue, so there would be a huge headache in splitting them up as the system would be unable to cope with 2 different inodes from 2 different hard disks. :-/ Sincerely, Jason - Original Message - From: "Peter Billson" <[EMAIL PROTECTED]> To: "Jason Lim" <[EMAIL PROTECTED]> Cc: Sent: Friday, June 08, 2001 8:04 PM Subject: Re: Finding the Bottleneck > > Additionally, as far as I can see, most emails get sent to the same > > moderately large list of domains (eg. aol), so the local DNS server > > would've cache them already anyway. > > This has been a long thread so forgive me if this has already been > discussed but... > If you are usually delivering multiple messages to a handful of > domains wouldn't the performance be greatly improved by using Exim or > Sendmail that is capable of delivering multiple messages per connection? > Particularly if AOL is involved since I have seen it take four or five > attempts to get a connection to their SPAM clogged mail servers. > If I understand q-mail correctly it does one message per connection > and opening/closing the connection often takes longer then transmitting > the data. > > DISCLAIMER: This post is not intended to start a religious war over > which mail transport program is best! :-) > > Pete > -- > http://www.elbnet.com > ELB Internet Services, Inc. > Web Design, Computer Consulting, Internet Hosting > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > >
Re: Finding the Bottleneck
BTW, I think I noticed something as well (before and after the below optimization). sh-2.05# qmail-qstat messages in queue: 17957 messages in queue but not yet preprocessed: 1229 With everything queue running off one hard disk (disk 1), I never noticed such a few emails not even being able to be preprocessed. It seems the system's ability to preprocess the messages has declined since putting the queue on disk 2. I don't see any reason why... but anyway, facts are facts :-/ Sincerely, Jason - Original Message - From: "Jason Lim" <[EMAIL PROTECTED]> To: Cc: "Russell Coker" <[EMAIL PROTECTED]> Sent: Friday, June 08, 2001 10:14 PM Subject: Re: Finding the Bottleneck > I agree with you that splitting the mail queue to another server wouldn't > help, especially since you've seen the top results, and know that it isn't > very heavily loaded with other jobs in the first place. So I think you are > very correct in saying that the hard disk is the limit here. > > Today I played around with hdparm to see if I could tweak some additional > performance out of the existing drives, and it helped by about 10% (not a > huge jump, but anything helps!). > > Specifically, I set /sbin/hdparm -a4 -c3 -d1 -m16 -u1 /dev/hdc: > >-a Get/set sector count for filesystem read-ahead. > This is used to improve performance in sequential > reads of large files, by prefetching additional > blocks in anticipation of them being needed by the > running task. In the current kernel version > (2.0.10) this has a default setting of 8 sectors > (4KB). This value seems good for most purposes, > but in a system where most file accesses are random > seeks, a smaller setting might provide better per > formance. Also, many IDE drives also have a sepa > rate built-in read-ahead function, which alleviates > the need for a filesystem read-ahead in many situa > tions. > (Since most the emails are small and randomly placed around, I thought > maybe 2KB read-ahead might make more sense. Tell me if I'm wrong... > because the performance jump may not be due to this setting) > >-c Query/enable (E)IDE 32-bit I/O support. A numeric > parameter can be used to enable/disable 32-bit I/O > support: Currently supported values include 0 to > disable 32-bit I/O support, 1 to enable 32-bit data > transfers, and 3 to enable 32-bit data transfers > with a special sync sequence required by many > chipsets. The value 3 works with nearly all 32-bit > IDE chipsets, but incurs slightly more overhead. > Note that "32-bit" refers to data transfers across > a PCI or VLB bus to the interface card only; all > (E)IDE drives still have only a 16-bit connection > over the ribbon cable from the interface card. > > (Couldn't hurt to have it going 32 bit rather than 16 bit) > >-d Disable/enable the "using_dma" flag for this drive. > This option only works with a few combinations of > drives and interfaces which support DMA and which > are known to the IDE driver (and with all supported > XT interfaces). In particular, the Intel Triton > chipset is supported for bus-mastered DMA operation > with many drives (experimental). It is also a good > idea to use the -X34 option in combination with -d1 > to ensure that the drive itself is programmed for > multiword DMA mode2. Using DMA does not necessar > ily provide any improvement in throughput or system > performance, but many folks swear by it. Your > mileage may vary. > (this is a dma100 7200 drive so setting this couldn't hurt either. Didn't > see much performance increase with this though) > >-m Get/set sector count for multiple sector I/O on the > drive. A setting of 0 disables this feature. Mul > tiple sector mode (aka IDE Block Mode), is a fea > ture of most modern IDE hard drives, permitting the > transfer of multiple sectors per I/O interrupt, > rather than the usual one sector per interrupt. > When this feature is enabled, it typically reduces > operating system overhead for disk I/O by 30-50%. >
Re: Finding the Bottleneck
This statement makes me wonder: "Also even the slowest part of a 45G drive will be twice as fast as the fastest part of a 15G drive." Are you sure? I never heard this before... might be a 1% difference there, but twice as fast? The 15G runs at 7200rpms, and I assume the 45G also does unless its a 1rpm unit (but I don't think there are any 1000rpm units for ata interfaces, only scsi, right?) The cache on the 15G is 2Mb, which I assume is the same on the 45G disk. "That an average of over 11 messages per second. Not bad with such tiny hardware." Well, it doesn't seem to come to raw hardware performance here. From "top" output, the 500mhz CPU isn't loaded and swap is not touched at all (totally in ram). Many small-mid range servers don't run Raid configurations. However, it does appear that a Raid setup will help greatly in this case, even with the lower end hardware specs, don't you think? As soon as you clarify the first point, I think we'll be looking far more closely at Raid solutions (either software or cheap hardware which would be better if we go with software the bottleneck might then be shifted to the CPU?!) Sincerely, Jason - Original Message - From: "Russell Coker" <[EMAIL PROTECTED]> To: "Jason Lim" <[EMAIL PROTECTED]>; Sent: Friday, June 08, 2001 10:43 AM Subject: Re: Finding the Bottleneck On Friday 08 June 2001 00:05, Jason Lim wrote: > Thanks for your detailed reply. > > As reliability is not of great importance (only the mail queue will be > there and no critical system files), I'd go for speed and cheap price. > The client doesn't have the huge wads of cash for the optimal system > with scsi drives and 64M cache raid card :-/ So I guess if it comes > down to the crunch, speed and cheap price is it. OK. Software RAID on ATA drives is what you will be using then. > I'll also scratch the NFS idea since qmail wouldn't work well on it, Qmail won't work at all on NFS... > Okay... the server right now is an AMD k6-2 500mhz 128M with 2 15G IBM > drives. At this moment, it is really having trouble processing the mail > queue. > sh-2.05# qmail-qstat > messages in queue: 297121 > messages in queue but not yet preprocessed: 72333 > > I checked the log and yesterday it sent about 1 million emails. Do you > think that the above hardware configuration is performing at it's > realistic limit? That an average of over 11 messages per second. Not bad with such tiny hardware. The first thing to do is to buy some decent hard drives. Get new IBM drives of 40G or more capacity and use the first 15M of them (it reduces seek times and also the first part has a higher transfer rate). Also even the slowest part of a 45G drive will be twice as fast as the fastest part of a 15G drive. Put the spool and queue on a software RAID-1 over two 45G drives and put the rest on a software RAID-1 over the two old 15G drives. Then the system should be considerably more than twice as fast. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page
Re: Finding the Bottleneck
circumstances, such failures can result in massive filesystem corruption. (I set it to 16... do you think 32 would make more sense?) -u Get/set interrupt-unmask flag for the drive. A setting of 1 permits the driver to unmask other interrupts during processing of a disk interrupt, which greatly improves Linux's responsiveness and eliminates "serial port overrun" errors. Use this feature with caution: some drive/controller combi nations do not tolerate the increased I/O latencies possible when this feature is enabled, resulting in massive filesystem corruption. In particular, CMD-640B and RZ1000 (E)IDE interfaces can be unre liable (due to a hardware flaw) when this option is used with kernel versions earlier than 2.0.13. Disabling the IDE prefetch feature of these inter faces (usually a BIOS/CMOS setting) provides a safe fix for the problem for use with earlier kernels. (this seem to have the largest performance boost) Anyway... there it is. Maybe someone else could use these results to get a free 10% increase as well. I stupidly set write_cache on as well, which ended up trashing a bunch of stuff. Thank goodness at that time the server was not being used, and I immediately rebuilt the mail queue. Does anyone have any better configs than above, or some utility that could further boost performance? Sincerely, Jason - Original Message - From: "Russell Coker" <[EMAIL PROTECTED]> To: "Jason Lim" <[EMAIL PROTECTED]>; "Brian May" <[EMAIL PROTECTED]> Cc: Sent: Friday, June 08, 2001 7:17 PM Subject: Re: Finding the Bottleneck On Friday 08 June 2001 12:25, Jason Lim wrote: > The network is connected via 100Mb to a switch, so server to server > connections would be at that limit. Even 10Mb wouldn't be a problem as > I don't think that much data would be crossing the cable.. would it? 10Mb shouldn't be a problem for DNS. Of course there's the issue of what else is on the same cable. There will of course be a few extra milli-seconds latency, but you are correct that it shouldn't make a difference. > As for the "single machine" issue, that would depend. If you're talking > about either getting a couple of SCSI disks, putting them on a hardware > raid, or getting an additional small server just for the queue, then I > think the cost would end up approximately the same. This client doesn't > have the cash for a huge upgrade, but something moderate would be okay. However getting an extra server will not make things faster, in fact it will probably make things slower (maybe a lot slower). Faster hard drives is what you need! > BTW, just to clarify for people who are not familar with qmail, qmail > stores outgoing email in a special queue, not in Maildir. Only incoming > mail is stored in Maildir. The Maildirs are actually stored on Disk 1 > (along with the operating system and everything else except the queue). > I know Maildir can be put in a NFS disk... BUT i've never heard of > anyone putting the mail queue on NFS, so I'm not sure if the file > locking issues you mention would pertain to that as well. For the queue, Qmail creates file names that match Inode numbers (NFS doesn't have Inodes). Qmail also relies on certain link operations being atomic and reliable, while on NFS they aren't guaranteed to be atomic, and packet loss can cause big reliability problems. Consider "ln file file2", when an NFS packet is sent to the server the server will create the link and return success, if the return packet is lost due to packet corruption then the client will re-send the request. The server will notice that file2 exists and return an error message. The result is that the operation succeeded but the client thinks it failed! There are many other issues with NFS for this type of thing. NFS is only good for data that has simple access patterns (read-only files and simple operations like mounting a home directory and editing a file with "vi"), and for applications which have carefully been written to work with NFS (Maildir programs). -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
It is strange that we each get such different results.
I haven't run these in totally sanitized environments (haven't had the
luxury or time to do so) so I can't say mine were really scientific, but i
just looked at the real results of each command.
I switch back to using 4Kb readahead as you suggested.
I couldn't set the -m setting any higher:
/dev/hdc:
Model=IBM-DTLA-307015, FwRev=TX2OA50C, SerialNo=YF0YFT68281
Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=40
BuffType=DualPortCache, BuffSize=1916kB, MaxMultSect=16, MultSect=16
CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=30003120
IORDY=on/off, tPIO={min:240,w/IORDY:120}, tDMA={min:120,rec:120}
PIO modes: pio0 pio1 pio2 pio3 pio4
DMA modes: mdma0 mdma1 mdma2 udma0 udma1 *udma2 udma3 udma4 udma5
The maxMultSect is 16 :-/
We haven't run nor tested 2.4 kernels yet... but sooner or later we'll get
there. We run a business and need things to be near 100% stable (or at
least try). Do you see any other way to tweak these disks?
- Original Message -
From: "Russell Coker" <[EMAIL PROTECTED]>
To: "Jason Lim" <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]>
Sent: Friday, June 08, 2001 10:49 PM
Subject: Re: Finding the Bottleneck
On Friday 08 June 2001 16:14, Jason Lim wrote:
> Today I played around with hdparm to see if I could tweak some
>
> Specifically, I set /sbin/hdparm -a4 -c3 -d1 -m16 -u1 /dev/hdc:
>
>-a Get/set sector count for filesystem read-ahead.
> This is used to improve performance in sequential
> reads of large files, by prefetching additional
> blocks in anticipation of them being needed by the
> running task. In the current kernel version
> (2.0.10) this has a default setting of 8 sectors
> (4KB). This value seems good for most purposes,
> but in a system where most file accesses are random
> seeks, a smaller setting might provide better per
> formance. Also, many IDE drives also have a sepa
> rate built-in read-ahead function, which alleviates
> the need for a filesystem read-ahead in many situa
> tions.
> (Since most the emails are small and randomly placed around, I thought
> maybe 2KB read-ahead might make more sense. Tell me if I'm wrong...
> because the performance jump may not be due to this setting)
What is the block size on the file system? What file system are you
using?
If you use Ext2 then I suggest using a 4K block size. It will make FSCK
much faster, and it will reduce fragmentation and file system overhead
and generally make things faster.
If you use a file system with a 4K block size then it probably makes
sense to have a 4K read-ahead (but I have never tested this option).
>-c Query/enable (E)IDE 32-bit I/O support. A numeric
>
> (Couldn't hurt to have it going 32 bit rather than 16 bit)
If you have DMA on then the 32bit IO flag is ignored...
>-d Disable/enable the "using_dma" flag for this drive.
> (this is a dma100 7200 drive so setting this couldn't hurt either.
> Didn't see much performance increase with this though)
This is where you expect to see some performance increase, particularly
when combined with the "-m" option.
>-m Get/set sector count for multiple sector I/O on the
> (I set it to 16... do you think 32 would make more sense?)
I suggest setting it to the maximum. Also why not use kernel 2.4.4 and
compile your kernel to enable DMA and multi-mode by default? The 2.4
series of kernels should increase performance for disk IO even if you use
the same settings, and it has better tuning options.
>-u Get/set interrupt-unmask flag for the drive. A
> (this seem to have the largest performance boost)
That's strange. Last time I played with that one I couldn't find any
benefit to it. I'll have to test again.
--
http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/ Postal SMTP/POP benchmark
http://www.coker.com.au/projects.html Projects I am working on
http://www.coker.com.au/~russell/ My home page
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
On Friday 08 June 2001 16:14, Jason Lim wrote: > Today I played around with hdparm to see if I could tweak some > > Specifically, I set /sbin/hdparm -a4 -c3 -d1 -m16 -u1 /dev/hdc: > >-a Get/set sector count for filesystem read-ahead. > This is used to improve performance in sequential > reads of large files, by prefetching additional > blocks in anticipation of them being needed by the > running task. In the current kernel version > (2.0.10) this has a default setting of 8 sectors > (4KB). This value seems good for most purposes, > but in a system where most file accesses are random > seeks, a smaller setting might provide better per > formance. Also, many IDE drives also have a sepa > rate built-in read-ahead function, which alleviates > the need for a filesystem read-ahead in many situa > tions. > (Since most the emails are small and randomly placed around, I thought > maybe 2KB read-ahead might make more sense. Tell me if I'm wrong... > because the performance jump may not be due to this setting) What is the block size on the file system? What file system are you using? If you use Ext2 then I suggest using a 4K block size. It will make FSCK much faster, and it will reduce fragmentation and file system overhead and generally make things faster. If you use a file system with a 4K block size then it probably makes sense to have a 4K read-ahead (but I have never tested this option). >-c Query/enable (E)IDE 32-bit I/O support. A numeric > > (Couldn't hurt to have it going 32 bit rather than 16 bit) If you have DMA on then the 32bit IO flag is ignored... >-d Disable/enable the "using_dma" flag for this drive. > (this is a dma100 7200 drive so setting this couldn't hurt either. > Didn't see much performance increase with this though) This is where you expect to see some performance increase, particularly when combined with the "-m" option. >-m Get/set sector count for multiple sector I/O on the > (I set it to 16... do you think 32 would make more sense?) I suggest setting it to the maximum. Also why not use kernel 2.4.4 and compile your kernel to enable DMA and multi-mode by default? The 2.4 series of kernels should increase performance for disk IO even if you use the same settings, and it has better tuning options. >-u Get/set interrupt-unmask flag for the drive. A > (this seem to have the largest performance boost) That's strange. Last time I played with that one I couldn't find any benefit to it. I'll have to test again. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
On Friday 08 June 2001 16:26, Jason Lim wrote: > This statement makes me wonder: > "Also even the slowest part of a 45G drive will be twice as fast as the > fastest part of a 15G drive." > > Are you sure? I never heard this before... might be a 1% difference There is a huge difference. I have tested many drives, I have not tested the IBM 15G drives, but the IBM 10G and 13G drives are quite slow, I typically see speeds of around 10MB/s compared to >30MB/s for the fast part of a 45G drive! One of the main ways that modern drives are getting better performance is by having more sectors per track. If you have twice the number of sectors per track and the same rotational speed then the maximum bandwidth is doubled. > there, but twice as fast? The 15G runs at 7200rpms, and I assume the > 45G also does unless its a 1rpm unit (but I don't think there are > any 1000rpm units for ata interfaces, only scsi, right?) The cache on > the 15G is 2Mb, which I assume is the same on the 45G disk. Of course it will depend on other factors too, the version of the kernel (newer kernels give better results), the motherboard, etc. Download my Bonnie++ package and try the zcav program that is included... > "That an average of over 11 messages per second. Not bad with such > tiny hardware." > > Well, it doesn't seem to come to raw hardware performance here. From > "top" output, the 500mhz CPU isn't loaded and swap is not touched at > all (totally in ram). Many small-mid range servers don't run Raid > configurations. However, it does appear that a Raid setup will help > greatly in this case, even with the lower end hardware specs, don't you > think? Definately. You should be able to get 4 times the performance by getting new drives and using RAID-1. > As soon as you clarify the first point, I think we'll be looking far > more closely at Raid solutions (either software or cheap hardware > which would be better if we go with software the bottleneck might > then be shifted to the CPU?!) RAID-1 involves writes going to two drives at the same time, which is exactly double the amount of CPU power for the system calls for writing the blocks of data. The writing of the data should take less CPU power than the file system overhead, and that should be less than the application uses to decide what to write. Almost nothing multiplied by two isn't much... For reads RAID-1 takes no more CPU power than a single drive, as you only read from a single drive. RAID-5 however is quite different. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
BTW, I think I noticed something as well (before and after the below optimization). sh-2.05# qmail-qstat messages in queue: 17957 messages in queue but not yet preprocessed: 1229 With everything queue running off one hard disk (disk 1), I never noticed such a few emails not even being able to be preprocessed. It seems the system's ability to preprocess the messages has declined since putting the queue on disk 2. I don't see any reason why... but anyway, facts are facts :-/ Sincerely, Jason - Original Message - From: "Jason Lim" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Cc: "Russell Coker" <[EMAIL PROTECTED]> Sent: Friday, June 08, 2001 10:14 PM Subject: Re: Finding the Bottleneck > I agree with you that splitting the mail queue to another server wouldn't > help, especially since you've seen the top results, and know that it isn't > very heavily loaded with other jobs in the first place. So I think you are > very correct in saying that the hard disk is the limit here. > > Today I played around with hdparm to see if I could tweak some additional > performance out of the existing drives, and it helped by about 10% (not a > huge jump, but anything helps!). > > Specifically, I set /sbin/hdparm -a4 -c3 -d1 -m16 -u1 /dev/hdc: > >-a Get/set sector count for filesystem read-ahead. > This is used to improve performance in sequential > reads of large files, by prefetching additional > blocks in anticipation of them being needed by the > running task. In the current kernel version > (2.0.10) this has a default setting of 8 sectors > (4KB). This value seems good for most purposes, > but in a system where most file accesses are random > seeks, a smaller setting might provide better per > formance. Also, many IDE drives also have a sepa > rate built-in read-ahead function, which alleviates > the need for a filesystem read-ahead in many situa > tions. > (Since most the emails are small and randomly placed around, I thought > maybe 2KB read-ahead might make more sense. Tell me if I'm wrong... > because the performance jump may not be due to this setting) > >-c Query/enable (E)IDE 32-bit I/O support. A numeric > parameter can be used to enable/disable 32-bit I/O > support: Currently supported values include 0 to > disable 32-bit I/O support, 1 to enable 32-bit data > transfers, and 3 to enable 32-bit data transfers > with a special sync sequence required by many > chipsets. The value 3 works with nearly all 32-bit > IDE chipsets, but incurs slightly more overhead. > Note that "32-bit" refers to data transfers across > a PCI or VLB bus to the interface card only; all > (E)IDE drives still have only a 16-bit connection > over the ribbon cable from the interface card. > > (Couldn't hurt to have it going 32 bit rather than 16 bit) > >-d Disable/enable the "using_dma" flag for this drive. > This option only works with a few combinations of > drives and interfaces which support DMA and which > are known to the IDE driver (and with all supported > XT interfaces). In particular, the Intel Triton > chipset is supported for bus-mastered DMA operation > with many drives (experimental). It is also a good > idea to use the -X34 option in combination with -d1 > to ensure that the drive itself is programmed for > multiword DMA mode2. Using DMA does not necessar > ily provide any improvement in throughput or system > performance, but many folks swear by it. Your > mileage may vary. > (this is a dma100 7200 drive so setting this couldn't hurt either. Didn't > see much performance increase with this though) > >-m Get/set sector count for multiple sector I/O on the > drive. A setting of 0 disables this feature. Mul > tiple sector mode (aka IDE Block Mode), is a fea > ture of most modern IDE hard drives, permitting the > transfer of multiple sectors per I/O interrupt, > rather than the usual one sector per interrupt. > When this feature is enabled, it typically reduces > operating system over
Re: Finding the Bottleneck
> Additionally, as far as I can see, most emails get sent to the same > moderately large list of domains (eg. aol), so the local DNS server > would've cache them already anyway. This has been a long thread so forgive me if this has already been discussed but... If you are usually delivering multiple messages to a handful of domains wouldn't the performance be greatly improved by using Exim or Sendmail that is capable of delivering multiple messages per connection? Particularly if AOL is involved since I have seen it take four or five attempts to get a connection to their SPAM clogged mail servers. If I understand q-mail correctly it does one message per connection and opening/closing the connection often takes longer then transmitting the data. DISCLAIMER: This post is not intended to start a religious war over which mail transport program is best! :-) Pete -- http://www.elbnet.com ELB Internet Services, Inc. Web Design, Computer Consulting, Internet Hosting
Re: Finding the Bottleneck
This statement makes me wonder: "Also even the slowest part of a 45G drive will be twice as fast as the fastest part of a 15G drive." Are you sure? I never heard this before... might be a 1% difference there, but twice as fast? The 15G runs at 7200rpms, and I assume the 45G also does unless its a 1rpm unit (but I don't think there are any 1000rpm units for ata interfaces, only scsi, right?) The cache on the 15G is 2Mb, which I assume is the same on the 45G disk. "That an average of over 11 messages per second. Not bad with such tiny hardware." Well, it doesn't seem to come to raw hardware performance here. From "top" output, the 500mhz CPU isn't loaded and swap is not touched at all (totally in ram). Many small-mid range servers don't run Raid configurations. However, it does appear that a Raid setup will help greatly in this case, even with the lower end hardware specs, don't you think? As soon as you clarify the first point, I think we'll be looking far more closely at Raid solutions (either software or cheap hardware which would be better if we go with software the bottleneck might then be shifted to the CPU?!) Sincerely, Jason - Original Message - From: "Russell Coker" <[EMAIL PROTECTED]> To: "Jason Lim" <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]> Sent: Friday, June 08, 2001 10:43 AM Subject: Re: Finding the Bottleneck On Friday 08 June 2001 00:05, Jason Lim wrote: > Thanks for your detailed reply. > > As reliability is not of great importance (only the mail queue will be > there and no critical system files), I'd go for speed and cheap price. > The client doesn't have the huge wads of cash for the optimal system > with scsi drives and 64M cache raid card :-/ So I guess if it comes > down to the crunch, speed and cheap price is it. OK. Software RAID on ATA drives is what you will be using then. > I'll also scratch the NFS idea since qmail wouldn't work well on it, Qmail won't work at all on NFS... > Okay... the server right now is an AMD k6-2 500mhz 128M with 2 15G IBM > drives. At this moment, it is really having trouble processing the mail > queue. > sh-2.05# qmail-qstat > messages in queue: 297121 > messages in queue but not yet preprocessed: 72333 > > I checked the log and yesterday it sent about 1 million emails. Do you > think that the above hardware configuration is performing at it's > realistic limit? That an average of over 11 messages per second. Not bad with such tiny hardware. The first thing to do is to buy some decent hard drives. Get new IBM drives of 40G or more capacity and use the first 15M of them (it reduces seek times and also the first part has a higher transfer rate). Also even the slowest part of a 45G drive will be twice as fast as the fastest part of a 15G drive. Put the spool and queue on a software RAID-1 over two 45G drives and put the rest on a software RAID-1 over the two old 15G drives. Then the system should be considerably more than twice as fast. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
circumstances, such failures can result in massive filesystem corruption. (I set it to 16... do you think 32 would make more sense?) -u Get/set interrupt-unmask flag for the drive. A setting of 1 permits the driver to unmask other interrupts during processing of a disk interrupt, which greatly improves Linux's responsiveness and eliminates "serial port overrun" errors. Use this feature with caution: some drive/controller combi nations do not tolerate the increased I/O latencies possible when this feature is enabled, resulting in massive filesystem corruption. In particular, CMD-640B and RZ1000 (E)IDE interfaces can be unre liable (due to a hardware flaw) when this option is used with kernel versions earlier than 2.0.13. Disabling the IDE prefetch feature of these inter faces (usually a BIOS/CMOS setting) provides a safe fix for the problem for use with earlier kernels. (this seem to have the largest performance boost) Anyway... there it is. Maybe someone else could use these results to get a free 10% increase as well. I stupidly set write_cache on as well, which ended up trashing a bunch of stuff. Thank goodness at that time the server was not being used, and I immediately rebuilt the mail queue. Does anyone have any better configs than above, or some utility that could further boost performance? Sincerely, Jason - Original Message - From: "Russell Coker" <[EMAIL PROTECTED]> To: "Jason Lim" <[EMAIL PROTECTED]>; "Brian May" <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Friday, June 08, 2001 7:17 PM Subject: Re: Finding the Bottleneck On Friday 08 June 2001 12:25, Jason Lim wrote: > The network is connected via 100Mb to a switch, so server to server > connections would be at that limit. Even 10Mb wouldn't be a problem as > I don't think that much data would be crossing the cable.. would it? 10Mb shouldn't be a problem for DNS. Of course there's the issue of what else is on the same cable. There will of course be a few extra milli-seconds latency, but you are correct that it shouldn't make a difference. > As for the "single machine" issue, that would depend. If you're talking > about either getting a couple of SCSI disks, putting them on a hardware > raid, or getting an additional small server just for the queue, then I > think the cost would end up approximately the same. This client doesn't > have the cash for a huge upgrade, but something moderate would be okay. However getting an extra server will not make things faster, in fact it will probably make things slower (maybe a lot slower). Faster hard drives is what you need! > BTW, just to clarify for people who are not familar with qmail, qmail > stores outgoing email in a special queue, not in Maildir. Only incoming > mail is stored in Maildir. The Maildirs are actually stored on Disk 1 > (along with the operating system and everything else except the queue). > I know Maildir can be put in a NFS disk... BUT i've never heard of > anyone putting the mail queue on NFS, so I'm not sure if the file > locking issues you mention would pertain to that as well. For the queue, Qmail creates file names that match Inode numbers (NFS doesn't have Inodes). Qmail also relies on certain link operations being atomic and reliable, while on NFS they aren't guaranteed to be atomic, and packet loss can cause big reliability problems. Consider "ln file file2", when an NFS packet is sent to the server the server will create the link and return success, if the return packet is lost due to packet corruption then the client will re-send the request. The server will notice that file2 exists and return an error message. The result is that the operation succeeded but the client thinks it failed! There are many other issues with NFS for this type of thing. NFS is only good for data that has simple access patterns (read-only files and simple operations like mounting a home directory and editing a file with "vi"), and for applications which have carefully been written to work with NFS (Maildir programs). -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
I have also thought about that... but if you have a look at Qmail's website (http://www.qmail.org) then you'll see that a number of extremely large mail companies (hotmail for one) uses qmail for... get this... outgoing mail. They could've easily chosen sendmail, postfix (maybe it wasn't around when they designed their systems), zmailer, etc. but those chose qmail. Why would that be? I'm not sure about exim (haven't used it before) but qmail can open (after patching) over 500 or even 1000 concurrent outgoing connections. I've patched and recompiled qmail to send 500 concurrent outgoing emails, which I thought would be plenty. Unfortunately, I haven't had the opportunity to ever see it go that high because (as Russell is suggesting) the hard disks just cannot feed mail fast enough into the system. Most I've ever seen it go was approximately 250 concurrent emails. And who said bandwidth is always the limiting factor? If only there was some simple way to spread the mail queue over 2 disks naturally (not by raid). In the /var/qmail/queue directory, sh-2.05# ls -al total 44 drwxr-xr-x 11 qmailq qmail4096 Jun 8 09:04 . drwxr-xr-x 13 root root 4096 Jun 8 15:24 .. drwx--2 qmails qmail4096 Jun 8 21:54 bounce drwx-- 25 qmails qmail4096 Jun 8 09:04 info drwx-- 25 qmailq qmail4096 Jun 8 09:04 intd drwx-- 25 qmails qmail4096 Jun 8 09:04 local drwxr-x---2 qmailq qmail4096 Jun 8 09:04 lock drwxr-x--- 25 qmailq qmail4096 Jun 8 09:04 mess drwx--2 qmailq qmail4096 Jun 8 21:54 pid drwx-- 25 qmails qmail4096 Jun 8 09:04 remote drwxr-x--- 25 qmailq qmail4096 Jun 8 09:04 todo However, as Russell also mentioned in a previous email, qmail uses inode numbers extensively for the queue, so there would be a huge headache in splitting them up as the system would be unable to cope with 2 different inodes from 2 different hard disks. :-/ Sincerely, Jason - Original Message - From: "Peter Billson" <[EMAIL PROTECTED]> To: "Jason Lim" <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Friday, June 08, 2001 8:04 PM Subject: Re: Finding the Bottleneck > > Additionally, as far as I can see, most emails get sent to the same > > moderately large list of domains (eg. aol), so the local DNS server > > would've cache them already anyway. > > This has been a long thread so forgive me if this has already been > discussed but... > If you are usually delivering multiple messages to a handful of > domains wouldn't the performance be greatly improved by using Exim or > Sendmail that is capable of delivering multiple messages per connection? > Particularly if AOL is involved since I have seen it take four or five > attempts to get a connection to their SPAM clogged mail servers. > If I understand q-mail correctly it does one message per connection > and opening/closing the connection often takes longer then transmitting > the data. > > DISCLAIMER: This post is not intended to start a religious war over > which mail transport program is best! :-) > > Pete > -- > http://www.elbnet.com > ELB Internet Services, Inc. > Web Design, Computer Consulting, Internet Hosting > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > > -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
On Friday 08 June 2001 12:25, Jason Lim wrote: > The network is connected via 100Mb to a switch, so server to server > connections would be at that limit. Even 10Mb wouldn't be a problem as > I don't think that much data would be crossing the cable.. would it? 10Mb shouldn't be a problem for DNS. Of course there's the issue of what else is on the same cable. There will of course be a few extra milli-seconds latency, but you are correct that it shouldn't make a difference. > As for the "single machine" issue, that would depend. If you're talking > about either getting a couple of SCSI disks, putting them on a hardware > raid, or getting an additional small server just for the queue, then I > think the cost would end up approximately the same. This client doesn't > have the cash for a huge upgrade, but something moderate would be okay. However getting an extra server will not make things faster, in fact it will probably make things slower (maybe a lot slower). Faster hard drives is what you need! > BTW, just to clarify for people who are not familar with qmail, qmail > stores outgoing email in a special queue, not in Maildir. Only incoming > mail is stored in Maildir. The Maildirs are actually stored on Disk 1 > (along with the operating system and everything else except the queue). > I know Maildir can be put in a NFS disk... BUT i've never heard of > anyone putting the mail queue on NFS, so I'm not sure if the file > locking issues you mention would pertain to that as well. For the queue, Qmail creates file names that match Inode numbers (NFS doesn't have Inodes). Qmail also relies on certain link operations being atomic and reliable, while on NFS they aren't guaranteed to be atomic, and packet loss can cause big reliability problems. Consider "ln file file2", when an NFS packet is sent to the server the server will create the link and return success, if the return packet is lost due to packet corruption then the client will re-send the request. The server will notice that file2 exists and return an error message. The result is that the operation succeeded but the client thinks it failed! There are many other issues with NFS for this type of thing. NFS is only good for data that has simple access patterns (read-only files and simple operations like mounting a home directory and editing a file with "vi"), and for applications which have carefully been written to work with NFS (Maildir programs). -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page
Re: Finding the Bottleneck
I agree. I always thought that doing a local lookup would be far faster than doing one on a remote dns cache. We use bind, and set the forwarders to 2 other DNS servers that are only lightly loaded and on the same network. Additionally, as far as I can see, most emails get sent to the same moderately large list of domains (eg. aol), so the local DNS server would've cache them already anyway. Sincerely, Jason - Original Message - From: "Russell Coker" <[EMAIL PROTECTED]> To: "Rich Puhek" <[EMAIL PROTECTED]>; "Jason Lim" <[EMAIL PROTECTED]> Cc: Sent: Friday, June 08, 2001 6:18 PM Subject: Re: Finding the Bottleneck On Friday 08 June 2001 05:47, Rich Puhek wrote: > In addition to checking the disk usage, memory, and the other > suggestions that have come up on the list, have you looked at DNS? > Quite often you'll find that DNS lookups are severely limiting the > performance of something like a mailing list. Make sure that the mail > server itself isn't running a DNS server. Make sure you've got one or Why not? When DNS speed is important I ALWAYS install a local DNS. Requests to 127.0.0.1 have to be faster than any other requests... > two DNS servers in close proximity to the mail server. Make sure that > the DNS server process isn't swapping on the DNS servers (for the kind The output of "top" that he recently posted suggests that nothing is swapping. > with 128 MB of RAM as your DNS server. Also, if possible, I like to > have the DNS server I'm querying kept free from being the authoratative > server for any domains (not always practical in a real life situation, > I know). How does that help? If DNS caching is the issue then probably the only place to look for a solution is djb-dnscache. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page
Re: Finding the Bottleneck
On Friday 08 June 2001 05:47, Rich Puhek wrote: > In addition to checking the disk usage, memory, and the other > suggestions that have come up on the list, have you looked at DNS? > Quite often you'll find that DNS lookups are severely limiting the > performance of something like a mailing list. Make sure that the mail > server itself isn't running a DNS server. Make sure you've got one or Why not? When DNS speed is important I ALWAYS install a local DNS. Requests to 127.0.0.1 have to be faster than any other requests... > two DNS servers in close proximity to the mail server. Make sure that > the DNS server process isn't swapping on the DNS servers (for the kind The output of "top" that he recently posted suggests that nothing is swapping. > with 128 MB of RAM as your DNS server. Also, if possible, I like to > have the DNS server I'm querying kept free from being the authoratative > server for any domains (not always practical in a real life situation, > I know). How does that help? If DNS caching is the issue then probably the only place to look for a solution is djb-dnscache. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page
Re: Finding the Bottleneck
Hi, The network is connected via 100Mb to a switch, so server to server connections would be at that limit. Even 10Mb wouldn't be a problem as I don't think that much data would be crossing the cable.. would it? As for the "single machine" issue, that would depend. If you're talking about either getting a couple of SCSI disks, putting them on a hardware raid, or getting an additional small server just for the queue, then I think the cost would end up approximately the same. This client doesn't have the cash for a huge upgrade, but something moderate would be okay. BTW, just to clarify for people who are not familar with qmail, qmail stores outgoing email in a special queue, not in Maildir. Only incoming mail is stored in Maildir. The Maildirs are actually stored on Disk 1 (along with the operating system and everything else except the queue). I know Maildir can be put in a NFS disk... BUT i've never heard of anyone putting the mail queue on NFS, so I'm not sure if the file locking issues you mention would pertain to that as well. Sincerely, Jason - Original Message - From: "Russell Coker" <[EMAIL PROTECTED]> To: "Brian May" <[EMAIL PROTECTED]> Cc: "Jason Lim" <[EMAIL PROTECTED]>; Sent: Friday, June 08, 2001 5:45 PM Subject: Re: Finding the Bottleneck On Friday 08 June 2001 10:49, Brian May wrote: > Russell> If the NFS server has the same disk system then you will > Russell> only make things worse. Anything you could do to give > Russell> the NFS server better IO performance could more > Russell> productively be done to the main server. Also many of > Russell> the common Unix mail server programs are specifically > Russell> designed to have the queue on a local file system with > Russell> standard Unix semantics (Inode numbers etc). Qmail is > Russell> one mail server that I have found to not work with it's > Russell> queue on NFS, I haven't seriously tried any others. I've > Russell> CC'd Brian May because he does a lot more NFS stuff with > Russell> Debian than most people and he may be able to advise you. > > It depends on how much you want to share via NFS, what speed/type > network you have, etc. Jason has previously stated that he only wants a single machine and wants things to be as cheap as possible. > The idea of putting any queue on NFS is generally discouraged (due to > file locking issues). Maildir is one exception to this rule. Maildir != queue... -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
Hi, Yes that is correct. The design of qmail forces a connection for each email message. Changing that behaviour would require massive patching. Sincerely, Jason - Original Message - From: "Tomasz Papszun" <[EMAIL PROTECTED]> To: "Rich Puhek" <[EMAIL PROTECTED]> Cc: "Jason Lim" <[EMAIL PROTECTED]>; Sent: Friday, June 08, 2001 6:04 PM Subject: Re: Finding the Bottleneck > On Thu, 07 Jun 2001 at 22:47:09 -0500, Rich Puhek wrote: > [...] > > Also, there are probably some optimizations you can do for queue sort > > order. I'm most familiar with Sendmail, not qmail, so I don't know the > > exact settings, but try to process the queue according to recipient > > domain. That way, you gain some advantages with holding SMTP connections > > open to a server, rather than closing and reopening a session, etc. > > > > --Rich > > If my memory serves me well, qmail opens a new session for each message, > even if this message is to be delivered to the same server. > I may be wrong though. > > -- > Tomasz Papszun SysAdm @ TP S.A. Lodz, Poland | And it's only > [EMAIL PROTECTED] http://www.lodz.tpsa.pl/ | ones and zeros. > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > >
Re: Finding the Bottleneck
On Thu, 07 Jun 2001 at 22:47:09 -0500, Rich Puhek wrote: [...] > Also, there are probably some optimizations you can do for queue sort > order. I'm most familiar with Sendmail, not qmail, so I don't know the > exact settings, but try to process the queue according to recipient > domain. That way, you gain some advantages with holding SMTP connections > open to a server, rather than closing and reopening a session, etc. > > --Rich If my memory serves me well, qmail opens a new session for each message, even if this message is to be delivered to the same server. I may be wrong though. -- Tomasz Papszun SysAdm @ TP S.A. Lodz, Poland | And it's only [EMAIL PROTECTED] http://www.lodz.tpsa.pl/ | ones and zeros.
Re: Finding the Bottleneck
On Friday 08 June 2001 10:49, Brian May wrote: > Russell> If the NFS server has the same disk system then you will > Russell> only make things worse. Anything you could do to give > Russell> the NFS server better IO performance could more > Russell> productively be done to the main server. Also many of > Russell> the common Unix mail server programs are specifically > Russell> designed to have the queue on a local file system with > Russell> standard Unix semantics (Inode numbers etc). Qmail is > Russell> one mail server that I have found to not work with it's > Russell> queue on NFS, I haven't seriously tried any others. I've > Russell> CC'd Brian May because he does a lot more NFS stuff with > Russell> Debian than most people and he may be able to advise you. > > It depends on how much you want to share via NFS, what speed/type > network you have, etc. Jason has previously stated that he only wants a single machine and wants things to be as cheap as possible. > The idea of putting any queue on NFS is generally discouraged (due to > file locking issues). Maildir is one exception to this rule. Maildir != queue... -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page
Re: Finding the Bottleneck
> Additionally, as far as I can see, most emails get sent to the same > moderately large list of domains (eg. aol), so the local DNS server > would've cache them already anyway. This has been a long thread so forgive me if this has already been discussed but... If you are usually delivering multiple messages to a handful of domains wouldn't the performance be greatly improved by using Exim or Sendmail that is capable of delivering multiple messages per connection? Particularly if AOL is involved since I have seen it take four or five attempts to get a connection to their SPAM clogged mail servers. If I understand q-mail correctly it does one message per connection and opening/closing the connection often takes longer then transmitting the data. DISCLAIMER: This post is not intended to start a religious war over which mail transport program is best! :-) Pete -- http://www.elbnet.com ELB Internet Services, Inc. Web Design, Computer Consulting, Internet Hosting -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
Maybe a local caching nameserver will help here as well. (Just a quick though.) Cheers, Marcel Rich Puhek <[EMAIL PROTECTED]> 7 Jun 2001, at 22:47: > By the way, > > In addition to checking the disk usage, memory, and the other > suggestions that have come up on the list, have you looked at DNS? > Quite often you'll find that DNS lookups are severely limiting the > performance of something like a mailing list. Make sure that the mail > server itself isn't running a DNS server. Make sure you've got one or > two DNS servers in close proximity to the mail server. Make sure that > the DNS server process isn't swapping on the DNS servers (for the kind > of traffic you're pushing through, you may need a pentium class > machine with 128 MB of RAM as your DNS server. Also, if possible, I > like to have the DNS server I'm querying kept free from being the > authoratative server for any domains (not always practical in a real > life situation, I know). > > Also, there are probably some optimizations you can do for queue sort > order. I'm most familiar with Sendmail, not qmail, so I don't know the > exact settings, but try to process the queue according to recipient > domain. That way, you gain some advantages with holding SMTP > connections open to a server, rather than closing and reopening a > session, etc. > > --Rich > > Jason Lim wrote: > > > > Hi all, > > > > I was wondering if there is a way to find out what/where the > > bottleneck of a large mail server is. > > > > A client is running a huge mail server that we set up for them > > (running qmail), but performance seems to be limited somewhere. > > Qmail has already been optimized as far as it can go (big-todo > > patch, large concurrency patch, etc.). > > > > We're thinking one of the Hard Disks may be the main bottleneck (the > > mail queue is already on a seperate disk on a seperate IDE channel > > from other disks). Is there any way to find out how "utilized" the > > IDE channel/hard disk is, or how hard it is going? Seems that right > > now the only way we really know is by looking at the light on the > > server case (how technical ;-) ). Must be a better way... > > > > The bottleneck wouldn't be bandwidth... it is definately with the > > server. Perhaps the CPU or kernel is the bottleneck (load average: > > 4.84, 3.94, 3.88, going up to 5 or 6 during heavy mailing)? Is that > > normal for a large mail server? We haven't run such a large mail > > server before (anywhere between 500k to 1M per day so far, > > increasing each day), so ANY tips and pointers would be greatly > > appreciated. We've already been playing around with hdparm to see if > > we can tweak the disks, but doesn't seem to help much. Maybe some > > cache settings we can fiddle with? Maybe the mail queue disk could > > use a different file cache setting (each email being from 1K to 10K > > on average)? > > > > Thanks in advance! > > > > Sincerely, > > Jason -- .-. / ´ ´ ` ` ` //_\\ || / ` || |\ .´ / \`.__,´ \ ` Debian / GNU Linux ` .
Re: Finding the Bottleneck
> "Russell" == Russell Coker <[EMAIL PROTECTED]> writes: Russell> If the NFS server has the same disk system then you will Russell> only make things worse. Anything you could do to give Russell> the NFS server better IO performance could more Russell> productively be done to the main server. Also many of Russell> the common Unix mail server programs are specifically Russell> designed to have the queue on a local file system with Russell> standard Unix semantics (Inode numbers etc). Qmail is Russell> one mail server that I have found to not work with it's Russell> queue on NFS, I haven't seriously tried any others. I've Russell> CC'd Brian May because he does a lot more NFS stuff with Russell> Debian than most people and he may be able to advise you. It depends on how much you want to share via NFS, what speed/type network you have, etc. The idea of putting any queue on NFS is generally discouraged (due to file locking issues). Maildir is one exception to this rule. -- Brian May <[EMAIL PROTECTED]>
Re: Finding the Bottleneck
On Friday 08 June 2001 12:25, Jason Lim wrote: > The network is connected via 100Mb to a switch, so server to server > connections would be at that limit. Even 10Mb wouldn't be a problem as > I don't think that much data would be crossing the cable.. would it? 10Mb shouldn't be a problem for DNS. Of course there's the issue of what else is on the same cable. There will of course be a few extra milli-seconds latency, but you are correct that it shouldn't make a difference. > As for the "single machine" issue, that would depend. If you're talking > about either getting a couple of SCSI disks, putting them on a hardware > raid, or getting an additional small server just for the queue, then I > think the cost would end up approximately the same. This client doesn't > have the cash for a huge upgrade, but something moderate would be okay. However getting an extra server will not make things faster, in fact it will probably make things slower (maybe a lot slower). Faster hard drives is what you need! > BTW, just to clarify for people who are not familar with qmail, qmail > stores outgoing email in a special queue, not in Maildir. Only incoming > mail is stored in Maildir. The Maildirs are actually stored on Disk 1 > (along with the operating system and everything else except the queue). > I know Maildir can be put in a NFS disk... BUT i've never heard of > anyone putting the mail queue on NFS, so I'm not sure if the file > locking issues you mention would pertain to that as well. For the queue, Qmail creates file names that match Inode numbers (NFS doesn't have Inodes). Qmail also relies on certain link operations being atomic and reliable, while on NFS they aren't guaranteed to be atomic, and packet loss can cause big reliability problems. Consider "ln file file2", when an NFS packet is sent to the server the server will create the link and return success, if the return packet is lost due to packet corruption then the client will re-send the request. The server will notice that file2 exists and return an error message. The result is that the operation succeeded but the client thinks it failed! There are many other issues with NFS for this type of thing. NFS is only good for data that has simple access patterns (read-only files and simple operations like mounting a home directory and editing a file with "vi"), and for applications which have carefully been written to work with NFS (Maildir programs). -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
I agree. I always thought that doing a local lookup would be far faster than doing one on a remote dns cache. We use bind, and set the forwarders to 2 other DNS servers that are only lightly loaded and on the same network. Additionally, as far as I can see, most emails get sent to the same moderately large list of domains (eg. aol), so the local DNS server would've cache them already anyway. Sincerely, Jason - Original Message - From: "Russell Coker" <[EMAIL PROTECTED]> To: "Rich Puhek" <[EMAIL PROTECTED]>; "Jason Lim" <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Friday, June 08, 2001 6:18 PM Subject: Re: Finding the Bottleneck On Friday 08 June 2001 05:47, Rich Puhek wrote: > In addition to checking the disk usage, memory, and the other > suggestions that have come up on the list, have you looked at DNS? > Quite often you'll find that DNS lookups are severely limiting the > performance of something like a mailing list. Make sure that the mail > server itself isn't running a DNS server. Make sure you've got one or Why not? When DNS speed is important I ALWAYS install a local DNS. Requests to 127.0.0.1 have to be faster than any other requests... > two DNS servers in close proximity to the mail server. Make sure that > the DNS server process isn't swapping on the DNS servers (for the kind The output of "top" that he recently posted suggests that nothing is swapping. > with 128 MB of RAM as your DNS server. Also, if possible, I like to > have the DNS server I'm querying kept free from being the authoratative > server for any domains (not always practical in a real life situation, > I know). How does that help? If DNS caching is the issue then probably the only place to look for a solution is djb-dnscache. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
On Friday 08 June 2001 05:47, Rich Puhek wrote: > In addition to checking the disk usage, memory, and the other > suggestions that have come up on the list, have you looked at DNS? > Quite often you'll find that DNS lookups are severely limiting the > performance of something like a mailing list. Make sure that the mail > server itself isn't running a DNS server. Make sure you've got one or Why not? When DNS speed is important I ALWAYS install a local DNS. Requests to 127.0.0.1 have to be faster than any other requests... > two DNS servers in close proximity to the mail server. Make sure that > the DNS server process isn't swapping on the DNS servers (for the kind The output of "top" that he recently posted suggests that nothing is swapping. > with 128 MB of RAM as your DNS server. Also, if possible, I like to > have the DNS server I'm querying kept free from being the authoratative > server for any domains (not always practical in a real life situation, > I know). How does that help? If DNS caching is the issue then probably the only place to look for a solution is djb-dnscache. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
Hi, The network is connected via 100Mb to a switch, so server to server connections would be at that limit. Even 10Mb wouldn't be a problem as I don't think that much data would be crossing the cable.. would it? As for the "single machine" issue, that would depend. If you're talking about either getting a couple of SCSI disks, putting them on a hardware raid, or getting an additional small server just for the queue, then I think the cost would end up approximately the same. This client doesn't have the cash for a huge upgrade, but something moderate would be okay. BTW, just to clarify for people who are not familar with qmail, qmail stores outgoing email in a special queue, not in Maildir. Only incoming mail is stored in Maildir. The Maildirs are actually stored on Disk 1 (along with the operating system and everything else except the queue). I know Maildir can be put in a NFS disk... BUT i've never heard of anyone putting the mail queue on NFS, so I'm not sure if the file locking issues you mention would pertain to that as well. Sincerely, Jason - Original Message - From: "Russell Coker" <[EMAIL PROTECTED]> To: "Brian May" <[EMAIL PROTECTED]> Cc: "Jason Lim" <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]> Sent: Friday, June 08, 2001 5:45 PM Subject: Re: Finding the Bottleneck On Friday 08 June 2001 10:49, Brian May wrote: > Russell> If the NFS server has the same disk system then you will > Russell> only make things worse. Anything you could do to give > Russell> the NFS server better IO performance could more > Russell> productively be done to the main server. Also many of > Russell> the common Unix mail server programs are specifically > Russell> designed to have the queue on a local file system with > Russell> standard Unix semantics (Inode numbers etc). Qmail is > Russell> one mail server that I have found to not work with it's > Russell> queue on NFS, I haven't seriously tried any others. I've > Russell> CC'd Brian May because he does a lot more NFS stuff with > Russell> Debian than most people and he may be able to advise you. > > It depends on how much you want to share via NFS, what speed/type > network you have, etc. Jason has previously stated that he only wants a single machine and wants things to be as cheap as possible. > The idea of putting any queue on NFS is generally discouraged (due to > file locking issues). Maildir is one exception to this rule. Maildir != queue... -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
Hi, Yes that is correct. The design of qmail forces a connection for each email message. Changing that behaviour would require massive patching. Sincerely, Jason - Original Message - From: "Tomasz Papszun" <[EMAIL PROTECTED]> To: "Rich Puhek" <[EMAIL PROTECTED]> Cc: "Jason Lim" <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]> Sent: Friday, June 08, 2001 6:04 PM Subject: Re: Finding the Bottleneck > On Thu, 07 Jun 2001 at 22:47:09 -0500, Rich Puhek wrote: > [...] > > Also, there are probably some optimizations you can do for queue sort > > order. I'm most familiar with Sendmail, not qmail, so I don't know the > > exact settings, but try to process the queue according to recipient > > domain. That way, you gain some advantages with holding SMTP connections > > open to a server, rather than closing and reopening a session, etc. > > > > --Rich > > If my memory serves me well, qmail opens a new session for each message, > even if this message is to be delivered to the same server. > I may be wrong though. > > -- > Tomasz Papszun SysAdm @ TP S.A. Lodz, Poland | And it's only > [EMAIL PROTECTED] http://www.lodz.tpsa.pl/ | ones and zeros. > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > > -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
On Thu, 07 Jun 2001 at 22:47:09 -0500, Rich Puhek wrote: [...] > Also, there are probably some optimizations you can do for queue sort > order. I'm most familiar with Sendmail, not qmail, so I don't know the > exact settings, but try to process the queue according to recipient > domain. That way, you gain some advantages with holding SMTP connections > open to a server, rather than closing and reopening a session, etc. > > --Rich If my memory serves me well, qmail opens a new session for each message, even if this message is to be delivered to the same server. I may be wrong though. -- Tomasz Papszun SysAdm @ TP S.A. Lodz, Poland | And it's only [EMAIL PROTECTED] http://www.lodz.tpsa.pl/ | ones and zeros. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
On Friday 08 June 2001 10:49, Brian May wrote: > Russell> If the NFS server has the same disk system then you will > Russell> only make things worse. Anything you could do to give > Russell> the NFS server better IO performance could more > Russell> productively be done to the main server. Also many of > Russell> the common Unix mail server programs are specifically > Russell> designed to have the queue on a local file system with > Russell> standard Unix semantics (Inode numbers etc). Qmail is > Russell> one mail server that I have found to not work with it's > Russell> queue on NFS, I haven't seriously tried any others. I've > Russell> CC'd Brian May because he does a lot more NFS stuff with > Russell> Debian than most people and he may be able to advise you. > > It depends on how much you want to share via NFS, what speed/type > network you have, etc. Jason has previously stated that he only wants a single machine and wants things to be as cheap as possible. > The idea of putting any queue on NFS is generally discouraged (due to > file locking issues). Maildir is one exception to this rule. Maildir != queue... -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
Maybe a local caching nameserver will help here as well. (Just a quick though.) Cheers, Marcel Rich Puhek <[EMAIL PROTECTED]> 7 Jun 2001, at 22:47: > By the way, > > In addition to checking the disk usage, memory, and the other > suggestions that have come up on the list, have you looked at DNS? > Quite often you'll find that DNS lookups are severely limiting the > performance of something like a mailing list. Make sure that the mail > server itself isn't running a DNS server. Make sure you've got one or > two DNS servers in close proximity to the mail server. Make sure that > the DNS server process isn't swapping on the DNS servers (for the kind > of traffic you're pushing through, you may need a pentium class > machine with 128 MB of RAM as your DNS server. Also, if possible, I > like to have the DNS server I'm querying kept free from being the > authoratative server for any domains (not always practical in a real > life situation, I know). > > Also, there are probably some optimizations you can do for queue sort > order. I'm most familiar with Sendmail, not qmail, so I don't know the > exact settings, but try to process the queue according to recipient > domain. That way, you gain some advantages with holding SMTP > connections open to a server, rather than closing and reopening a > session, etc. > > --Rich > > Jason Lim wrote: > > > > Hi all, > > > > I was wondering if there is a way to find out what/where the > > bottleneck of a large mail server is. > > > > A client is running a huge mail server that we set up for them > > (running qmail), but performance seems to be limited somewhere. > > Qmail has already been optimized as far as it can go (big-todo > > patch, large concurrency patch, etc.). > > > > We're thinking one of the Hard Disks may be the main bottleneck (the > > mail queue is already on a seperate disk on a seperate IDE channel > > from other disks). Is there any way to find out how "utilized" the > > IDE channel/hard disk is, or how hard it is going? Seems that right > > now the only way we really know is by looking at the light on the > > server case (how technical ;-) ). Must be a better way... > > > > The bottleneck wouldn't be bandwidth... it is definately with the > > server. Perhaps the CPU or kernel is the bottleneck (load average: > > 4.84, 3.94, 3.88, going up to 5 or 6 during heavy mailing)? Is that > > normal for a large mail server? We haven't run such a large mail > > server before (anywhere between 500k to 1M per day so far, > > increasing each day), so ANY tips and pointers would be greatly > > appreciated. We've already been playing around with hdparm to see if > > we can tweak the disks, but doesn't seem to help much. Maybe some > > cache settings we can fiddle with? Maybe the mail queue disk could > > use a different file cache setting (each email being from 1K to 10K > > on average)? > > > > Thanks in advance! > > > > Sincerely, > > Jason -- .-. / ´ ´ ` ` ` //_\\ || / ` || |\ .´ / \`.__,´ \ ` Debian / GNU Linux ` . -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
> "Russell" == Russell Coker <[EMAIL PROTECTED]> writes: Russell> If the NFS server has the same disk system then you will Russell> only make things worse. Anything you could do to give Russell> the NFS server better IO performance could more Russell> productively be done to the main server. Also many of Russell> the common Unix mail server programs are specifically Russell> designed to have the queue on a local file system with Russell> standard Unix semantics (Inode numbers etc). Qmail is Russell> one mail server that I have found to not work with it's Russell> queue on NFS, I haven't seriously tried any others. I've Russell> CC'd Brian May because he does a lot more NFS stuff with Russell> Debian than most people and he may be able to advise you. It depends on how much you want to share via NFS, what speed/type network you have, etc. The idea of putting any queue on NFS is generally discouraged (due to file locking issues). Maildir is one exception to this rule. -- Brian May <[EMAIL PROTECTED]> -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
By the way, In addition to checking the disk usage, memory, and the other suggestions that have come up on the list, have you looked at DNS? Quite often you'll find that DNS lookups are severely limiting the performance of something like a mailing list. Make sure that the mail server itself isn't running a DNS server. Make sure you've got one or two DNS servers in close proximity to the mail server. Make sure that the DNS server process isn't swapping on the DNS servers (for the kind of traffic you're pushing through, you may need a pentium class machine with 128 MB of RAM as your DNS server. Also, if possible, I like to have the DNS server I'm querying kept free from being the authoratative server for any domains (not always practical in a real life situation, I know). Also, there are probably some optimizations you can do for queue sort order. I'm most familiar with Sendmail, not qmail, so I don't know the exact settings, but try to process the queue according to recipient domain. That way, you gain some advantages with holding SMTP connections open to a server, rather than closing and reopening a session, etc. --Rich Jason Lim wrote: > > Hi all, > > I was wondering if there is a way to find out what/where the bottleneck of > a large mail server is. > > A client is running a huge mail server that we set up for them (running > qmail), but performance seems to be limited somewhere. Qmail has already > been optimized as far as it can go (big-todo patch, large concurrency > patch, etc.). > > We're thinking one of the Hard Disks may be the main bottleneck (the mail > queue is already on a seperate disk on a seperate IDE channel from other > disks). Is there any way to find out how "utilized" the IDE channel/hard > disk is, or how hard it is going? Seems that right now the only way we > really know is by looking at the light on the server case (how technical > ;-) ). Must be a better way... > > The bottleneck wouldn't be bandwidth... it is definately with the server. > Perhaps the CPU or kernel is the bottleneck (load average: 4.84, 3.94, > 3.88, going up to 5 or 6 during heavy mailing)? Is that normal for a large > mail server? We haven't run such a large mail server before (anywhere > between 500k to 1M per day so far, increasing each day), so ANY tips and > pointers would be greatly appreciated. We've already been playing around > with hdparm to see if we can tweak the disks, but doesn't seem to help > much. Maybe some cache settings we can fiddle with? Maybe the mail queue > disk could use a different file cache setting (each email being from 1K to > 10K on average)? > > Thanks in advance! > > Sincerely, > Jason > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] -- _ Rich Puhek ETN Systems Inc. _
Re: Finding the Bottleneck
On Friday 08 June 2001 00:05, Jason Lim wrote: > Thanks for your detailed reply. > > As reliability is not of great importance (only the mail queue will be > there and no critical system files), I'd go for speed and cheap price. > The client doesn't have the huge wads of cash for the optimal system > with scsi drives and 64M cache raid card :-/ So I guess if it comes > down to the crunch, speed and cheap price is it. OK. Software RAID on ATA drives is what you will be using then. > I'll also scratch the NFS idea since qmail wouldn't work well on it, Qmail won't work at all on NFS... > Okay... the server right now is an AMD k6-2 500mhz 128M with 2 15G IBM > drives. At this moment, it is really having trouble processing the mail > queue. > sh-2.05# qmail-qstat > messages in queue: 297121 > messages in queue but not yet preprocessed: 72333 > > I checked the log and yesterday it sent about 1 million emails. Do you > think that the above hardware configuration is performing at it's > realistic limit? That an average of over 11 messages per second. Not bad with such tiny hardware. The first thing to do is to buy some decent hard drives. Get new IBM drives of 40G or more capacity and use the first 15M of them (it reduces seek times and also the first part has a higher transfer rate). Also even the slowest part of a 45G drive will be twice as fast as the fastest part of a 15G drive. Put the spool and queue on a software RAID-1 over two 45G drives and put the rest on a software RAID-1 over the two old 15G drives. Then the system should be considerably more than twice as fast. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page
Finding the Bottleneck
Hi all, I was wondering if there is a way to find out what/where the bottleneck of a large mail server is. A client is running a huge mail server that we set up for them (running qmail), but performance seems to be limited somewhere. Qmail has already been optimized as far as it can go (big-todo patch, large concurrency patch, etc.). We're thinking one of the Hard Disks may be the main bottleneck (the mail queue is already on a seperate disk on a seperate IDE channel from other disks). Is there any way to find out how "utilized" the IDE channel/hard disk is, or how hard it is going? Seems that right now the only way we really know is by looking at the light on the server case (how technical ;-) ). Must be a better way... The bottleneck wouldn't be bandwidth... it is definately with the server. Perhaps the CPU or kernel is the bottleneck (load average: 4.84, 3.94, 3.88, going up to 5 or 6 during heavy mailing)? Is that normal for a large mail server? We haven't run such a large mail server before (anywhere between 500k to 1M per day so far, increasing each day), so ANY tips and pointers would be greatly appreciated. We've already been playing around with hdparm to see if we can tweak the disks, but doesn't seem to help much. Maybe some cache settings we can fiddle with? Maybe the mail queue disk could use a different file cache setting (each email being from 1K to 10K on average)? Thanks in advance! Sincerely, Jason
Re: Finding the Bottleneck
By the way, In addition to checking the disk usage, memory, and the other suggestions that have come up on the list, have you looked at DNS? Quite often you'll find that DNS lookups are severely limiting the performance of something like a mailing list. Make sure that the mail server itself isn't running a DNS server. Make sure you've got one or two DNS servers in close proximity to the mail server. Make sure that the DNS server process isn't swapping on the DNS servers (for the kind of traffic you're pushing through, you may need a pentium class machine with 128 MB of RAM as your DNS server. Also, if possible, I like to have the DNS server I'm querying kept free from being the authoratative server for any domains (not always practical in a real life situation, I know). Also, there are probably some optimizations you can do for queue sort order. I'm most familiar with Sendmail, not qmail, so I don't know the exact settings, but try to process the queue according to recipient domain. That way, you gain some advantages with holding SMTP connections open to a server, rather than closing and reopening a session, etc. --Rich Jason Lim wrote: > > Hi all, > > I was wondering if there is a way to find out what/where the bottleneck of > a large mail server is. > > A client is running a huge mail server that we set up for them (running > qmail), but performance seems to be limited somewhere. Qmail has already > been optimized as far as it can go (big-todo patch, large concurrency > patch, etc.). > > We're thinking one of the Hard Disks may be the main bottleneck (the mail > queue is already on a seperate disk on a seperate IDE channel from other > disks). Is there any way to find out how "utilized" the IDE channel/hard > disk is, or how hard it is going? Seems that right now the only way we > really know is by looking at the light on the server case (how technical > ;-) ). Must be a better way... > > The bottleneck wouldn't be bandwidth... it is definately with the server. > Perhaps the CPU or kernel is the bottleneck (load average: 4.84, 3.94, > 3.88, going up to 5 or 6 during heavy mailing)? Is that normal for a large > mail server? We haven't run such a large mail server before (anywhere > between 500k to 1M per day so far, increasing each day), so ANY tips and > pointers would be greatly appreciated. We've already been playing around > with hdparm to see if we can tweak the disks, but doesn't seem to help > much. Maybe some cache settings we can fiddle with? Maybe the mail queue > disk could use a different file cache setting (each email being from 1K to > 10K on average)? > > Thanks in advance! > > Sincerely, > Jason > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] -- _ Rich Puhek ETN Systems Inc. _ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
On Friday 08 June 2001 00:05, Jason Lim wrote: > Thanks for your detailed reply. > > As reliability is not of great importance (only the mail queue will be > there and no critical system files), I'd go for speed and cheap price. > The client doesn't have the huge wads of cash for the optimal system > with scsi drives and 64M cache raid card :-/ So I guess if it comes > down to the crunch, speed and cheap price is it. OK. Software RAID on ATA drives is what you will be using then. > I'll also scratch the NFS idea since qmail wouldn't work well on it, Qmail won't work at all on NFS... > Okay... the server right now is an AMD k6-2 500mhz 128M with 2 15G IBM > drives. At this moment, it is really having trouble processing the mail > queue. > sh-2.05# qmail-qstat > messages in queue: 297121 > messages in queue but not yet preprocessed: 72333 > > I checked the log and yesterday it sent about 1 million emails. Do you > think that the above hardware configuration is performing at it's > realistic limit? That an average of over 11 messages per second. Not bad with such tiny hardware. The first thing to do is to buy some decent hard drives. Get new IBM drives of 40G or more capacity and use the first 15M of them (it reduces seek times and also the first part has a higher transfer rate). Also even the slowest part of a 45G drive will be twice as fast as the fastest part of a 15G drive. Put the spool and queue on a software RAID-1 over two 45G drives and put the rest on a software RAID-1 over the two old 15G drives. Then the system should be considerably more than twice as fast. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
Hi Bertrand, Thanks for your insightful email! Just so you know, this server is an: AMD K6-2 500Mhz, 128M-133Mhz, 2 UDMA100 drives (IBM), 10M bandwidth. The server runs Apache, Qmail, vpopmail (for pop3). The webserver is not primary (doesn't have to have the fastest response time), as this is mainly for the mailing lists. The 2 hard disks are on 2 different IDE channels, as putting both disks on the same cable would drastically reduce performance of both disks. The way it is organized is that the mail spool/queue is on the 2nd disk, while the OS and programs are on disk 1. Logging is also performed on disk 1, so that writing to the mail log won't interfere with the mail queue (as they commonly both occur simultaneously). >From MY understanding, the "load average" shows how many programs are running, and not really how "stressed" the CPU is. I'm not sure exactly sure how this works (please correct me if i'm wrong) but 1 program taking 80% CPU might have load average of 2, while 100 programs taking 0.5% each would take 50% CPU and have load average of 8. Is that correct thinking? The reason I'm saying that is because qmail spawns a program called "qmail-remote" for EACH email to be sent. So if 200 concurrent emails are being sent at one time, then 200 qmail-remotes are created. Granted... each of these takes up tiny amounts of ram and CPU time, but I suspect (if the above statements are correct) that the load average would be artificially inflated because of it. We don't use NFS on this server. NFS on linux, as you said, is pretty crummy and should be avoided if possible. We simply put the mail queue on a seperate hard disk. pop3 load is extremely minimal. Its mainly an outgoing mail server (mail list server). People essentially use the website to send mail to the mailing list, so load of the pop3 server won't be an issue. About the HK job market, do you have any official qualifications (eg. a degree, diploma, cert., etc.)? In HK bosses like to see that... even more than in some other countries like Australia (not sure bout US). BTW. was your mother headmistress of St. Paul before? Sincerely, Jason - Original Message - From: "schemerz" <[EMAIL PROTECTED]> To: "Jason Lim" <[EMAIL PROTECTED]> Sent: Wednesday, June 06, 2001 3:57 PM Subject: Re: Finding the Bottleneck On Wed, Jun 06, 2001 at 11:53:22AM +0800, Jason Lim wrote: > Hi all, > > I was wondering if there is a way to find out what/where the bottleneck of > a large mail server is. > > A client is running a huge mail server that we set up for them (running > qmail), but performance seems to be limited somewhere. Qmail has already > been optimized as far as it can go (big-todo patch, large concurrency > patch, etc.). > > We're thinking one of the Hard Disks may be the main bottleneck (the mail > queue is already on a seperate disk on a seperate IDE channel from other > disks). Is there any way to find out how "utilized" the IDE channel/hard > disk is, or how hard it is going? Seems that right now the only way we > really know is by looking at the light on the server case (how technical > ;-) ). Must be a better way... > > The bottleneck wouldn't be bandwidth... it is definately with the server. > Perhaps the CPU or kernel is the bottleneck (load average: 4.84, 3.94, > 3.88, going up to 5 or 6 during heavy mailing)? Is that normal for a large > mail server? We haven't run such a large mail server before (anywhere > between 500k to 1M per day so far, increasing each day), so ANY tips and > pointers would be greatly appreciated. We've already been playing around > with hdparm to see if we can tweak the disks, but doesn't seem to help > much. Maybe some cache settings we can fiddle with? Maybe the mail queue > disk could use a different file cache setting (each email being from 1K to > 10K on average)? > > Thanks in advance! > > Sincerely, > Jason > Jason, I am a lurker on the list. I don't run linux anymore but recently a friend of mine encountered a similar load problem. Granted, he was running sendmail, but his main bottleneck wasn't the mta at all. I will explain his situation and see if you will find any similarites with yours... The discussion pertains to the mailserver at sunflower.com. When my friend took over the previous admin left the place in a mess. One of the machines he inherited was a dual 400 mhz penta 2 with about 256 megs of ram. It would be quite adequate for serving about 5000 accounts except... 1) The server was running on a box using the promise IDE raid controllers. IDE for a small server would work fine, but this box was getting hit alot. 2) This server was also the pop server. One of the reasons of design to run pop and sendmail on the same box was becau
Re: Finding the Bottleneck
On Wednesday 06 June 2001 10:51, Jason Lim wrote: > Just so you know, this server is an: > AMD K6-2 500Mhz, 128M-133Mhz, 2 UDMA100 drives (IBM), 10M bandwidth. How much swap is being used? If you have any serious amount of mail being delivered then having a mere 128M of RAM will seriously hurt performance! RAM is also cheap and easy to upgrade... > mainly for the mailing lists. The 2 hard disks are on 2 different IDE > channels, as putting both disks on the same cable would drastically reduce > performance of both disks. In my tests so far I have not been able to show drastic performance difference. I have shown about a 20% performance benefit for using separate cables... > The way it is organized is that the mail spool/queue is on the 2nd disk, > while the OS and programs are on disk 1. Logging is also performed on disk > 1, so that writing to the mail log won't interfere with the mail queue (as > they commonly both occur simultaneously). Where is swap? Take note of the following paragraph from syslog.conf(5): You may prefix each entry with the minus ``-'' sign to omit syncing the file after every logging. Note that you might lose information if the system crashes right behind a write attempt. Nevertheless this might give you back some performance, especially if you run programs that use logging in a very verbose manner. Do that for all logs apart from kern.log! Then syslogd will hardly use any disk access. > From MY understanding, the "load average" shows how many programs are > running, and not really how "stressed" the CPU is. I'm not sure exactly > sure how this works (please correct me if i'm wrong) but 1 program taking > 80% CPU might have load average of 2, while 100 programs taking 0.5% each > would take 50% CPU and have load average of 8. Is that correct thinking? Not. 1 program taking up all CPU time will give a load average of 1.00. 1 program being blocked on disk IO (EG reading from a floppy disk) will give a load average of 1.00. Two programs blocked on disk IO to different disks and a third program that's doing a lot of CPU usage will result in load average of 3.00 while the machine is running as efficiently as it can. Load average isn't a very good way of measuring system use! > We don't use NFS on this server. NFS on linux, as you said, is pretty > crummy and should be avoided if possible. We simply put the mail queue on > a seperate hard disk. Actually if you have the latest patches then NFS should be quite solid. Now firstly the OS and the syslog will not use the disk much at all if you have enough RAM that the machine doesn't swap and has some spare memory for caching. Boost the machine to 256M. Don't bother with DDR RAM as it won't gain you anything, get 384 or 512M if you can afford it. Next the most important thing for local mail delivery is to have the queue on a separate disk to the spool. Queue and spool writes are independant and the data is immidiately sync'd. Having them on separate disks can provide serious performance benefits. Also if your data is at all important to you then you should be using RAID. Software RAID-1 in the 2.4.x kernels and with the patch for 2.2.x kernels is very solid. I suggest getting 4 drives and running two RAID-1 sets, one for the OS and queue, the other for the spool. RAID-1 will improve read speed as the system will be able to execute two read requests from the RAID-1 at the same time. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page
Re: Finding the Bottleneck
Hi Russell, Here is the result of "top": 05:51:18 up 5 days, 22:38, 1 user, load average: 6.60, 7.40, 6.51 119 processes: 106 sleeping, 11 running, 2 zombie, 0 stopped CPU states: 16.4% user, 18.3% system, 0.0% nice, 65.3% idle Mem:128236K total, 124348K used, 3888K free,72392K buffers Swap: 289160K total,0K used, 289160K free, 9356K cached And of "qmail-qstat": sh-2.05# qmail-qstat messages in queue: 108903 messages in queue but not yet preprocessed: 19537 Swap is on Disk 1, because mail queue/spool is on Disk 2. I also already added the "-" in front of most entries except the emergency or critical ones (if I didn't do it, the load was way higher just writing the log files). Concerning the mail queue and spool being on the same disk, the reason is that there is virtually no emails incoming, 99.999% outgoing. About running software raid... I've heard that the CPU usage is increased dramatically if you use any form of software raid. Is that true? Actually... i doubt the customer would be willing to pay us to implement this for him on a hardware level. Good raid cards with good amounts of ram don't come cheap last time I checked... :-/ Sincerely, Jason - Original Message - From: "Russell Coker" <[EMAIL PROTECTED]> To: "Jason Lim" <[EMAIL PROTECTED]>; Sent: Wednesday, June 06, 2001 8:05 PM Subject: Re: Finding the Bottleneck On Wednesday 06 June 2001 10:51, Jason Lim wrote: > Just so you know, this server is an: > AMD K6-2 500Mhz, 128M-133Mhz, 2 UDMA100 drives (IBM), 10M bandwidth. How much swap is being used? If you have any serious amount of mail being delivered then having a mere 128M of RAM will seriously hurt performance! RAM is also cheap and easy to upgrade... > mainly for the mailing lists. The 2 hard disks are on 2 different IDE > channels, as putting both disks on the same cable would drastically reduce > performance of both disks. In my tests so far I have not been able to show drastic performance difference. I have shown about a 20% performance benefit for using separate cables... > The way it is organized is that the mail spool/queue is on the 2nd disk, > while the OS and programs are on disk 1. Logging is also performed on disk > 1, so that writing to the mail log won't interfere with the mail queue (as > they commonly both occur simultaneously). Where is swap? Take note of the following paragraph from syslog.conf(5): You may prefix each entry with the minus ``-'' sign to omit syncing the file after every logging. Note that you might lose information if the system crashes right behind a write attempt. Nevertheless this might give you back some performance, especially if you run programs that use logging in a very verbose manner. Do that for all logs apart from kern.log! Then syslogd will hardly use any disk access. > From MY understanding, the "load average" shows how many programs are > running, and not really how "stressed" the CPU is. I'm not sure exactly > sure how this works (please correct me if i'm wrong) but 1 program taking > 80% CPU might have load average of 2, while 100 programs taking 0.5% each > would take 50% CPU and have load average of 8. Is that correct thinking? Not. 1 program taking up all CPU time will give a load average of 1.00. 1 program being blocked on disk IO (EG reading from a floppy disk) will give a load average of 1.00. Two programs blocked on disk IO to different disks and a third program that's doing a lot of CPU usage will result in load average of 3.00 while the machine is running as efficiently as it can. Load average isn't a very good way of measuring system use! > We don't use NFS on this server. NFS on linux, as you said, is pretty > crummy and should be avoided if possible. We simply put the mail queue on > a seperate hard disk. Actually if you have the latest patches then NFS should be quite solid. Now firstly the OS and the syslog will not use the disk much at all if you have enough RAM that the machine doesn't swap and has some spare memory for caching. Boost the machine to 256M. Don't bother with DDR RAM as it won't gain you anything, get 384 or 512M if you can afford it. Next the most important thing for local mail delivery is to have the queue on a separate disk to the spool. Queue and spool writes are independant and the data is immidiately sync'd. Having them on separate disks can provide serious performance benefits. Also if your data is at all important to you then you should be using RAID. Software RAID-1 in the 2.4.x kernels and with the patch for 2.2.x kernels is very solid. I suggest getting 4 drives and running two RAID-1 sets, one for the OS and queue, the other for the spool. RAID-1 will imp
Re: Finding the Bottleneck
On Wed, 6 Jun 2001, Jason Lim wrote: > I was wondering if there is a way to find out what/where the bottleneck of > a large mail server is. Look at vmstat. vmstat can tell you about number of processeses waiting for run time, amount of memory swapped to disk, blocks per second sent (and received) from disks, number of interrupts and context switches per second, CPU usage, and more. The difficult part of using this is to have a baseline to compare it with. For example, this is from a lightly-loaded system: procs memoryswap io system cpu r b w swpd free buff cache si sobibo incs us sy id 0 0 0 3528 1804 1508 13868 0 0 0 1 104 7 1 1 98 0 0 0 3528 1804 1508 13868 0 0 0 0 104 7 1 1 98 BSD systems have detailed systat, vmstat, and fstat utilities that are useful for tracking down bottlenecks. (It sure seems like Linux would have similar tools, but I don't know where.) Jeremy C. Reed ... ISP-FAQ.com -- find answers to your questions http://www.isp-faq.com/
Re: Finding the Bottleneck
On Thursday 07 June 2001 00:11, Jason Lim wrote: > 05:51:18 up 5 days, 22:38, 1 user, load average: 6.60, 7.40, 6.51 > 119 processes: 106 sleeping, 11 running, 2 zombie, 0 stopped > CPU states: 16.4% user, 18.3% system, 0.0% nice, 65.3% idle > Mem:128236K total, 124348K used, 3888K free,72392K buffers > Swap: 289160K total,0K used, 289160K free, 9356K cached > > And of "qmail-qstat": > sh-2.05# qmail-qstat > messages in queue: 108903 > messages in queue but not yet preprocessed: 19537 > > Swap is on Disk 1, because mail queue/spool is on Disk 2. > > I also already added the "-" in front of most entries except the emergency > or critical ones (if I didn't do it, the load was way higher just writing > the log files). > > Concerning the mail queue and spool being on the same disk, the reason is > that there is virtually no emails incoming, 99.999% outgoing. > > About running software raid... I've heard that the CPU usage is increased > dramatically if you use any form of software raid. Is that true? > Actually... i doubt the customer would be willing to pay us to implement > this for him on a hardware level. Good raid cards with good amounts of ram > don't come cheap last time I checked... :-/ CPU usage isn't increased that much, and as you're only using 30% of the CPU time it shouldn't be problem if you use more anyway... Disk access is the main bottleneck, anything that alleviates it is something you want to do. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page
Re: Finding the Bottleneck
Hi, I found vmstat on the server, but could not find your other "systat" or "fstat". I think this is exactly what I need... especially fstat. Does anyone know a similar program for linux? Sincerely, Jason - Original Message - From: "Jeremy C. Reed" <[EMAIL PROTECTED]> To: "Jason Lim" <[EMAIL PROTECTED]> Cc: Sent: Thursday, June 07, 2001 6:43 AM Subject: Re: Finding the Bottleneck > On Wed, 6 Jun 2001, Jason Lim wrote: > > > I was wondering if there is a way to find out what/where the bottleneck of > > a large mail server is. > > Look at vmstat. > > vmstat can tell you about number of processeses waiting for run time, > amount of memory swapped to disk, blocks per second sent (and > received) from disks, number of interrupts and context switches per > second, CPU usage, and more. > > The difficult part of using this is to have a baseline to compare it > with. For example, this is from a lightly-loaded system: > >procs memoryswap io system > cpu > r b w swpd free buff cache si sobibo incs us > sy id > 0 0 0 3528 1804 1508 13868 0 0 0 1 104 7 1 > 1 98 > 0 0 0 3528 1804 1508 13868 0 0 0 0 104 7 1 > 1 98 > > BSD systems have detailed systat, vmstat, and fstat utilities that are > useful for tracking down bottlenecks. (It sure seems like Linux would have > similar tools, but I don't know where.) > > Jeremy C. Reed > ... > ISP-FAQ.com -- find answers to your questions > http://www.isp-faq.com/ > >
Re: Finding the Bottleneck
Hi, I agree with you... it seems more and more likely that the Disks are the limiting factor here. I guess the next big thing to do would be to run some form of Raid (software or hardware) for the mail queue. Does anyone know of a cheap but adequate raid hardware solution? The one's I've seen seem to cost quite a bit. I know that the common cheap ATA100 Raid cards available now (using the Highpoint HPT370) don't work properly on Linux beacuse of the bad driver support. Anyone know of an alternative? Actually... do you think setting up a seperate box (connected via NFS) PURELY for mail queue processing would help at all? Or would the bottleneck then be shifted to NFS? Sincerely, Jason - Original Message - From: "Russell Coker" <[EMAIL PROTECTED]> To: "Jason Lim" <[EMAIL PROTECTED]>; Sent: Thursday, June 07, 2001 3:21 PM Subject: Re: Finding the Bottleneck On Thursday 07 June 2001 00:11, Jason Lim wrote: > 05:51:18 up 5 days, 22:38, 1 user, load average: 6.60, 7.40, 6.51 > 119 processes: 106 sleeping, 11 running, 2 zombie, 0 stopped > CPU states: 16.4% user, 18.3% system, 0.0% nice, 65.3% idle > Mem:128236K total, 124348K used, 3888K free,72392K buffers > Swap: 289160K total,0K used, 289160K free, 9356K cached > > And of "qmail-qstat": > sh-2.05# qmail-qstat > messages in queue: 108903 > messages in queue but not yet preprocessed: 19537 > > Swap is on Disk 1, because mail queue/spool is on Disk 2. > > I also already added the "-" in front of most entries except the emergency > or critical ones (if I didn't do it, the load was way higher just writing > the log files). > > Concerning the mail queue and spool being on the same disk, the reason is > that there is virtually no emails incoming, 99.999% outgoing. > > About running software raid... I've heard that the CPU usage is increased > dramatically if you use any form of software raid. Is that true? > Actually... i doubt the customer would be willing to pay us to implement > this for him on a hardware level. Good raid cards with good amounts of ram > don't come cheap last time I checked... :-/ CPU usage isn't increased that much, and as you're only using 30% of the CPU time it shouldn't be problem if you use more anyway... Disk access is the main bottleneck, anything that alleviates it is something you want to do. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
Hi, Thanks for your detailed reply. As reliability is not of great importance (only the mail queue will be there and no critical system files), I'd go for speed and cheap price. The client doesn't have the huge wads of cash for the optimal system with scsi drives and 64M cache raid card :-/ So I guess if it comes down to the crunch, speed and cheap price is it. I'll also scratch the NFS idea since qmail wouldn't work well on it, and as you said, there wouldn't be much benefit if the other server has the same problems as well. Okay... the server right now is an AMD k6-2 500mhz 128M with 2 15G IBM drives. At this moment, it is really having trouble processing the mail queue. sh-2.05# qmail-qstat messages in queue: 297121 messages in queue but not yet preprocessed: 72333 I checked the log and yesterday it sent about 1 million emails. Do you think that the above hardware configuration is performing at it's realistic limit? Sincerely, Jason - Original Message - From: "Russell Coker" <[EMAIL PROTECTED]> To: "Jason Lim" <[EMAIL PROTECTED]>; Cc: <[EMAIL PROTECTED]> Sent: Friday, June 08, 2001 3:42 AM Subject: Re: Finding the Bottleneck On Thursday 07 June 2001 20:14, Jason Lim wrote: > I agree with you... it seems more and more likely that the Disks are > the limiting factor here. > > I guess the next big thing to do would be to run some form of Raid > (software or hardware) for the mail queue. > > Does anyone know of a cheap but adequate raid hardware solution? The > one's I've seen seem to cost quite a bit. I know that the common cheap > ATA100 Raid cards available now (using the Highpoint HPT370) don't work > properly on Linux beacuse of the bad driver support. Anyone know of an > alternative? For RAID hardware there are three criteria you desire, reliability, low price, and speed. You can have at most two of them. There are a number of cheap hardware RAID solutions out there which would be quite OK for home use, but not for a server of the type you are considering. If you had several redundant servers with the same data then one of the cheap hardware RAID solutions might do well though. > Actually... do you think setting up a seperate box (connected via NFS) > PURELY for mail queue processing would help at all? Or would the > bottleneck then be shifted to NFS? If the NFS server has the same disk system then you will only make things worse. Anything you could do to give the NFS server better IO performance could more productively be done to the main server. Also many of the common Unix mail server programs are specifically designed to have the queue on a local file system with standard Unix semantics (Inode numbers etc). Qmail is one mail server that I have found to not work with it's queue on NFS, I haven't seriously tried any others. I've CC'd Brian May because he does a lot more NFS stuff with Debian than most people and he may be able to advise you. I suggest just getting 4 IDE disks and putting everything on a software RAID-10 apart from /boot (which must be RAID-1 for the boot loader to work). It'll get the most bang for the buck! -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page
Re: Finding the Bottleneck
On Thursday 07 June 2001 20:14, Jason Lim wrote: > I agree with you... it seems more and more likely that the Disks are > the limiting factor here. > > I guess the next big thing to do would be to run some form of Raid > (software or hardware) for the mail queue. > > Does anyone know of a cheap but adequate raid hardware solution? The > one's I've seen seem to cost quite a bit. I know that the common cheap > ATA100 Raid cards available now (using the Highpoint HPT370) don't work > properly on Linux beacuse of the bad driver support. Anyone know of an > alternative? For RAID hardware there are three criteria you desire, reliability, low price, and speed. You can have at most two of them. There are a number of cheap hardware RAID solutions out there which would be quite OK for home use, but not for a server of the type you are considering. If you had several redundant servers with the same data then one of the cheap hardware RAID solutions might do well though. > Actually... do you think setting up a seperate box (connected via NFS) > PURELY for mail queue processing would help at all? Or would the > bottleneck then be shifted to NFS? If the NFS server has the same disk system then you will only make things worse. Anything you could do to give the NFS server better IO performance could more productively be done to the main server. Also many of the common Unix mail server programs are specifically designed to have the queue on a local file system with standard Unix semantics (Inode numbers etc). Qmail is one mail server that I have found to not work with it's queue on NFS, I haven't seriously tried any others. I've CC'd Brian May because he does a lot more NFS stuff with Debian than most people and he may be able to advise you. I suggest just getting 4 IDE disks and putting everything on a software RAID-10 apart from /boot (which must be RAID-1 for the boot loader to work). It'll get the most bang for the buck! -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Finding the Bottleneck
On Thursday 07 June 2001 20:14, Jason Lim wrote: > I agree with you... it seems more and more likely that the Disks are > the limiting factor here. > > I guess the next big thing to do would be to run some form of Raid > (software or hardware) for the mail queue. > > Does anyone know of a cheap but adequate raid hardware solution? The > one's I've seen seem to cost quite a bit. I know that the common cheap > ATA100 Raid cards available now (using the Highpoint HPT370) don't work > properly on Linux beacuse of the bad driver support. Anyone know of an > alternative? For RAID hardware there are three criteria you desire, reliability, low price, and speed. You can have at most two of them. There are a number of cheap hardware RAID solutions out there which would be quite OK for home use, but not for a server of the type you are considering. If you had several redundant servers with the same data then one of the cheap hardware RAID solutions might do well though. > Actually... do you think setting up a seperate box (connected via NFS) > PURELY for mail queue processing would help at all? Or would the > bottleneck then be shifted to NFS? If the NFS server has the same disk system then you will only make things worse. Anything you could do to give the NFS server better IO performance could more productively be done to the main server. Also many of the common Unix mail server programs are specifically designed to have the queue on a local file system with standard Unix semantics (Inode numbers etc). Qmail is one mail server that I have found to not work with it's queue on NFS, I haven't seriously tried any others. I've CC'd Brian May because he does a lot more NFS stuff with Debian than most people and he may be able to advise you. I suggest just getting 4 IDE disks and putting everything on a software RAID-10 apart from /boot (which must be RAID-1 for the boot loader to work). It'll get the most bang for the buck! -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page
Re: Finding the Bottleneck
Hi, Thanks for your detailed reply. As reliability is not of great importance (only the mail queue will be there and no critical system files), I'd go for speed and cheap price. The client doesn't have the huge wads of cash for the optimal system with scsi drives and 64M cache raid card :-/ So I guess if it comes down to the crunch, speed and cheap price is it. I'll also scratch the NFS idea since qmail wouldn't work well on it, and as you said, there wouldn't be much benefit if the other server has the same problems as well. Okay... the server right now is an AMD k6-2 500mhz 128M with 2 15G IBM drives. At this moment, it is really having trouble processing the mail queue. sh-2.05# qmail-qstat messages in queue: 297121 messages in queue but not yet preprocessed: 72333 I checked the log and yesterday it sent about 1 million emails. Do you think that the above hardware configuration is performing at it's realistic limit? Sincerely, Jason - Original Message - From: "Russell Coker" <[EMAIL PROTECTED]> To: "Jason Lim" <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Friday, June 08, 2001 3:42 AM Subject: Re: Finding the Bottleneck On Thursday 07 June 2001 20:14, Jason Lim wrote: > I agree with you... it seems more and more likely that the Disks are > the limiting factor here. > > I guess the next big thing to do would be to run some form of Raid > (software or hardware) for the mail queue. > > Does anyone know of a cheap but adequate raid hardware solution? The > one's I've seen seem to cost quite a bit. I know that the common cheap > ATA100 Raid cards available now (using the Highpoint HPT370) don't work > properly on Linux beacuse of the bad driver support. Anyone know of an > alternative? For RAID hardware there are three criteria you desire, reliability, low price, and speed. You can have at most two of them. There are a number of cheap hardware RAID solutions out there which would be quite OK for home use, but not for a server of the type you are considering. If you had several redundant servers with the same data then one of the cheap hardware RAID solutions might do well though. > Actually... do you think setting up a seperate box (connected via NFS) > PURELY for mail queue processing would help at all? Or would the > bottleneck then be shifted to NFS? If the NFS server has the same disk system then you will only make things worse. Anything you could do to give the NFS server better IO performance could more productively be done to the main server. Also many of the common Unix mail server programs are specifically designed to have the queue on a local file system with standard Unix semantics (Inode numbers etc). Qmail is one mail server that I have found to not work with it's queue on NFS, I haven't seriously tried any others. I've CC'd Brian May because he does a lot more NFS stuff with Debian than most people and he may be able to advise you. I suggest just getting 4 IDE disks and putting everything on a software RAID-10 apart from /boot (which must be RAID-1 for the boot loader to work). It'll get the most bang for the buck! -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
