Re: [Q] qmail speed again
Marc Slemko writes: You put a limit of x connections in. One remote system uses all or nearly all of them. No one else can connect. Wrong again. New connections continue to be accepted and added to the kernel's (large) table of TCBs. Each of the old connections receives a message from the remote client and is done within a few seconds, to be replaced immediately by one of the new connections. ---Dan
Re: [Q] qmail speed again
On Wed 1999-04-14 (12:18), Silver CHEN wrote: Dear Keith: I think you got my point, thanks. My experience on sendmail at this tpoic *is* quite good - sorry, I don't have such experience on qmail yet. My intuition tells me that the first way will be better, since it will send only one copy of messages on network. If I have a 30KB+ message size, and 400K subscribers, the first method 'seems' to be much more efficient than the second one. However, this is what I 'think' - no real world experience on qmail yet. I'll examine this behavior on qmail soon, the machine is still in the box, I don't have time to open/assamble/install it these days. Our newsletters have 'sponsors', and some information is time-intensive, so my life is harder and harder :( Anyway, thanks for any comment on this tpoic. Please note that I only speculated that sendmail would be faster in this one particular case. For pretty much any other scenario qmail is significantly faster. I have noticed that using multiple-rcpt messages to one particular host appeared to be faster than many single message individual smtp sessions, but as my fellow list members have pointed out this does not necessarily mean that this would always be the case. If you're looking for speed, qmail is definitely the way to go. If all of your 400K subscribers are on the same remote host then perhaps do some profiling to see whether or not you'd gain by using multiple rcpt to's. Otherwise IMHO don't bother. Just use qmail :-) - Keith Silver CHEN -- Keith Burdis - MSc (Com Sci) - Rhodes University, South Africa Email : [EMAIL PROTECTED] WWW : http://www.rucus.ru.ac.za/~keith/ IRC : Panthras JAPH "Any technology sufficiently advanced is indistinguishable from a perl script" Standard disclaimer. ---
Re: [Q] qmail speed again
I'm replying to several messages here (see References), but I'm not going to bother attributing each quote. qmail will always be faster than sendmail [unless you send one message to a large number of addresses on the same remote host]. No, qmail will usually win here, too, because sendmail serializes. Sendmail only wins when the message is huge. Sendmail will win if you use multiple rcpt-to's. No, SMTP requires too many round trips per recipient. qmail will always be faster than sendmail [unless you send one message to a large number of addresses on the same remote host]. No, qmail will usually win here, too, because sendmail serializes. Sendmail only wins when the message is huge. Actually, if you are unfortunate enough to have a list of addresses sorted by the right side of the @, qmail can be a big loser here. This is because it will completely overload many remote hosts if there are a bunch of recipients. eg. concurrencyremote = 120, you have 200 users @somedomain, qmail will sit there with 120 connections to somedomain's mailserver open while they all crawl along because somedomain can't handle 120 connections at once. somedomain is poorly configured. Should qmail assume all sites are poorly configured? Should properly configured sites suffer because some sites are poorly run? qmail is great that way at inflicting remote DoS attacks against other mailers. That's loaded language. If you want constructive debate, you might want to avoid that. I don't know? Why does "qmail" accept connections that it cannot handle? Mine doesn't. I restrict it via tcpserver to an acceptable, conservative number of connections that, in my experience, the system will be able to handle in most foreseeable circumstances. This is great in theory. In practice, does your tcpserver setup automatically start refusing connections when you're low on queue space or when you know the load will be too high? I bet it doesn't. No, but my tcpserver connection limit is sufficiently conservative that it can almost always handle a full compliment of SMTP clients. If it can't, so what? Chances are, a few too many SMTP connections when the system is hosed are the least of my worries. The reason why it doesn't isn't because it's poorly designed. It's because it's hard to do this. You have to be partially psychic in order to always catch it. It's hard (impossible?) to dynamically determine if the system has sufficient resources to handle a request before accepting the connection, but, in practice, I've never found it necessary to do that. Even on a mail server, setting the connection limit conservatively has always worked fine. Only if they are silly enough to accept more connections than they can handle. :) One of the things a sys admin is "supposed" to do is tune his machines for performance. If you cannot limit the number of connections you will accept to something your system can handle, you need to re-think your setup. Erm... you just described a classic DoS attack. You put a limit of x connections in. One remote system uses all or nearly all of them. No one else can connect. I would call this inadvertant, temporary denial of service, at worst. It's not an attack, IMO, unless it's intentional and sustained. Since qmail only sends one message per connection, other senders are going to have plenty of opportunities to grab connections. And many of them (e.g., sendmail) will hold onto their connection until they've dumped their entire load. Enough of them, and they'll be "denying service" to any other senders. You may have a point here. Is there a well-defined rubric within which we can assert, "It is ill-mannered to consume all available connections to a remote server, just because those services are needed?" Could be, I suppose--that's a question for admins. The point is that, in a lot of cases, they aren't needed. If you have 50 messages to go out with 100 messages to each of 5000 hosts, the claim that it is necessary to open 100 connections at once to any single remote server is obviously wrong. The claim that qmail forces you to do that is also obviously wrong. IF you don't want that to happen, don't sort your list by MX. Heck, even if you had 50 messages to go to 1 host that doesn't mean you have to open 50 connections (well, bounded only by your local configuration and qmail's limits) to the remote host. Of course not. What's your point? 256 is 50. If I've got 50 different messages to deliver to a single site, I sure as hell want to use as many simultaneous connections as I can. I assume the receiving site will behave responsibly and only accept as many as it can handle. If the don't, that's their problem. -Dave
Re: [Q] qmail speed again
On Tue, 13 Apr 1999, Dave Sill wrote: qmail will always be faster than sendmail [unless you send one message to a large number of addresses on the same remote host]. No, qmail will usually win here, too, because sendmail serializes. Sendmail only wins when the message is huge. Actually, if you are unfortunate enough to have a list of addresses sorted by the right side of the @, qmail can be a big loser here. This is because it will completely overload many remote hosts if there are a bunch of recipients. eg. concurrencyremote = 120, you have 200 users @somedomain, qmail will sit there with 120 connections to somedomain's mailserver open while they all crawl along because somedomain can't handle 120 connections at once. somedomain is poorly configured. Should qmail assume all sites are poorly configured? Should properly configured sites suffer because some sites are poorly run? Geesh, this sort of crazy idealism is why qmail gets a bad name. Please tell me how I can configure qmail so that it isn't "poorly configured". Please give me an example of how to set it up so that a remote site can open as many connections as it wants (which you think it should be able to do) without monopolizing the system. On top of that, you are still completely ignore the fact that hammering each remote host as hard as possible, in turn, results in taking a lot longer to deliver all the mail than if you nicely avoid hammering each host. Part of this is due to qmail's silly 256 concurrencyremote limit, which makes connections from the sender a very valuable commodity when trying to send a lot of mail. [...] I don't know? Why does "qmail" accept connections that it cannot handle? Mine doesn't. I restrict it via tcpserver to an acceptable, conservative number of connections that, in my experience, the system will be able to handle in most foreseeable circumstances. Then do you really think it is appropriate for one remote system to monopolize your mailer for no reason? I consider that abusive. You can go on about "oh, there is really no way to know what you can do so you just have to try to blast as much as you can" but that ignores the basic principles that have worked behind all internet services for a heck of a long time: just because you _can_ do something, doesn't mean you should do it if it isn't a friendly thing to do. This is great in theory. In practice, does your tcpserver setup automatically start refusing connections when you're low on queue space or when you know the load will be too high? I bet it doesn't. No, but my tcpserver connection limit is sufficiently conservative that it can almost always handle a full compliment of SMTP clients. If it can't, so what? Chances are, a few too many SMTP connections when the system is hosed are the least of my worries. "if I can't accept any mail from any other systems because one system decided to flood me with a huge amount just beacuse it thought it would be cool to schedule deliveries that way, then it isn't a big deal" Erm... I don't think that way. [...] You may have a point here. Is there a well-defined rubric within which we can assert, "It is ill-mannered to consume all available connections to a remote server, just because those services are needed?" Could be, I suppose--that's a question for admins. The point is that, in a lot of cases, they aren't needed. If you have 50 messages to go out with 100 messages to each of 5000 hosts, the claim that it is necessary to open 100 connections at once to any single remote server is obviously wrong. The claim that qmail forces you to do that is also obviously wrong. IF you don't want that to happen, don't sort your list by MX. My claim is that qmail's behaviour ranges from extremely unfriendly to abusive. It is easy to end up with addresses sorted by hostname for a variety of reasons. It is a flaw in qmail that, if the address list is sorted in such a way, it takes ~twice as long, in my experience, to send out a large quantity of mail. Heck, even if you had 50 messages to go to 1 host that doesn't mean you have to open 50 connections (well, bounded only by your local configuration and qmail's limits) to the remote host. Of course not. What's your point? 256 is 50. If I've got 50 different messages to deliver to a single site, I sure as hell want to use as many simultaneous connections as I can. I assume the receiving You just said that, if qmail didn't have an arbitrary 256 simultaneous connection limit and if your machine could handle it, then you consider it perfectly acceptable to try to open 50 simultaneous connections. site will behave responsibly and only accept as many as it can handle. If the don't, that's their problem. That sums up the problems with your attitude quite nicely, and the problems with far too many "qmail idealists" that are more concernend about proclaiming qmail to be the
Re: [Q] qmail speed again
At 08:42 AM Tuesday 4/13/99, Marc Slemko wrote: On Tue, 13 Apr 1999, Dave Sill wrote: qmail will always be faster than sendmail [unless you send one message to a large number of addresses on the same remote host]. No, qmail will usually win here, too, because sendmail serializes. Sendmail only wins when the message is huge. Actually, if you are unfortunate enough to have a list of addresses sorted by the right side of the @, qmail can be a big loser here. This is because it will completely overload many remote hosts if there are a bunch of recipients. eg. concurrencyremote = 120, you have 200 users @somedomain, qmail will sit there with 120 connections to somedomain's mailserver open while they all crawl along because somedomain can't handle 120 connections at once. somedomain is poorly configured. Should qmail assume all sites are poorly configured? Should properly configured sites suffer because some sites are poorly run? Geesh, this sort of crazy idealism is why qmail gets a bad name. It may well be. Your trolling is pretty much the other side of the coin. It'd be nice if you added value rather than repeated what has been said on this list a number of times before. Much of what you say is based on speculation rather than fact. And what little useful discussion you have to contribute is loaded with words like "crazy", "idealism", "silly", "hammering", "abusive", "extremely unfriendly", "flaw", etc. Do you seriously expect people to listen to you when you degenerate into that sort of language? Furthermore, to claim that we are not interested in running "functional and interoperable mail systems" just tells us that you have no clue about the people on this list and what they do. What did you actually want to achieve with this sort of inflamatory remark? Surely not constructive discussion? As a small note, the systems I have setup running qmail have probably delivered well over 500+ million mails to just about every domain on the planet. Others on this list have done much more, so saying we have no interest in functional mail systems is a load or rubbish at best and clear bait at worst. In short, we've heard your story already, ok? In the spirit of publicly available sources, either say something new, change qmail in ways you think are better and share it with the rest of the world or be prepared to be recognized as yet another "whinger" who finds it easier to complain than do something. So what's it to be? Are you more interested in the easy game of flame or are you actually interested in doing something useful? Regards.
Re: [Q] qmail speed again
Marc Slemko [EMAIL PROTECTED] wrote: ...Please give me an example of how to set it up so that a remote site can open as many connections as it wants (which you think it should be able to do) without monopolizing the system. I don't care if a remote site uses all available SMTP connections if: o it doesn't "hog" connections, and o it's not sending spam. qmail doesn't hog connections because it only sends one message per connection. Spam is a separate problem with, IMHO, no technical solution. On top of that, you are still completely ignore the fact that hammering each remote host as hard as possible, in turn, results in taking a lot longer to deliver all the mail than if you nicely avoid hammering each host. Only if the remote host accepts too many connections. Senders can "hammer" my qmails all they want: I've pre-throttled them to a level they can sustain. It's only common sense. Arguments like "we never had to configure our MTA's intelligently before, so if they break now it's your fault" just don't work. Part of this is due to qmail's silly 256 concurrencyremote limit, which makes connections from the sender a very valuable commodity when trying to send a lot of mail. I find it very hard to hit that "silly" limit, even on my busiest, fastest server. I don't see how raising the limit would help dramatically. And I'd expect Chicken Little's to raise a big stink about qmails capable of opening thousands of simultanenous connections. Nonetheless, hardware's getting faster and I expect qmail 2.0 will raise this limit. Then do you really think it is appropriate for one remote system to monopolize your mailer for no reason? No reason? No. For the reason of sending me mail? Yes. I consider that abusive. You can go on about "oh, there is really no way to know what you can do so you just have to try to blast as much as you can" but that ignores the basic principles that have worked behind all internet services for a heck of a long time: just because you _can_ do something, doesn't mean you should do it if it isn't a friendly thing to do. Hey, this isn't your father's Internet. Sendmail-like performance just doesn't cut it. Poorly managed services/servers are likely to drown. It may not be "nice" or "friendly", but my nice, friendly users aren't infinitely patient: they want their mail delivered *now*. No, but my tcpserver connection limit is sufficiently conservative that it can almost always handle a full compliment of SMTP clients. If it can't, so what? Chances are, a few too many SMTP connections when the system is hosed are the least of my worries. "if I can't accept any mail from any other systems because one system decided to flood me with a huge amount just beacuse it thought it would be cool to schedule deliveries that way, then it isn't a big deal" Erm... I don't think that way. I do. My claim is that qmail's behaviour ranges from extremely unfriendly to abusive. It is easy to end up with addresses sorted by hostname for a variety of reasons. And very easy to prevent it, if it's a problem. It is a flaw in qmail that, if the address list is sorted in such a way, it takes ~twice as long, in my experience, to send out a large quantity of mail. No complex system can be optimal at all times. In my experience, this "flaw" just isn't a problem. Of course not. What's your point? 256 is 50. If I've got 50 different messages to deliver to a single site, I sure as hell want to use as many simultaneous connections as I can. I assume the receiving You just said that, if qmail didn't have an arbitrary 256 simultaneous connection limit and if your machine could handle it, then you consider it perfectly acceptable to try to open 50 simultaneous connections. No I didn't. I said there's a big difference between 256 and 50. Even with "well behaved" MTA's, a busy server today can expect on the order of 256 connections from various hosts, so if my qmail server should happen to get them all, the server shouldn't be swamped. However, the busiest, most capable SMTP server in existence couldn't handle 50 connections, so I consider it unreasonable to attempt that many. Ultimately, though, if my server accepts that many connections, it's my responsibility to see that it can cope. site will behave responsibly and only accept as many as it can handle. If the don't, that's their problem. That sums up the problems with your attitude quite nicely, and the problems with far too many "qmail idealists" that are more concernend about proclaiming qmail to be the greatest thing on earth than about actually running functional and interoperable mail systems. It is obvious that you aren't interested in real life problems or actually using qmail in real life, but only on insisting that qmail is flawless, so there isn't too much point in continuing this discussion. You're obviously wrong, though, because I use qmail daily, and have for years, to deliver high volumes of
Re: [Q] qmail speed again
Keith Burdis wrote/schrieb/scribsit: Remember that we're talking about sending one message to a large number of addresses on the same remote host. In general qmail is faster, but I think in this case any MTA that does multiple rcpt to's will be quicker. AFAIK, sendmail has a limit of "RCPT TO"s it will send in one connection. So in this situation a message will be split into several multi-rcpt connections. Will these connections ocurr one at a time or simultaneously? The former would impose a great speed penalty compared to qmail. Anyway, the One True Answer on this topic is: "profile, don't speculate". Stefan
Re: [Q] qmail speed again
Keith Burdis [EMAIL PROTECTED] writes on 13 April 1999 at 21:39:09 + What I had in mind was that with sendmail you can do: HELO MAIL FROM RCPT TO: address-1 RCPT TO: address-2 RCPT TO: address-n DATA ... whereas with qmail, since it doesn't do multiple rcpts, you'd have to do: for i = 1 to n HELO MAIL FROM RCPT TO: address-i DATA ... QUIT Remember that we're talking about sending one message to a large number of addresses on the same remote host. In general qmail is faster, but I think in this case any MTA that does multiple rcpt to's will be quicker. Yes, it does seem logical that the sendmail approach would be faster. Have you experimented with this situation in the real world? People who have report that in fact the qmail way is faster. Performance analysis is important, but that analysis must not stop with examining the algorithms; it must be carried out into the field, to see what *really* happens in the real world. While there are cases where qmail loses on delivery speed, they're far rarer than an armchair analysis might suggest. Or, to put it differently, in performance analysis, what "seems right" is not always what *is* right. -- David Dyer-Bennet [EMAIL PROTECTED] http://www.ddb.com/~ddb (photos, sf) Minicon: http://www.mnstf.org/minicon http://ouroboros.demesne.com/ The Ouroboros Bookworms Join the 20th century before it's too late!
Re: [Q] qmail speed again
I think you should get information on latency for message from a machine running a large distribution lst to see where it's spending its time probably waiting for those slug domains which have either * slow links * slow dns * are no longer in services but, that is just a quick guess ;) note that all of these really are not dependent upon whether you do one connection and pipe a bazillion messages over it, or a bazillion connection of one message each. Imho, i still think that the relatively random distribution of target addresses for a server makes this relatively moot, unless you are bulk-mailing 300,000 messages to AOL. however, in that case, shouldn't the bulk mailer just talk SMTP and save everyone the hassle? -- craig
Re: [Q] qmail speed again
On Tue, 13 Apr 1999, Keith Burdis wrote: Remember that we're talking about sending one message to a large number of addresses on the same remote host. In general qmail is faster, but I think in this case any MTA that does multiple rcpt to's will be quicker. if the effect latency of the connection is greater than the bandwidth between the end systems multiple connections will complete faster than multiple rcpts. In general this is true: the latency of my dialup connection is a greater effect than the bandwidth on sending messages), the latency between my ISP and the rest of the Internet is greater than its bandwidth... I think you should get information on latency for message from a machine running a large distribution lst to see where it's spending its time Richard
Re: [Q] qmail speed again
What I had in mind was that with sendmail you can do: HELO MAIL FROM RCPT TO: address-1 RCPT TO: address-2 RCPT TO: address-n DATA ... whereas with qmail, since it doesn't do multiple rcpts, you'd have to do: for i = 1 to n HELO MAIL FROM RCPT TO: address-i DATA ... QUIT Remember that we're talking about sending one message to a large number of addresses on the same remote host. In general qmail is faster, but I think in this case any MTA that does multiple rcpt to's will be quicker. -Dave - Keith Dear Keith: I think you got my point, thanks. My experience on sendmail at this tpoic *is* quite good - sorry, I don't have such experience on qmail yet. My intuition tells me that the first way will be better, since it will send only one copy of messages on network. If I have a 30KB+ message size, and 400K subscribers, the first method 'seems' to be much more efficient than the second one. However, this is what I 'think' - no real world experience on qmail yet. I'll examine this behavior on qmail soon, the machine is still in the box, I don't have time to open/assamble/install it these days. Our newsletters have 'sponsors', and some information is time-intensive, so my life is harder and harder :( Anyway, thanks for any comment on this tpoic. -- Regards Silver CHEN 1999/4/14
Re: [Q] qmail speed again
Silver CHEN [EMAIL PROTECTED] wrote: The mail reason that I can't switch to qmail is that I'm NOT familiar with qmail in early days, so I chose sendmail. You can install qmail without removing/breaking sendmail, so you can revert to sendmail easily. I've read all the articles about the topic 'qmail speed'. Now I'm wonder that if qmail can do this too? If the design spirit of qmail can't afford such high-loading task in short time, then sendmail takes the chance back. qmail was designed for higher performance than sendmail. The "qmail speed" thread was about a particular type of mass mailing that's much harder than sending the same message to a large list: in this case, the message was customized for each recipient. Instead of one message with 100,000 recipients, he was sending 100,000 different messages to one recipient each. As I know, hotmail use qmail as its base(?) but hotmail is slow, slow, and unreliable 'sometimes'. I don't know what will happen if hotmail choose sendmail as its base now, but I'm curious on this topic. Hotmail, last time I checked, used qmail for outgoing mail, zmailer for incoming mail. If it's slow and unreliable, it's not because it uses qmail, it's because it's too busy or poorly run. If sendmail would have been better than qmail+zmailer, they probably would have used it. -Dave
Re: [Q] qmail speed again
On Mon, Apr 12, 1999 at 11:55:17AM -0400, Dave Sill wrote: Hotmail, last time I checked, used qmail for outgoing mail, zmailer for incoming mail. If it's slow and unreliable, it's not because it uses qmail, it's because it's too busy or poorly run. If sendmail would have been better than qmail+zmailer, they probably would have used it. The 'qmail for outgoing' claim is correct. The 'zmailer for incoming' claim is very ridiculous. Until a couple of weeks ago (maybe even shorter) hotmail used a very nice SMTP agent of their own making. I don't know what it is they're using now, but it is quite broken: telnet mail.hotmail.com 25 Trying 209.1.112.253... Connected to mail.hotmail.com. Escape character is '^]'. 220-HotMail (NO UCE) ESMTP server ready at Mon Apr 12 09:01:20 1999 220 ESMTP spoken here EHLO attic.vuurwerk.nl an esmtp server that does not understand EHLO... Greetz, Peter -- | 'He broke my heart,| Peter van Dijk | I broke his neck' | [EMAIL PROTECTED] | nognixz - As the sun |Hardbeat@ircnet - #cistron/#linux.nl | | Hardbeat@undernet - #groningen/#kinkfm/#vdh |
Re: [Q] qmail speed again
Peter van Dijk [EMAIL PROTECTED] wrote: The 'qmail for outgoing' claim is correct. The 'zmailer for incoming' claim is very ridiculous. See: http://www.ornl.gov/its/archives/mailing-lists/qmail/1997/08/msg01208.html http://www.ornl.gov/its/archives/mailing-lists/qmail/1998/05/msg00660.html They may not be running it now, but they used to. -Dave
Re: [Q] qmail speed again
On Mon, 12 Apr 1999, Dave Sill wrote: Silver CHEN [EMAIL PROTECTED] wrote: The mail reason that I can't switch to qmail is that I'm NOT familiar with qmail in early days, so I chose sendmail. You can install qmail without removing/breaking sendmail, so you can revert to sendmail easily. On our server (SGI Indy -- IRIX 6.5) this was mostly true, with one exception-- BSD mail users. We had a problem with a sendmail that was re-installed by default on our IRIX upgrades. ...because it seemed to be called by those users still using BSD mail on our system. Other users (like pine, IMAP, POP) had no problems. So we made a simple wrapper of sendmail that piped messages to qmail-inject. FWIW--- Samuel Daffner Mills College ITS
Re: [Q] qmail speed again
Samuel Dries-Daffner [EMAIL PROTECTED] wrote: On Mon, 12 Apr 1999, Dave Sill wrote: You can install qmail without removing/breaking sendmail, so you can revert to sendmail easily. On our server (SGI Indy -- IRIX 6.5) this was mostly true, with one exception-- BSD mail users. We had a problem with a sendmail that was re-installed by default on our IRIX upgrades. ...because it seemed to be called by those users still using BSD mail on our system. No, IRIX upgrades/patches will always re-install sendmail because SGI considers it a necessary piece of the operating system. Most qmail-aware SGI admins know this, and remove the sendmail binary and relink it to qmail after upgrading. Other users (like pine, IMAP, POP) had no problems. So we made a simple wrapper of sendmail that piped messages to qmail-inject. You reinvented /var/qmail/bin/sendmail? Sounds like you didn't read the installation instructions very carefully. -Dave
RE: [Q] qmail speed again
We tried it but there were lots of funny interactions with BSD mail...for example it would add extra 's at the end of addresses and cause lots of bounces. And another wierd thing was the interaction with the vacation program, also adding extra 's and not working consistently in general. (Since then we have switched to Peter Samuel's version which works very nicely). I posted to this and other lists and eventually just wrote a oneliner that solved all these problems. Granted I never really figured out what they were exactly, but this worked for us: ella 28# more sendmail #!/bin/sh cat | /var/qmail/bin/predate /var/qmail/bin/qmail-inject [new mail arrives from Dave Sill] qmail-aware SGI admins know this, and remove the sendmail binary and relink it to qmail after upgrading. I didn't do the qmail...install it was already there when I got the system. And, it was my first IRIX upgrade...but this is exactly what I learned and resolved as I explained above. You reinvented /var/qmail/bin/sendmail? Sounds like you didn't read the installation instructions very carefully. Well if the one liner I wrote above is reinventing then I guess so :) But mostly I was just trying to satisfy a few users who insist on using the BSD mail client. Several qmail list experts (Harald/Mate) worked with me extensively and we ran outr of avenues, but I am still eternally grateful! Samuel Daffner Mills College ITS On Mon, 12 Apr 1999, Soffen, Matthew wrote: Umm.. Why didn't you use /var/qmail/bin/sendmail ? -Original Message- From: Samuel Dries-Daffner [SMTP:[EMAIL PROTECTED]] Sent: Monday, April 12, 1999 12:19 PM To: Dave Sill Cc: [EMAIL PROTECTED] Subject:Re: [Q] qmail speed "again" On Mon, 12 Apr 1999, Dave Sill wrote: Silver CHEN [EMAIL PROTECTED] wrote: The mail reason that I can't switch to qmail is that I'm NOT familiar with qmail in early days, so I chose sendmail. You can install qmail without removing/breaking sendmail, so you can revert to sendmail easily. On our server (SGI Indy -- IRIX 6.5) this was mostly true, with one exception-- BSD mail users. We had a problem with a sendmail that was re-installed by default on our IRIX upgrades. ...because it seemed to be called by those users still using BSD mail on our system. Other users (like pine, IMAP, POP) had no problems. So we made a simple wrapper of sendmail that piped messages to qmail-inject. FWIW--- Samuel Daffner Mills College ITS
RE: [Q] qmail speed again
Umm.. Why didn't you use /var/qmail/bin/sendmail ? -Original Message- From: Samuel Dries-Daffner [SMTP:[EMAIL PROTECTED]] Sent: Monday, April 12, 1999 12:19 PM To: Dave Sill Cc: [EMAIL PROTECTED] Subject: Re: [Q] qmail speed "again" On Mon, 12 Apr 1999, Dave Sill wrote: Silver CHEN [EMAIL PROTECTED] wrote: The mail reason that I can't switch to qmail is that I'm NOT familiar with qmail in early days, so I chose sendmail. You can install qmail without removing/breaking sendmail, so you can revert to sendmail easily. On our server (SGI Indy -- IRIX 6.5) this was mostly true, with one exception-- BSD mail users. We had a problem with a sendmail that was re-installed by default on our IRIX upgrades. ...because it seemed to be called by those users still using BSD mail on our system. Other users (like pine, IMAP, POP) had no problems. So we made a simple wrapper of sendmail that piped messages to qmail-inject. FWIW--- Samuel Daffner Mills College ITS
Re: [Q] qmail speed again
On Mon 1999-04-12 (11:41), Dave Sill wrote: "Fred Lindberg" [EMAIL PROTECTED] wrote: qmail will always be faster than sendmail [unless you send one message to a large number of addresses on the same remote host]. No, qmail will usually win here, too, because sendmail serializes. Sendmail only wins when the message is huge. Sendmail will win if you use multiple rcpt-to's. - Keith -Dave -- Keith Burdis - MSc (Com Sci) - Rhodes University, South Africa Email : [EMAIL PROTECTED] WWW : http://www.rucus.ru.ac.za/~keith/ IRC : Panthras JAPH "Any technology sufficiently advanced is indistinguishable from a perl script" Standard disclaimer. ---
Re: [Q] qmail speed again
On Mon, 12 Apr 1999, Dave Sill wrote: "Fred Lindberg" [EMAIL PROTECTED] wrote: qmail will always be faster than sendmail [unless you send one message to a large number of addresses on the same remote host]. No, qmail will usually win here, too, because sendmail serializes. Sendmail only wins when the message is huge. Actually, if you are unfortunate enough to have a list of addresses sorted by the right side of the @, qmail can be a big loser here. This is because it will completely overload many remote hosts if there are a bunch of recipients. eg. concurrencyremote = 120, you have 200 users @somedomain, qmail will sit there with 120 connections to somedomain's mailserver open while they all crawl along because somedomain can't handle 120 connections at once. qmail is great that way at inflicting remote DoS attacks against other mailers.
Re: [Q] qmail speed again
Only if they are silly enough to accept more connections than they can handle. :) One of the things a sys admin is "supposed" to do is tune his machines for performance. If you cannot limit the number of connections you will accept to something your system can handle, you need to re-think your setup. On Mon, 12 Apr 1999, Marc Slemko wrote: On Mon, 12 Apr 1999, Dave Sill wrote: "Fred Lindberg" [EMAIL PROTECTED] wrote: qmail will always be faster than sendmail [unless you send one message to a large number of addresses on the same remote host]. No, qmail will usually win here, too, because sendmail serializes. Sendmail only wins when the message is huge. Actually, if you are unfortunate enough to have a list of addresses sorted by the right side of the @, qmail can be a big loser here. This is because it will completely overload many remote hosts if there are a bunch of recipients. eg. concurrencyremote = 120, you have 200 users @somedomain, qmail will sit there with 120 connections to somedomain's mailserver open while they all crawl along because somedomain can't handle 120 connections at once. qmail is great that way at inflicting remote DoS attacks against other mailers. - Timothy L. Mayo mailto:[EMAIL PROTECTED] Senior Systems Administrator localconnect(sm) http://www.localconnect.net/ The National Business Network Inc. http://www.nb.net/ One Monroeville Center, Suite 850 Monroeville, PA 15146 (412) 810- Phone (412) 810-8886 Fax
Re: [Q] qmail speed again
On Mon, 12 Apr 1999, Keith Burdis wrote: qmail is great that way at inflicting remote DoS attacks against other mailers. Well, the obvious question is why do mailers accept connections that they cannot handle? If the remote host accepts the mail it should be prepared to deal with it. blah blah blah. I don't know? Why does "qmail" accept connections that it cannot handle? It is a pretty simple concept: even if a mailer can "handle" it, if you send 1 simultaneous bit of mail to 1 machine, while using all your other slots for sending to other machines, you will get x% of each machine's resources. If you send 120 simultaneous messages, then you only get y% of the machine's resources, where y is between a little (eg. if the machine normally has 1 incoming connections) to a huge amount (eg. if the machine normally has 2 incoming connections) less than x. Because of somewhat nonsensical hard coded limits, when sending a lot of outbound mail with qmail, the number of simultaneous connections open total often is a big limitation. Because of that, the fastest way to send mail is to ensure you aren't overloading any remote machine more than necessary at any given time, so that each connection to that machine takes the least time possible, even if that involves more seialization. Trying to blame the remote mailer for qmail's less than great behaviour in this area is silly.
Re: [Q] qmail speed again
Marc Slemko [EMAIL PROTECTED] writes on 12 April 1999 at 13:13:51 -0700 qmail is great that way at inflicting remote DoS attacks against other mailers. Well, the other side knows about its capability to accept additional connections. We don't. Basic self-protection, from both enthusiastic mail-movers like qmail *and* deliberate DoS attacks, would call for not accepting connections you can't handle. Rate-limiting should be applied by the side that has the information to do it right. -- David Dyer-Bennet [EMAIL PROTECTED] http://www.ddb.com/~ddb (photos, sf) Minicon: http://www.mnstf.org/minicon http://ouroboros.demesne.com/ The Ouroboros Bookworms Join the 20th century before it's too late!
Re: [Q] qmail speed again
On Mon, 12 Apr 1999, Timothy L. Mayo wrote: Only if they are silly enough to accept more connections than they can handle. :) One of the things a sys admin is "supposed" to do is tune his machines for performance. If you cannot limit the number of connections you will accept to something your system can handle, you need to re-think your setup. Erm... you just described a classic DoS attack. You put a limit of x connections in. One remote system uses all or nearly all of them. No one else can connect. You can hack whatever sorts of other "oh, limit one remote IP to only so many, do this, do that" hacks on top of that you want, and some of them are actually quite useful, but it doesn't change the fact that the root problem is qmail's rudeness in this area. As someone using qmail (for various reasons...), I can't control every other system in the world, and even if I could, simple math still says that a remote machine only has so many resources so trying to make it do too many things at once is counterproductive and just eats up time in qmail-remotes, which is a limited resource in qmail.
Re: [Q] qmail speed again
Marc Slemko [EMAIL PROTECTED] writes on 12 April 1999 at 13:24:34 -0700 On Mon, 12 Apr 1999, Keith Burdis wrote: qmail is great that way at inflicting remote DoS attacks against other mailers. Well, the obvious question is why do mailers accept connections that they cannot handle? If the remote host accepts the mail it should be prepared to deal with it. blah blah blah. I don't know? Why does "qmail" accept connections that it cannot handle? Because it's not intended to run as a standalone daemon; it runs either under inetd or tcpserver or whatever, and load-based decisions about accepting connections should be implemented at that level? It is a pretty simple concept: even if a mailer can "handle" it, if you send 1 simultaneous bit of mail to 1 machine, while using all your other slots for sending to other machines, you will get x% of each machine's resources. If you send 120 simultaneous messages, then you only get y% of the machine's resources, where y is between a little (eg. if the machine normally has 1 incoming connections) to a huge amount (eg. if the machine normally has 2 incoming connections) less than x. The other pretty simple concept is that carefully sorting the queue out by the IP that will actually handle the connection, either to aggregate connections OR to deliberately spread out the load, is *very* expensive (DNS queries), and tends to swamp any benefit to be gained (at least on the aggregation side; the other side, deliberately spreading the load, hasn't to my knowledge been studied as carefully). Because of somewhat nonsensical hard coded limits, when sending a lot of outbound mail with qmail, the number of simultaneous connections open total often is a big limitation. Because of that, the fastest way to send mail is to ensure you aren't overloading any remote machine more than necessary at any given time, so that each connection to that machine takes the least time possible, even if that involves more seialization. I think I can guess what your on about here, and I agree that a byte isn't an appropriate size for the counter on outstanding qmail-remote's. Doesn't affect me personally, but people with heavy mail hubs might well be. Remember, when initially coded this was such a great step up in concurrency that nobody noticed the funny limit. Is this number (either the current count, or perhaps as an id number for a particular task) ever passed through a pipe? I wonder if the limit really comes from simplifying that? Trying to blame the remote mailer for qmail's less than great behaviour in this area is silly. I don't think you've made much of a case for "less than great behaviour", personally. -- David Dyer-Bennet [EMAIL PROTECTED] http://www.ddb.com/~ddb (photos, sf) Minicon: http://www.mnstf.org/minicon http://ouroboros.demesne.com/ The Ouroboros Bookworms Join the 20th century before it's too late!
Re: [Q] qmail speed again
Marc Slemko [EMAIL PROTECTED] writes: On Mon, 12 Apr 1999, Timothy L. Mayo wrote: If you cannot limit the number of connections you will accept to something your system can handle, you need to re-think your setup. Erm... you just described a classic DoS attack. You put a limit of x connections in. One remote system uses all or nearly all of them. No one else can connect. Erm... DoS may be a sensitive subject on this list! :-) I think DJB has given a mathematically rigorous proof that "program which is vulnerable to DoS" is one definition of "server". When a server is serving all it can, it will refuse to serve any more. QED The upshot is that _preventing_ a DoS is not at all possible. Any rate-limiting, or other countermeasures you adopt, merely adjusts the parameters within which a DoS must be conducted. Limiting concurrency, ulimits, etc., are not about _preventing_ DoS, which is impossible, but about ensuring that even under maxed-out conditions the mail server will not die or become unstable. ...the root problem is qmail's rudeness in this area. You may have a point here. Is there a well-defined rubric within which we can assert, "It is ill-mannered to consume all available connections to a remote server, just because those services are needed?" Could be, I suppose--that's a question for admins. Given that all service requests were legitimate, I would not mind my server being pegged with connections from one host. When such conditions become routine, from particular hosts, taking appropriate action is why Sys Admins get paid. Of course, that's just MHO. Len. -- 103. In Company of your Betters be not longer in eating than they are lay not your Arm but only your hand upon the table. -- George Washington, "Rules of Civility Decent Behaviour"
Re: [Q] qmail speed again
I think DJB has given a mathematically rigorous proof that "program which is vulnerable to DoS" is one definition of "server". When a server is serving all it can, it will refuse to serve any more. QED this is accurate for a single function server. in a multifunction server, it would be smart for the administrators to construct it in such a fashion such that one function can't seriously degrade other offered functionality. ...the root problem is qmail's rudeness in this area. You may have a point here. Is there a well-defined rubric within which we can assert, "It is ill-mannered to consume all available connections to a remote server, just because those services are needed?" Could be, I suppose--that's a question for admins. i would add ineffient as well as ill-mannered. consider the system overhead requred to fork processes/threads on both servers combined with the network and application startup time for the connections, and it becomes obvious that when delivering a statistically large amount of mail to a single recipient server, it makes more sense to pipeline the mail. of course, we all knew this to begin with. OTOH, qmail works because in non-bulk sending mode (hint to folks) there is a reasonably random distribution of domain names which really only sees problems upon death/unreachability of a specific target. As for those bulk senders, a relatively painless solution is to randomize the order of the domains and/or pull out the top 5 and serially send those. Your tools *do* talk smtp, don't they? Given that all service requests were legitimate, I would not mind my server being pegged with connections from one host. When such conditions become routine, from particular hosts, taking appropriate action is why Sys Admins get paid. Of course, that's just MHO. better pegged from legit mail than pegged by script kiddies ;) -- craig
Re: [Q] qmail speed again
On Tue, Apr 13, 1999 at 05:35:19PM -0400, Craig I. Hagan wrote: As for those bulk senders, a relatively painless solution is to randomize the order of the domains and/or pull out the top 5 and serially send those. Your tools *do* talk smtp, don't they? qmail-remote will handle multiple RCPT TOs in a single SMTP transaction. If you're into bulk mailing, you can write a script to sort your recipient list and call qmail-remote directly, one invocation for each remote host. Chris
Re: [Q] qmail speed again
Timothy L Mayo [EMAIL PROTECTED] writes: Neither Keith nor I are talking about the mailer. If the remote MACHINE cannot handle the load, it should be set up to reject the excess connections. This is true no matter WHAT MTA is running on the remote system. This is great in theory. In practice, does your tcpserver setup automatically start refusing connections when you're low on queue space or when you know the load will be too high? I bet it doesn't. The reason why it doesn't isn't because it's poorly designed. It's because it's hard to do this. You have to be partially psychic in order to always catch it. sendmail has a bunch of load-limiting features, particularly in the latest versions. It can start queuing instead of mailing directly, it can refuse connections entirely, and it can do this on the basis of the current number of running daemons, system load, or disk space. It's still relatively easy to make a machine running sendmail fall over under load. It's harder for qmail primarily because qmail is small and fast, not because it has any better load management capabilities near the boundary of what the machine can cope with than sendmail does. -- Russ Allbery ([EMAIL PROTECTED]) URL:http://www.eyrie.org/~eagle/
Re: [Q] qmail speed again
I've read all the articles about the topic 'qmail speed'. Now I'm wonder that if qmail can do this too? If the design spirit of qmail can't afford such high-loading task in short time, then sendmail takes the chance back. qmail, exim and postfix are all much faster than sendmail and you should compare these 3 MTAs. Maybe you could also take a look at postfix, it's more like sendmail and I found it very easy to understand (I also like qmail very much and it's simple too, but different from other MTAs).
