Re: how do you use a deferral host in qmail?
Jeremy Hansen writes: > > You're cocky and absolutely useless. And your qmail server gets "clogged up" with deferrals. That's two things you've said that go counter to my experience. -- -russ nelson <[EMAIL PROTECTED]> http://russnelson.com Crynwr sells support for free software | PGPok | "Ask not what your country 521 Pleasant Valley Rd. | +1 315 268 1925 voice | can force other people to Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | do for you..." -Perry M.
Re: how do you use a deferral host in qmail?
Jeremy Hansen writes: > I'm wondering if it's possible to setup a deferral host in qmail? If a > message gets deferred, then the mail goes to another machine in charge of > just retrying deferred messages instead of clogging up the main mailer > machine. This is not necessary. Sendmail gets clogged up and needs this feature. Qmail does not and does not. -- -russ nelson <[EMAIL PROTECTED]> http://russnelson.com Crynwr sells support for free software | PGPok | "Ask not what your country 521 Pleasant Valley Rd. | +1 315 268 1925 voice | can force other people to Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | do for you..." -Perry M.
Re: how do you use a deferral host in qmail?
Ian Lance Taylor <[EMAIL PROTECTED]> wrote: >smtproutes is read by qmail-remote, and a new qmail-remote process is >spawned for each outgoing mail message anyhow. Of course. Thanks. -Dave
Re: how do you use a deferral host in qmail?
Date: Fri, 31 Mar 2000 10:37:25 -0500 (EST) From: Dave Sill <[EMAIL PROTECTED]> >The remaining problem is to have a current set of smtproutes >available which could probably be generated from log file >analysis in near-realtime. Doing it anywhere near realtime would require many qmail restarts on both systems, which is not good for performance. smtproutes is read by qmail-remote, and a new qmail-remote process is spawned for each outgoing mail message anyhow. This plan shouldn't have a significant performance impact. (Well, I suppose a very large smtproutes file might be problematical.) But I agree that using distributing the mail randomly across multiple outgoing mail servers seems like a better and simpler approach. Ian
Re: how do you use a deferral host in qmail?
[EMAIL PROTECTED] wrote: >On Thu, Mar 30, 2000 at 03:23:21PM -0500, Dave Sill wrote: >> Logging with multilog is very cheap. > >sounds interesting... where do I look? ftp://koobera.math.uic.edu/www/daemontools.html >:-) Just an idea on how to split the mail: > >I guess that "a large percentage" of said deferred mail goes to a >handful of domains. So I would take that spare machine and set it >up to handle such mail, and configure the main mail server to >forward that mail to the second mail server using "smtproutes". >So you would deliver most of that mail immediately to the >secondary server, thus freeing the main server's resources, >and have the secondary mail server trying the slower deliveries. That's an interesting approach. I'm still unclear about the advantages of splitting the load between servers by deliverability vs. some arbitrary scheme. How does having the main server pumping out all highly deliverable mail and the fallback server plodding away on frequently deferred mail win over two (or three or N) servers doing a mix of both? >The remaining problem is to have a current set of smtproutes >available which could probably be generated from log file >analysis in near-realtime. Doing it anywhere near realtime would require many qmail restarts on both systems, which is not good for performance. -Dave
Re: how do you use a deferral host in qmail?
On Thu, Mar 30, 2000 at 01:29:21PM -0800, Jon Rust wrote: > At 4:08 PM -0500 3/30/00, Jeremy Hansen wrote: > >You're cocky and absolutely useless. > > > >Thanks > >-jeremy > > Whoa, you're so far off base now, I'd guess you just lost all > interest from anyone else worthwhile on the list. Mine anyway... > Dave Sill has been, and continues to be, a tremendous support > resource on the list and through LWQ. Just because he didn't give the > answer you wanted doesn't mean he's "absolutely useless". Correct. Dave rulez :) > Take a deep breath, play some Q3A or whatever, and realize that he > and John Levine have pointed you in the right direction. Correct. Now, I want killfiles for mutt :) Greetz, Peter. -- Peter van Dijk - student/sysadmin/ircoper/madly in love/pretending coder | | 'C makes it easy to shoot yourself in the foot; | C++ makes it harder, but when you do it blows your whole leg off.' | Bjarne Stroustrup, Inventor of C++
RE: how do you use a deferral host in qmail?
Hear hear!! We appreciate your work, Dave. > -Original Message- > From: Ismal Hisham Darus [mailto:[EMAIL PROTECTED]] > Sent: Thursday, March 30, 2000 11:02 PM > To: [EMAIL PROTECTED] > Subject: Re: how do you use a deferral host in qmail? > > > > Jeremy Hansen <[EMAIL PROTECTED]> wrote: > > > > > > You're cocky and absolutely useless. > > dave is cocky and useless ? haha .. i don't know whether this is a > joke or what .. he is the one who i think make 90% of my qmail server > works .. and LWQ .. gees first time i try .. everything works to the > perfection. > > *salute* you dave.. > > > > > > > Ismal Hisham Mohd Darus > Asst. Manager, System Support > John Hancock Life Insurance (Malaysia) Berhad
Re: how do you use a deferral host in qmail?
> Jeremy Hansen <[EMAIL PROTECTED]> wrote: > > > > You're cocky and absolutely useless. dave is cocky and useless ? haha .. i don't know whether this is a joke or what .. he is the one who i think make 90% of my qmail server works .. and LWQ .. gees first time i try .. everything works to the perfection. *salute* you dave.. Ismal Hisham Mohd Darus Asst. Manager, System Support John Hancock Life Insurance (Malaysia) Berhad
Re: how do you use a deferral host in qmail?
At 4:08 PM -0500 3/30/00, Jeremy Hansen wrote: >You're cocky and absolutely useless. > >Thanks >-jeremy Whoa, you're so far off base now, I'd guess you just lost all interest from anyone else worthwhile on the list. Dave Sill has been, and continues to be, a tremendous support resource on the list and through LWQ. Just because he didn't give the answer you wanted doesn't mean he's "absolutely useless". Take a deep breath, play some Q3A or whatever, and realize that he and John Levine have pointed you in the right direction. jon
Re: how do you use a deferral host in qmail?
Jeremy Hansen <[EMAIL PROTECTED]> wrote: > > You're cocky and absolutely useless. And you're a jerk. Actually, Dave is arguably one of the most useful people around when it comes to qmail. Until the long-awaited O'Reilly book comes out, his "Life with qmail" is pretty much the qmail bible. If you don't understand what he's telling you, then go ahead and ask questions--but you really ought to be respectful or at least polite. As Dave said, this isn't rocket science. Finally, Dave is correct that Dan won't add any "feature that seems to be useful" to qmail. Remember, the SMTP server is one of the primary entry routes for crackers, thanks to Eric Allman. Dan will not risk security or reliability for any old feature you dream up. As Dan said before: In general, I'm not going to blindly copy sendmail features---even useful features. I want to understand what problem they're trying to solve; then I'll provide the best solution for that problem. Len. -- On the bright side, if a local user notices the system slowing down, he can monitor the drop directory, decide that it's probably a spammer, and destroy all new messages, without bothering to wake up the sysadmin. ``It's not a security disaster; it's an anti-spam feature!'' -- Dan Bernstein, about Postfix
Re: how do you use a deferral host in qmail?
You're cocky and absolutely useless. Thanks -jeremy > Jeremy Hansen <[EMAIL PROTECTED]> wrote: > > >So it can't be done is what you're saying. > > In my first reply I said it couldn't be done "out of the box". > > qmail is highly modular, though, so a fairly simple qmail-inject > wrapper like John Levine suggested could be used to implement this > functionality: try to send the message directly with qmail-remote. If > that succeeds, you're done. If it fails, queue the message on your > fallback server. This isn't rocket science. > > >I haven't really seen any good > >arguments as to why it shouldn't be done, > > I haven't seen any good arguments as to why it *should* be done. Dan > shuns features that *might* work. > > >but obviously the DJ cronies aren't going to argue his logic. > > I'm not a DJB crony, but I'm not above second guessing him at > times. :-) > > >It's frustrating for someone like me who > >can recognize the many advantages of qmail, even with this little set back > >it kills sendmail, but when you run into a feature that seems to be useful > >(I'm sure I'm ot theonly one), then you're screwed because Dan says so. > > Oh, and with other MTA's you're not at the whim of the developer? If > you wish for a sendmail or PostFix feature, it will come to pass, > even against the will of the author? Fascinating. I didn't know > that. That would certainly explain how many "features" made it into > sendmail, though. :-) > > -Dave > http://www.xxedgexx.com | [EMAIL PROTECTED] -
Re: how do you use a deferral host in qmail?
Jeremy Hansen <[EMAIL PROTECTED]> wrote: >So it can't be done is what you're saying. In my first reply I said it couldn't be done "out of the box". qmail is highly modular, though, so a fairly simple qmail-inject wrapper like John Levine suggested could be used to implement this functionality: try to send the message directly with qmail-remote. If that succeeds, you're done. If it fails, queue the message on your fallback server. This isn't rocket science. >I haven't really seen any good >arguments as to why it shouldn't be done, I haven't seen any good arguments as to why it *should* be done. Dan shuns features that *might* work. >but obviously the DJ cronies aren't going to argue his logic. I'm not a DJB crony, but I'm not above second guessing him at times. :-) >It's frustrating for someone like me who >can recognize the many advantages of qmail, even with this little set back >it kills sendmail, but when you run into a feature that seems to be useful >(I'm sure I'm ot theonly one), then you're screwed because Dan says so. Oh, and with other MTA's you're not at the whim of the developer? If you wish for a sendmail or PostFix feature, it will come to pass, even against the will of the author? Fascinating. I didn't know that. That would certainly explain how many "features" made it into sendmail, though. :-) -Dave
Re: how do you use a deferral host in qmail?
So it can't be done is what you're saying. I haven't really seen any good arguments as to why it shouldn't be done, but obviously the DJ cronies aren't going to argue his logic. It's frustrating for someone like me who can recognize the many advantages of qmail, even with this little set back it kills sendmail, but when you run into a feature that seems to be useful (I'm sure I'm ot theonly one), then you're screwed because Dan says so. -jeremy > Jeremy Hansen <[EMAIL PROTECTED]> wrote: > > >My remote concurrency is 500, so that's not a problem. > > 500 is not a magic number guaranteed to be sufficient in all > applications. > > >In what I've seen > >and you have to understand, I'm just an admin here, I really have nothing > >to do with the "quality" of the mail addresses that come through > >here. That's a story within itself, but out of 2 million emails, 40% of > >those on average are deferred. That's a lot of emails sitting getting > >retried, > > They're only retried when their schedule says it's time to retry them. > > >I've often seen my remote concurrency consist completely of > >deferral retries, > > That right there says that your concurrencyremote probably isn't high > enough. > > >especially of for some reason qmail need to be restarted > >or something... > > Restarts are exceptional. > > >so if it has to sort through 800,000 mails that may get > >deferred again, that's wasted time and resources. > > Yeah, avoid restarts. > > >I know I have options, but a deferral host just seems like it would be a > >"nice thing" to have available. > > Ah, but qmail was engineered. Dan doesn't throw in every feature that > ``just seems like it would be a "nice thing" to have available''. It's > conceivable that a fallback host feature could make sense in some > applications, but I suspect Dan weighed the pros and cons and decided > it wasn't worth the effort. > > You can: implement it yourself, switch to a mailer that supports it, > or consider other options with qmail such as calling qmail-remote > directly and queuing to a fallback host if that fails. > > -Dave > http://www.xxedgexx.com | [EMAIL PROTECTED] -
Re: how do you use a deferral host in qmail?
Jeremy Hansen <[EMAIL PROTECTED]> wrote: >My remote concurrency is 500, so that's not a problem. 500 is not a magic number guaranteed to be sufficient in all applications. >In what I've seen >and you have to understand, I'm just an admin here, I really have nothing >to do with the "quality" of the mail addresses that come through >here. That's a story within itself, but out of 2 million emails, 40% of >those on average are deferred. That's a lot of emails sitting getting >retried, They're only retried when their schedule says it's time to retry them. >I've often seen my remote concurrency consist completely of >deferral retries, That right there says that your concurrencyremote probably isn't high enough. >especially of for some reason qmail need to be restarted >or something... Restarts are exceptional. >so if it has to sort through 800,000 mails that may get >deferred again, that's wasted time and resources. Yeah, avoid restarts. >I know I have options, but a deferral host just seems like it would be a >"nice thing" to have available. Ah, but qmail was engineered. Dan doesn't throw in every feature that ``just seems like it would be a "nice thing" to have available''. It's conceivable that a fallback host feature could make sense in some applications, but I suspect Dan weighed the pros and cons and decided it wasn't worth the effort. You can: implement it yourself, switch to a mailer that supports it, or consider other options with qmail such as calling qmail-remote directly and queuing to a fallback host if that fails. -Dave
Re: how do you use a deferral host in qmail?
My remote concurrency is 500, so that's not a problem. In what I've seen and you have to understand, I'm just an admin here, I really have nothing to do with the "quality" of the mail addresses that come through here. That's a story within itself, but out of 2 million emails, 40% of those on average are deferred. That's a lot of emails sitting getting retried, I've often seen my remote concurrency consist completely of deferral retries, especially of for some reason qmail need to be restarted or something...so if it has to sort through 800,000 mails that may get deferred again, that's wasted time and resources. There are a number of things I know I can do, nothing is balanced currently, but what I think would be nice is have the deferral host be load balanced across a few deferral machines, keeps those messages off the main mailers and we don't really care what happens to the deferred messages as long as they don't restrict outgoing mail. I know I have options, but a deferral host just seems like it would be a "nice thing" to have available. -jeremy > Jeremy Hansen <[EMAIL PROTECTED]> wrote: > > >Hmm, but qmail does get bogged down by deferrals from my experience. > > Not like sendmail does, though. qmail's quadratic backoff on retries > helps, as do its overall higher efficiency and its table of > nonresponding hosts (see "man qmail-tcpto"). > > >I can't really see how it cannot, through just the fact that it has > >to log future deferred attempts, etc. > > Logging with multilog is very cheap. > > >If you have a very large amount of mail that's constantly getting new > >messages for outgoing, those deferred messages when retried take up a > >qmail-remote process that could be dedicated to an actual deliverable mail > >instead of retrying something that was deferred and may get deferred again. > >I'd like slow mail, deferred or whatever on a host that's dedicated to > >retrying and not getting new mail. > > If you've got a spare host, why not split the load? What's the > advantage of shuffling deferred messages from one server to another? > That's a pretty expensive operation, even for qmail. > > >Even when new mail stops, qmail sits > >there and tries to deliver deferred mail until queuelifetime is > >exceeded. Why not have a host dedicated to those types of mails instead > >of bogging down your main mail machine. > > Most folks don't find deferred messages such a burden. They use up a > few qmail-remote's and some space in the queue, but that's no big > deal. Have you bumped up concurrencyremote to account for deferalls? > > >In my case it seems lke it would be useful. I'm delivering 1 - 2 million > >messages a day and a large percentage of that gets deferred. > > ``Profile. Don't speculate.'' > > What percentage of messages are deffered? How many attempts, on > average, does it take to deliver them? How much burden would be > shifted by delivering them to a fallback host after the first > deferral? Would a fallback host configuration be more > efficient/faster/more effective than a dual-host configuration? > > -Dave > http://www.xxedgexx.com | [EMAIL PROTECTED] -
Re: how do you use a deferral host in qmail?
Actually I've though of that and it would be easy enough, but the amount of error handling for failed messages would seem to be the real trick to calling qmail-remote directly. It would definitely make it fast, but I think there's enough in qmail that I don't want to reinvent the wheel in a lot of cases. I though about using qmqpc somehow to do what I want, but hmm...not sutre how to really piece it together or plan it logically. -jeremy > > I'd like slow mail, deferred or whatever on a host that's dedicated > > to retrying and not getting new mail. > > You may want to try a custom hack. I've heard that some high volume > sites call qmail-remote directly from the application that generates > the mail, then hand off messages that get soft failures. Often it's > enough just to hand them off to normal qmail, but I'd think it'd be > just as easy to pass them to another host using qmqp, using > qmail-qmqpc rather than qmail-queue. > > > > > -- > John R. Levine, IECC, POB 727, Trumansburg NY 14886 +1 607 387 6869 > [EMAIL PROTECTED], Village Trustee and Sewer Commissioner, http://iecc.com/johnl, > Member, Provisional board, Coalition Against Unsolicited Commercial E-mail > http://www.xxedgexx.com | [EMAIL PROTECTED] -
Re: how do you use a deferral host in qmail?
Jeremy Hansen <[EMAIL PROTECTED]> wrote: >Hmm, but qmail does get bogged down by deferrals from my experience. Not like sendmail does, though. qmail's quadratic backoff on retries helps, as do its overall higher efficiency and its table of nonresponding hosts (see "man qmail-tcpto"). >I can't really see how it cannot, through just the fact that it has >to log future deferred attempts, etc. Logging with multilog is very cheap. >If you have a very large amount of mail that's constantly getting new >messages for outgoing, those deferred messages when retried take up a >qmail-remote process that could be dedicated to an actual deliverable mail >instead of retrying something that was deferred and may get deferred again. >I'd like slow mail, deferred or whatever on a host that's dedicated to >retrying and not getting new mail. If you've got a spare host, why not split the load? What's the advantage of shuffling deferred messages from one server to another? That's a pretty expensive operation, even for qmail. >Even when new mail stops, qmail sits >there and tries to deliver deferred mail until queuelifetime is >exceeded. Why not have a host dedicated to those types of mails instead >of bogging down your main mail machine. Most folks don't find deferred messages such a burden. They use up a few qmail-remote's and some space in the queue, but that's no big deal. Have you bumped up concurrencyremote to account for deferalls? >In my case it seems lke it would be useful. I'm delivering 1 - 2 million >messages a day and a large percentage of that gets deferred. ``Profile. Don't speculate.'' What percentage of messages are deffered? How many attempts, on average, does it take to deliver them? How much burden would be shifted by delivering them to a fallback host after the first deferral? Would a fallback host configuration be more efficient/faster/more effective than a dual-host configuration? -Dave
Re: how do you use a deferral host in qmail?
> I'd like slow mail, deferred or whatever on a host that's dedicated > to retrying and not getting new mail. You may want to try a custom hack. I've heard that some high volume sites call qmail-remote directly from the application that generates the mail, then hand off messages that get soft failures. Often it's enough just to hand them off to normal qmail, but I'd think it'd be just as easy to pass them to another host using qmqp, using qmail-qmqpc rather than qmail-queue. -- John R. Levine, IECC, POB 727, Trumansburg NY 14886 +1 607 387 6869 [EMAIL PROTECTED], Village Trustee and Sewer Commissioner, http://iecc.com/johnl, Member, Provisional board, Coalition Against Unsolicited Commercial E-mail
Re: how do you use a deferral host in qmail?
Hmm, but qmail does get bogged down by deferrals from my experience. I can't really see how it cannot, through just the fact that it has to log future deferred attempts, etc. If you have a very large amount of mail that's constantly getting new messages for outgoing, those deferred messages when retried take up a qmail-remote process that could be dedicated to an actual deliverable mail instead of retrying something that was deferred and may get deferred again. I'd like slow mail, deferred or whatever on a host that's dedicated to retrying and not getting new mail. Even when new mail stops, qmail sits there and tries to deliver deferred mail until queuelifetime is exceeded. Why not have a host dedicated to those types of mails instead of bogging down your main mail machine. In my case it seems lke it would be useful. I'm delivering 1 - 2 million messages a day and a large percentage of that gets deferred. -jeremy > Jeremy Hansen <[EMAIL PROTECTED]> wrote: > > >I'm wondering if it's possible to setup a deferral host in qmail? If a > >message gets deferred, then the mail goes to another machine in charge of > >just retrying deferred messages instead of clogging up the main mailer > >machine. > > No, not out of the box. But it's not necessayr, because qmail doesn't > get bogged down by deferred messages. > > >Someone told me this is possible to do in sendmail. > > It is. > > >I haven't used sendmail in years so I don't know first hand, but it's > >been my impression that qmail can do anything sendmail can do. > > Not really... But then, not everything sendmail does is worth doing. > > -Dave > http://www.xxedgexx.com | [EMAIL PROTECTED] -
Re: how do you use a deferral host in qmail?
Jeremy Hansen <[EMAIL PROTECTED]> wrote: >I'm wondering if it's possible to setup a deferral host in qmail? If a >message gets deferred, then the mail goes to another machine in charge of >just retrying deferred messages instead of clogging up the main mailer >machine. No, not out of the box. But it's not necessayr, because qmail doesn't get bogged down by deferred messages. >Someone told me this is possible to do in sendmail. It is. >I haven't used sendmail in years so I don't know first hand, but it's >been my impression that qmail can do anything sendmail can do. Not really... But then, not everything sendmail does is worth doing. -Dave
how do you use a deferral host in qmail?
I'm wondering if it's possible to setup a deferral host in qmail? If a message gets deferred, then the mail goes to another machine in charge of just retrying deferred messages instead of clogging up the main mailer machine. Someone told me this is possible to do in sendmail. I haven't used sendmail in years so I don't know first hand, but it's been my impression that qmail can do anything sendmail can do. Thanks -jeremy
