RE: dealing with Yahoo slowness
> -Original Message- > From: owner-postfix-us...@postfix.org [mailto:owner-postfix- > us...@postfix.org] On Behalf Of Florin Andrei > Sent: Tuesday, 15 June 2010 6:00 a.m. > To: postfix-users@postfix.org > Subject: Re: dealing with Yahoo slowness > > On 06/10/2010 05:09 PM, Mike Hutchinson wrote: > > yahoo_destination_concurrency_limit = 4 > > yahoo_destination_rate_delay = 1s > > Well, we do that already (concurrency = 2, rate_delay = 2s). It's still > slow. Do you use multiple outbound email gateways? > > Maybe I should try to increase our existing parameters, it looks like > we're using half your values. [Michael Hutchinson] made a late reply: Sounds like you've run into the version problem I had some time ago, where the rate controls were present, but were a bit buggy. See Wietse's post if you haven't already. Once we'd performed the upgrade, and applied the rate limiting configuration everything went smoothly - perhaps try the same values from the original post and work from there. Cheers, Michael.
RE: dealing with Yahoo slowness
> > # Slow these destinations to avoid blacklisting, see /etc/postfix/transport > > for domains configured > > hotmail_destination_concurrency_limit = 2 > > hotmail_destination_rate_delay = 2s > > hotmail_destination_recipient_limit = 5 > > yahoo_destination_concurrency_limit = 4 > > yahoo_destination_rate_delay = 1s > > yahoo_destination_recipient_limit = 5 > > > > These settings can be tweaked depending on what server you're talking to. > > However, these values work for us, after having dealt with not getting > > 10,000 mails out per week. > > > > I hope this helps. > > Interesting. Really. [Michael Hutchinson] It is. I don't know of any other software that even comes close to supporting rate limiting for outgoing E-Mail. As I understand it, even companies that package Postfix in their NetApps are taking advantage of these features now. > FYI This should be documented better: Postfix's _rate_delay feature > forces a per-destination delivery concurrency of 1, so you could > drop the _destination_concurrency_limit settings. The Postfix > implementation is utterly simple: schedule one delivery, then > suspend delivery for N second, then schedule the next delivery. [Michael Hutchinson] I had thought, whilst I was writing the E-Mail, that this could deserve a howto or manual section, perhaps briefly describing a general situation that would reflect the real world problem of delivery of E-Mail to servers like Yahoo/Google, and how postfix can be configured to react in a more Yahoo/Google restricted manner. What assistance can I provide ? Cheers, Michael Hutchinson mhutchin...@manux.co.nz
RE: dealing with Yahoo slowness
> -Original Message- > Florin Andrei a écrit : > > There seems to be a massive difference between the speed of various > > providers, in terms of accepting messages for delivery. Yahoo stands out > > as, by far, the slowest of the big ones. > > > > Because the messages are legitimate, we signed up for the email feedback > > loop with Yahoo, so that messages flagged as spam by Yahoo users are > > reported back to us, so we can silence notifications for those accounts. > > That didn't seem to help. > > > > Messages @yahoo.com just accumulate in the deferred queue and stay > there > > for a long time. When they do get kicked back into the active queue, > > it's just a trickle. > > > > Everyone else's emails are delivered very quickly. It might be my > > imagination, but it seems some providers do accept messages more > quickly > > if you use the email feedback loop. > > > > Anyway, after a while, we end up with a bunch of @yahoo.com messages > in > > the queue, some domain typos, and not much else besides. > > > > One of the tricks some people seem to use is creating a dedicated > > transport for the slow destination. I'm reading the tuning and qshape > > README documents, and there are a lot of good suggestions there, but I > > was wondering what are the solutions that are being used *now* > > specifically for dealing with Yahoo. > > > > it looks like yahoo throttle you. > - if you send few mail, do nothing. just wait and things will hopefully > get better. [Michael Hutchinson] Only OK if you're sending one or two E-Mails. > - if on the other hand you send a lot of mail, then you need to get in > touch with yahoo. there's nothing we can do to help you. [Michael Hutchinson] This also does very little. Yahoo don't list a Network Operations Centre and are about the hardest company to deal with when it comes to E-Mail delivery resolution. I've filled out dozens of forms to get particular IP's whitelisted for specific domains on their Mail systems, with no reply, let alone result. > one thing you can do is to ask your recipients to explicitely tag your > mail as "not spam" if it was ever tagged as spam. or they can reply (of > course, this doesn't work for silly noreply mail. [sigh, I just got an > internal one with a URL that doesn't work, and I don't know whom to > contact...]). [Michael Hutchinson] Unfortunately it only takes ONE user to click on "This is Spam" and you're back to square one. (and being IP banned is Square one). [Michael Hutchinson] We have had this exact problem, delivering Retail newsletters to people who have opted in for it. A lot of them are on Gmail and Yahoo, and this can be difficult with Bulk E-Mail. Despite contact with Google themselves and signing up for all of their reporting services regarding Spammy Emails and Certified Senders, the best result we've had is to use some Postfix configuration to resolve the issue. It does this by gently delivering the E-Mails at an acceptable rate (discovered with a LOT of testing and a LOT of IP bans (good they're not permanent, huh :). In our environment, on our servers, this has resolved the issue, and delivers mail to those domains a LOT faster than not performing the config on Postfix. In fact, if we dont configure, we get banned straight away against those domains and cannot deliver for several hours afterwards. We setup the postfix transport file with these entries: # destination domains that need to be rate limited hotmail.com hotmail: msn.com hotmail: live.comhotmail: windowslive.com hotmail: yahoo.com.aryahoo: yahoo.com.auyahoo: yahoo.com.bryahoo: yahoo.cayahoo: yahoo.com.cnyahoo: - there's more but you get the idea. Then we setup master.cf: yahoo unix - - - - - smtp hotmail unix - - - - - smtp Then setup main.cf: # Slow these destinations to avoid blacklisting, see /etc/postfix/transport for domains configured hotmail_destination_concurrency_limit = 2 hotmail_destination_rate_delay = 2s hotmail_destination_recipient_limit = 5 yahoo_destination_concurrency_limit = 4 yahoo_destination_rate_delay = 1s yahoo_destination_recipient_limit = 5 These settings can be tweaked depending on what server you're talking to. However, these values work for us, after having dealt with not getting 10,000 mails out per week. I hope this helps. Cheers, Michael Hutchinson mhutchin...@manux.co.nz
RE: Mail to local domains
> > Currently we use postfixadmin for this, but this is undesirable as it will > > often double-up or triple-up entire sends (we intend to fix this, > > eventually). I know the previous system we used was basically a script that > > copied a file into everyone's Inbox under their Maildir folder - we no > > longer have the script, so that idea is out. > > This is not a postfix question. [Michael Hutchinson] Yes, but postfix people will know how to do it :) - sorry for hijacking the list. Thanks for the reply, anyway! > However, a script is almost certainly the easiest way to do this, and it's quite > trivial if all your users are using Maildirs. > > I use something like this (run via cron out of the user's own crontab) > > ORG="Name of our company/Organization" > BODY="text of the email I want to send" > DATE=`date '+%a, %d %b %Y %T %z'` > > echo "Subject: This is my broadcast email"; \ > echo "Date: $DATE"; \ > echo "From: $ORG Broadcast Service ";\ > echo ; \ > echo "$BODY") | /usr/local/maildrop -d > [Michael Hutchinson] I must admit, I've not seen people do mail broadcasts from Cron before - quite clever. > maildrop makes putting mail into maildirs trivial if you're using Courier. If > you're using something else. well, I'm sure there's something similar. Your > best bet is to ask this on the list for your mail package. [Michael Hutchinson] Well, we ended up using getting a Perl wizard to assist. We're now using a perl script, pulling real Mail addresses from MySQL, and using a Perl Mail module to submit the mail into the postfix queue. As this system hosts under 1000 mail addresses, we don't have to worry about message copies so much. If this box was hosting a couple of thousand mailboxes, I'd say we'd have gone with linking the broadcast message into their Inbox, to save mail delivery routines and load. At any rate, we've got what we need now. I thought I'd better be polite and post the script we're now using, as someone else may find it useful: http://pastebin.com/6PDAwRzq Credit goes to Bjorn Nilsen for writing the script. Cheers, Michael Hutchinson Manux Solutions Limited mhutchin...@manux.co.nz
Mail to local domains
Hello all, I am writing to ask for a procedure for sending a broadcast E-Mail to locally hosted domains on a postfix system. Currently we use postfixadmin for this, but this is undesirable as it will often double-up or triple-up entire sends (we intend to fix this, eventually). I know the previous system we used was basically a script that copied a file into everyone's Inbox under their Maildir folder - we no longer have the script, so that idea is out. I've checked out using maildir-bulletin, as this is available in software repo's (ubuntu). The problem with using this is that our hosted mailboxes have been setup thusly: /home/vmail/domain.name/mailbox.name. With all domain.name folders owned by vmail:vmail. Where the intended use of maildir-bulletin is to pick a group to send to, and mail is delivered to members of that group, under their Maildir folder. Sounds like a recipe for disaster, if I run this up on our system. Has anyone had success with this sort of system, or am I simply barking up the wrong tree? Please help, Thanks in advance, Yours Sincerely, Michael Hutchinson Manux Solutions Limited mhutchin...@manux.co.nz
RE: Rate control for SMTP delivery to speicific domain
> >> > >> What version of Postfix is this? > > > > Postfix mail_version = 2.5.1 > > That's what I suspected. Upgrade; see RELEASE_NOTES for > destination_rate_delay fix last year. Also show logs confirming your > special transport is actually being used. Excellent. Sounds like I have my answer. Thanks for the info, everyone, including Wietse. Hopefully all will be resolved today. Cheers, Michael Hutchinson.
RE: Rate control for SMTP delivery to speicific domain
> > > >>> And setting the wakeup timer would be like this: > >>> smtphotmail unix- - - 3 - smtp > >> > >> That is how you set the timer. > > > > Yes, not what I am doing / want to do. > > > >>> My apologies. Ill go back and start from scratch. > >> > >> What version of Postfix is this? > > > > Postfix mail_version = 2.5.1 > > > > Cheers, > > Michael Hutchinson > > > > > > Postfix 2.5.1 is pretty old, the current 2.5 patchlevel is 10. > There have been some changes that could affect your rate > delay controls. > > Anyway, for diagnostics you can add a '-o > syslog_name=smtphotmail' to your master.cf entry to verify the > correct transport is being used, or for more info add a -v to > your smtphotmail transport to see what it's up to. > > Also see the "slow" transport example under > http://www.postfix.org/QSHAPE_README.html#backlog Ahh, some direct logging would be nice. Thanks for the information and links. I will also check out whether my postfix needs to be upgraded. Cheers, Michael Hutchinson
RE: Rate control for SMTP delivery to speicific domain
> > > > What version of Postfix is this? > > > > Postfix mail_version = 2.5.1 > > The rate control features introduced in 2.5.0 were improved in later > patches, you must upgrade to the latest 2.5 release if you want to > enforce inter-message delays. > Sounds interesting. I shall investigate and persue upgrading our Postfix to a more recent version. Thanks for the information Cheers, Michael Hutchinson
RE: Rate control for SMTP delivery to speicific domain
> -Original Message- > From: owner-postfix-us...@postfix.org [mailto:owner-postfix- > us...@postfix.org] On Behalf Of Sahil Tandon > Sent: Tuesday, 30 March 2010 3:07 p.m. > To: postfix-users@postfix.org > Subject: Re: Rate control for SMTP delivery to speicific domain > > On Tue, 30 Mar 2010, Mike Hutchinson wrote: > > > As I understand it, this would be setting the Max Processes limit. > > smtphotmail unix- - - - 3 smtp > > That is how you set maxproc, but why do you need 3 for this transport? This transport points to a rather fragile mail server, in other words, we are going over their acceptable delivery rate and our Server IP gets banned unless we do something about rate control. I assumed the Max Processes would assist with this, but perhaps I am wrong. > > And setting the wakeup timer would be like this: > > smtphotmail unix- - - 3 - smtp > > That is how you set the timer. Yes, not what I am doing / want to do. > > My apologies. Ill go back and start from scratch. > > What version of Postfix is this? Postfix mail_version = 2.5.1 Cheers, Michael Hutchinson
RE: Rate control for SMTP delivery to speicific domain
> > > > smtphotmail unix - - - - 3 > smtp > > Who told you to set a wakeup timer of 3 seconds? Remove it. No-one did. I had intended to set a max processes limit.. As I understand it, this would be setting the Max Processes limit. smtphotmail unix- - - - 3 smtp And setting the wakeup timer would be like this: smtphotmail unix- - - 3 - smtp > > Transport: > > hotmail.com smtphotmail:hotmail.com > > How do you know that Postfix uses this? The cut-and-paste fragments > from the main.cf file show no evidence of that. I have the transport referenced in main.cf, and have used postmap to build the db. > As requested in the mailing list welcome message (see below) do > NOT send cut-and-paste fragments from the main.cf file. My apologies. Ill go back and start from scratch. Cheers, Michael
Rate control for SMTP delivery to speicific domain
Hello Everyone. Our company sends out newsletters to people who have subscribed their mail address in-store (retail). I have been working in attempt to slow down E-Mail deliveries to Hotmail, as our server attempts deliveries too quickly and will get blocked by their servers. I have googled and found some decent information and have even applied it, but it does not appear to do a single thing to slow delivery down for the domain. I have in my configuration files: Master.cf: smtphotmail unix - - - - 3 smtp Transport: hotmail.com smtphotmail:hotmail.com Main.cf: smtphotmail_destination_concurrency_limit = 2 smtphotmail_destination_rate_delay = 3s I was using the global limiters and these were working just fine - we were able to deliver OK, until I commented them out: #initial_destination_concurrency = 3 #local_destination_concurrency_limit = 3 #smtp_destination_concurrency_limit = 3 #smtp_destination_rate_delay = 2s It appears that whenever I use the smtphotmail transport rules, instead of the global ones, delivery attempts to the domain increase as if there was no limitation.. Ie: after a Postfix reload, I follow the mail.info log and find straight after the reload that there are instantly pages and pages of delivery attempts to hotmail.com. To me this doesn't make sense - the rate limit means I should only see a few, every couple of seconds, right? Am I missing something here? Thanks in advance, Yours Sincerely, Michael Hutchinson
