Re: [Mailman-Users] Server Configuration 2.1.12
On 2/17/20 2:46 AM, Dennis Putnam wrote: > > [Mon Feb 17 03:43:04.099892 2020] [authz_core:error] [pid 30156] [client > 162.230.29.192:49655] AH01630: client denied by server configuration: > /usr/lib/mailman/cgi-bin/listinfo > > Here is my mailman.conf file: > > # > # httpd configuration settings for use with mailman. > # > > ScriptAlias /mailman/ /usr/lib/mailman/cgi-bin/ > > AllowOverride None > Options ExecCGI > Order allow,deny > Allow from all This changed in apache 2.4. Instead of the above 2 lines, you need Require all granted > > > > Alias /pipermail/ /usr/lib/mailman/archives/public/ > > Options Indexes MultiViews FollowSymLinks > AllowOverride None > Order allow,deny > Allow from all Same here > -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan signature.asc Description: OpenPGP digital signature -- Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Server Configuration..
* Mark Sapiro m...@msapiro.net: We cannot help you directly. Hotmail and Yahoo think you are a spammer. You have to negotiate with them directly and convince them that you aren't and that their customers want your mail and get them to whitelist your server. With Yahoo!, this is relatively easy: http://feedbackloop.yahoo.net/index.php?page=%2Fmanage.php With Hotmail you need to go to https://postmaster.live.com/ -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Server Configuration..
On 6/26/2011 11:41 PM, Ralf Hildebrandt wrote: With Yahoo!, this is relatively easy: http://feedbackloop.yahoo.net/index.php?page=%2Fmanage.php With Hotmail you need to go to https://postmaster.live.com/ Thanks Ralf. I added the Yahoo link to the FAQ at http://wiki.list.org/x/4oA9. I had added a Hotmail link yesterday, and I added a bit more info to it too. -- Mark Sapiro m...@msapiro.netThe highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Server Configuration..
* Mark Sapiro m...@msapiro.net: On 6/26/2011 11:41 PM, Ralf Hildebrandt wrote: With Yahoo!, this is relatively easy: http://feedbackloop.yahoo.net/index.php?page=%2Fmanage.php With Hotmail you need to go to https://postmaster.live.com/ Thanks Ralf. I added the Yahoo link to the FAQ at http://wiki.list.org/x/4oA9. I had added a Hotmail link yesterday, and I added a bit more info to it too. That's great. There's a lot a postmaster can do, like: * making sure forward/reverse lookups for your IP/hostname match * make sure your IP is not blacklisted (use http://multirbl.valli.org/lookup/ for a one-stop check) * setup SPF (if you like) * setup DKIM (prerequisite for some feedback loops!) -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Server Configuration..
* Khalil Abbas khillo...@hotmail.com: well I bought this server: RAM 1x 4 GB DDR2-RAM ECC HDD 2x 500 GB SATA II-HDD 7.200 rpm Barebone 1x Fujitsu PRIMERGY RX100 S5 CPU 1x Intel Xeon X3320 Quadcore I did set the SMTP_MAX_RCPT to 5 .. then I created a list of 10,000 members, n it delivered them in like 5-6 minutes .. This should probably be handled on the postfix-users list (or perhaphs I should ninja-CC Ralf... good idea!), but with the above hardware, even considering that the disks are rather low end, you really shouldn't have to limit the recipients per mail transaction or throttle the injection rate. I'd predict that delivery of a message to a large number of recipients will even be faster if there are _more_ recipients per message. Go with SMTP_MAX_RCPT=1000 in Mailman but limit the number of Postfix processes, start with 20 (postconf -e default_process_limit=20 postfix reload), see how that works out for you and increase the default_process_limit if it works well until performance gets worse. Any Postfix people care to comment? Cheers Stefan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Server Configuration..
Stefan Foerster wrote: Go with SMTP_MAX_RCPT=1000 in Mailman but limit the number of Postfix processes, start with 20 (postconf -e default_process_limit=20 postfix reload), see how that works out for you and increase the default_process_limit if it works well until performance gets worse. Part of the OP's original issue is that with the default SMTP_MAX_RCPTS = 500, messages delivered from Mailman to Postfix had hundreds of Hotmail (and/or Yahoo) envelope recipients. Postfix would apparently then deliver the message to Hotmail, again in one transaction with all the Hotmail envelope recipients, and Hotmail would not accept an envelope with that many recipients (and similarly for Yahoo). Probably, SMTP_MAX_RCPTS could be larger than 5 and still avoid the problem, but it probably can't be more than 20 or so. Of course, the real solution is for the OP to negotiate with Microsoft and Yahoo and maybe others to get his server whitelisted if he can. -- Mark Sapiro m...@msapiro.netThe highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Server Configuration..
this sucks! I've created a list of 100,000 subscribers .. SMTP_MAX_RCPTS=5, throttle worked perfectly, 800 emails per minute.. the messages should be delivered in a 2 hours time .. now it's 4 hours and 47,000 messages are stuck because hotmail n yahoo are deferring .. I read somewhere that the biggest list that's using mailman is 500,000 in size.. is it really true or only 50,000 messages are going thru n the rest is delayed or even bounced back.. I don't know what to do ... I did everything in the book.. our viewers had subscribed to our lists with double opt-in .. mailman is properly configured.. I don't know what yahoo and hotmail really want! I'm so desperate and disappointed.. **sigh** -Original Message- From: Mark Sapiro Sent: Sunday, June 26, 2011 5:59 AM To: Khalil Abbas ; mailman-users@python.org Subject: Re: [Mailman-Users] Server Configuration.. Khalil Abbas wrote: lol I'm not shouting, just too lazy to hit caps :) Regardless of why it happened, it comes across to others as shouting. well I bought this server: RAM 1x 4 GB DDR2-RAM ECC HDD 2x 500 GB SATA II-HDD 7.200 rpm Barebone 1x Fujitsu PRIMERGY RX100 S5 CPU 1x Intel Xeon X3320 Quadcore I did set the SMTP_MAX_RCPT to 5 .. then I created a list of 10,000 members, n it delivered them in like 5-6 minutes .. there was about 500 yahoo addresses and yahoo deferred a few messages.. luckily Hotmail accepted all 5500 addresses.. is that good? It's not terrific. At that rate, assuming everything had completely left your MTA in that time, it would take between 17 and 20 hours to deliver to 2 million recipients. Is that acceptable? Also, it is not clear that just because Hotmail accepted all 5500 addresses out of 10,000, that they will accept the next 55,000 out of the next 100,000 (if there are that many). Also, once Yahoo starts deferring, their deferral rate will increase if you keep sending more, and this will add to the load on your MTA. do you think getting 2 or 3 or even 4 servers like this one and splitting the 2 million users over them is better? I can't say. Perhaps the bottleneck is the bandwidth of your internet connection. If so, increasinfg the ability to push more data faster won't help at all. about the throttle patch, ok I confess, I AM IGNORANT! I have no idea how to install the patch :) I have mailman 2.1.9 by the way .. The throttle.patch.txt file attached to this message contains a 2.1.9 version of the patch. The only difference is the line number of the last portion. To install it, save the file and cd to your mailman installation and give the command patch Mailman/Handlers/SMTPDirect.py /path/to/throttle.patch.txt and then restart Mailman. If anything goes wrong, you can remove the patch with the command patch -R Mailman/Handlers/SMTPDirect.py /path/to/throttle.patch.txt followed by another restart of Mailman. -- Mark Sapiro m...@msapiro.netThe highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Server Configuration..
Khalil, Looking online, it looks like you might have to lower that number a bit... http://mailvivo.zendesk.com/entries/45256-my-email-is-taking-a-long-time-to-send-should-i-be-concerned http://www.broadbandreports.com/forum/r3241890-Mail-problems-to-Yahoo-Hotmail-from-Megapath http://serverfault.com/questions/236218/postfix-smtp-refused-to-talk-to-me-451-too-many-connection-attempts Your real answer though is to get ahold of yahoo and hotmail and get them to whitelist your server. It's futile to fight this on your end because you could spend days tweaking it to get things working, and the next day they could change their policy and break it all again. -Ryan Stasel Systems Administrator School of Journalism and Communication University of Oregon On Jun 26, 2011, at 13:33 , Khalil Abbas wrote: this sucks! I've created a list of 100,000 subscribers .. SMTP_MAX_RCPTS=5, throttle worked perfectly, 800 emails per minute.. the messages should be delivered in a 2 hours time .. now it's 4 hours and 47,000 messages are stuck because hotmail n yahoo are deferring .. I read somewhere that the biggest list that's using mailman is 500,000 in size.. is it really true or only 50,000 messages are going thru n the rest is delayed or even bounced back.. I don't know what to do ... I did everything in the book.. our viewers had subscribed to our lists with double opt-in .. mailman is properly configured.. I don't know what yahoo and hotmail really want! I'm so desperate and disappointed.. **sigh** -Original Message- From: Mark Sapiro Sent: Sunday, June 26, 2011 5:59 AM To: Khalil Abbas ; mailman-users@python.org Subject: Re: [Mailman-Users] Server Configuration.. Khalil Abbas wrote: lol I'm not shouting, just too lazy to hit caps :) Regardless of why it happened, it comes across to others as shouting. well I bought this server: RAM 1x 4 GB DDR2-RAM ECC HDD 2x 500 GB SATA II-HDD 7.200 rpm Barebone 1x Fujitsu PRIMERGY RX100 S5 CPU 1x Intel Xeon X3320 Quadcore I did set the SMTP_MAX_RCPT to 5 .. then I created a list of 10,000 members, n it delivered them in like 5-6 minutes .. there was about 500 yahoo addresses and yahoo deferred a few messages.. luckily Hotmail accepted all 5500 addresses.. is that good? It's not terrific. At that rate, assuming everything had completely left your MTA in that time, it would take between 17 and 20 hours to deliver to 2 million recipients. Is that acceptable? Also, it is not clear that just because Hotmail accepted all 5500 addresses out of 10,000, that they will accept the next 55,000 out of the next 100,000 (if there are that many). Also, once Yahoo starts deferring, their deferral rate will increase if you keep sending more, and this will add to the load on your MTA. do you think getting 2 or 3 or even 4 servers like this one and splitting the 2 million users over them is better? I can't say. Perhaps the bottleneck is the bandwidth of your internet connection. If so, increasinfg the ability to push more data faster won't help at all. about the throttle patch, ok I confess, I AM IGNORANT! I have no idea how to install the patch :) I have mailman 2.1.9 by the way .. The throttle.patch.txt file attached to this message contains a 2.1.9 version of the patch. The only difference is the line number of the last portion. To install it, save the file and cd to your mailman installation and give the command patch Mailman/Handlers/SMTPDirect.py /path/to/throttle.patch.txt and then restart Mailman. If anything goes wrong, you can remove the patch with the command patch -R Mailman/Handlers/SMTPDirect.py /path/to/throttle.patch.txt followed by another restart of Mailman. -- Mark Sapiro m...@msapiro.netThe highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/rstasel%40uoregon.edu -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Server Configuration..
Khalil Abbas wrote: this sucks! I've created a list of 100,000 subscribers .. SMTP_MAX_RCPTS=5, throttle worked perfectly, 800 emails per minute.. the messages should be delivered in a 2 hours time .. now it's 4 hours and 47,000 messages are stuck because hotmail n yahoo are deferring .. We cannot help you directly. Hotmail and Yahoo think you are a spammer. You have to negotiate with them directly and convince them that you aren't and that their customers want your mail and get them to whitelist your server. I read somewhere that the biggest list that's using mailman is 500,000 in size.. is it really true or only 50,000 messages are going thru n the rest is delayed or even bounced back.. It is probably true, but it is also probably true that there aren't hundreds of thousands of Hotmail addresses on that list. I don't know what to do ... I did everything in the book.. our viewers had subscribed to our lists with double opt-in .. mailman is properly configured.. I don't know what yahoo and hotmail really want! I'm so desperate and disappointed.. How are they supposed to know your list is double opt-in if you don't talk to them? See the FAQ at http://wiki.list.org/x/4oA9. You are going to have to contact them and convince them to whitelist you if you can. -- Mark Sapiro m...@msapiro.netThe highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Server Configuration..
On 06/24/2011 09:10 PM, Khalil Abbas wrote: Hi, we are a local TV station newspaper having a mailing list with 2+ Million Subscribers.. what is the best server configuration (CPU, RAM) that can handle this list? taking in consideration that only 1 daily email is sent to the subscribers.. Thank you .. I am sure others can answer better, but budget is the primary (and missing) consideration in this equation. The fastest and most CPU cores and as much memory as possible in a 64 bit system is the answer. If you MUST sacrifice cpu or memory, then balance as well as possible, and let more memory win over more CPU speed. -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Server Configuration..
Khalil Abbas wrote: we are a local TV station newspaper having a mailing list with 2+ Million Subscribers.. what is the best server configuration (CPU, RAM) that can handle this list? taking in consideration that only 1 daily email is sent to the subscribers.. Consider that likely much more than half the impact is going to be from the MTA rather than Mailman. Perhaps you should be asking on a list specific to your MTA. I would suggest at a minimum that you could reduce your Mailman impact by breaking up your list into perhaps 240 roughly equal sub-lists and sending the post to one list at a time at 6 minute intervals. Then if your system can complete delivery of the post from one sub-list to its 8500+- members in under 6 minutes, you should be OK. -- Mark Sapiro m...@msapiro.netThe highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Server Configuration..
SMTP_MAX_RCPTS =5 WOULD BE OK? OR SHOULD I REDUCE OR INCREASE IT FOR HOTMAIL AND YAHOO TO ACCEPT OUR MESSAGES? -Original Message- From: Mark Sapiro Sent: Saturday, June 25, 2011 8:39 PM To: Khalil Abbas ; mailman-users@python.org Subject: Re: [Mailman-Users] Server Configuration.. Khalil Abbas wrote: we are a local TV station newspaper having a mailing list with 2+ Million Subscribers.. what is the best server configuration (CPU, RAM) that can handle this list? taking in consideration that only 1 daily email is sent to the subscribers.. Consider that likely much more than half the impact is going to be from the MTA rather than Mailman. Perhaps you should be asking on a list specific to your MTA. I would suggest at a minimum that you could reduce your Mailman impact by breaking up your list into perhaps 240 roughly equal sub-lists and sending the post to one list at a time at 6 minute intervals. Then if your system can complete delivery of the post from one sub-list to its 8500+- members in under 6 minutes, you should be OK. -- Mark Sapiro m...@msapiro.netThe highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Server Configuration..
On 6/25/2011 11:15 AM, Khalil Abbas wrote: SMTP_MAX_RCPTS =5 WOULD BE OK? OR SHOULD I REDUCE OR INCREASE IT FOR HOTMAIL AND YAHOO TO ACCEPT OUR MESSAGES? Have you read the FAQ sections on performance and searched the archive for discussions? There have been a few about this. Can your ISP deal with the load? Do you have appropriate SPF/etc set up. IP addresses that aren't likely to be blacklisted? Server performance is only half the battle. z! -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Server Configuration..
Khalil Abbas wrote: SMTP_MAX_RCPTS =5 WOULD BE OK? OR SHOULD I REDUCE OR INCREASE IT FOR HOTMAIL AND YAHOO TO ACCEPT OUR MESSAGES? WHY ARE YOU SHOUTING? I have no real information as to why Hotmail and Yahoo were not accepting your mail, but apparently at one point I thought that reducing the number of Hotmail and/or Yahoo envelope recipients in a single message transaction might help and apparently, it did. Presumably, nothing at Hotmail and Yahoo has changed, so I wouldn't advise changing SMTP_MAX_RCPTS. What I would advise is before changing other things including hardware, that you try the Postfix changes recommended to you at http://mail.python.org/pipermail/mailman-users/2011-June/071671.html which you apparently never tried or at least never reported trying, and if that doesn't solve the problem completely, that you try the patch in the attached throttle.patch.txt file. The settings in this patch will limit Mailman to send to 2000 recipients per minute which will spread your 2+ million recipients over about 17 hours and will be as effective as making many small lists. Note that contrary to your assertion at http://mail.python.org/pipermail/mailman-users/2011-June/071720.html that the patch is not stable and not tested before, I never said it was not tested. The FAQ says it has been only minimally tested and is not guaranteed to work. What that means is it hasn't seen significant use and it may not solve your problem, but it is almost certainly more stable than rebooting the server every hour. And if you do try it and it does work, we can remove the caveats, and if it doesn't solve th problem, at least we'll know that. Free, open source software is a community effort, not a one-way street. -- Mark Sapiro m...@msapiro.netThe highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Server Configuration..
Mark Sapiro wrote:
What I would advise is ... that you try the
patch in the attached throttle.patch.txt file.
Sorry, I forgot the attachment. Here it is.
--
Mark Sapiro m...@msapiro.netThe highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan
--- test-mailman-2.1/Mailman/Handlers/SMTPDirect.py 2010-07-03
14:12:22.0 -0700
+++ test-mailman/Mailman/Handlers/SMTPDirect.py 2011-02-04 19:29:49.234375000
-0800
@@ -52,6 +52,37 @@
True = 1
False = 0
+# Throttling settings. Do not send to more than THROTTLE_LIMIT recipients
+# within THROTTLE_TIME seconds. Set THROTTLE_LIMIT to 0 to not throttle.
+THROTTLE_TIME = 60
+THROTTLE_LIMIT = 2000
+THROTTLE_DATA = []
+
+# check throttling. Doesn't return until total recipients in THROTTLE_TIME
+# = THROTTLE_LIMIT.
+#
+def check_throttle(count):
+global THROTTLE_DATA
+if THROTTLE_LIMIT = 0:
+return
+if count = 0 or count THROTTLE_LIMIT:
+syslog('error', 'check_throttle: count=%d; THROTTLE_LIMIT=%d',
+ count, THROTTLE_LIMIT)
+return
+now = time.time()
+total = 0
+THROTTLE_DATA = [(num, tim) for (num, tim) in THROTTLE_DATA
+ if tim now - THROTTLE_TIME]
+for num, tim in THROTTLE_DATA:
+total += num
+total += count
+if total THROTTLE_LIMIT:
+time.sleep(THROTTLE_TIME - now + THROTTLE_DATA[0][1])
+check_throttle(count)
+else:
+THROTTLE_DATA.append((count, now))
+return
+
# Manage a connection to the SMTP server
@@ -118,10 +149,14 @@
chunks = [[recip] for recip in recips]
msgdata['personalize'] = 1
deliveryfunc = verpdeliver
-elif mm_cfg.SMTP_MAX_RCPTS = 0:
+elif mm_cfg.SMTP_MAX_RCPTS = 0 and (THROTTLE_LIMIT =0 or
+ THROTTLE_LIMIT = len(recips)):
chunks = [recips]
else:
-chunks = chunkify(recips, mm_cfg.SMTP_MAX_RCPTS)
+if THROTTLE_LIMIT = 0 or THROTTLE_LIMIT mm_cfg.SMTP_MAX_RCPTS:
+chunks = chunkify(recips, mm_cfg.SMTP_MAX_RCPTS)
+else:
+chunks = chunkify(recips, THROTTLE_LIMIT)
# See if this is an unshunted message for which some were undelivered
if msgdata.has_key('undelivered'):
chunks = msgdata['undelivered']
@@ -384,6 +419,8 @@
msgid = msg['message-id']
try:
# Send the message
+# but wait if necessary for throttling
+check_throttle(len(recips))
refused = conn.sendmail(envsender, recips, msgtext)
except smtplib.SMTPRecipientsRefused, e:
syslog('smtp-failure', 'All recipients refused: %s, msgid: %s',
--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe:
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Server Configuration..
lol I'm not shouting, just too lazy to hit caps :) well I bought this server: RAM 1x 4 GB DDR2-RAM ECC HDD 2x 500 GB SATA II-HDD 7.200 rpm Barebone 1x Fujitsu PRIMERGY RX100 S5 CPU 1x Intel Xeon X3320 Quadcore I did set the SMTP_MAX_RCPT to 5 .. then I created a list of 10,000 members, n it delivered them in like 5-6 minutes .. there was about 500 yahoo addresses and yahoo deferred a few messages.. luckily Hotmail accepted all 5500 addresses.. is that good? do you think getting 2 or 3 or even 4 servers like this one and splitting the 2 million users over them is better? about the throttle patch, ok I confess, I AM IGNORANT! I have no idea how to install the patch :) I have mailman 2.1.9 by the way .. Thanks .. -Original Message- From: Mark Sapiro Sent: Saturday, June 25, 2011 10:40 PM To: Khalil Abbas ; mailman-users@python.org Subject: Re: [Mailman-Users] Server Configuration.. Khalil Abbas wrote: SMTP_MAX_RCPTS =5 WOULD BE OK? OR SHOULD I REDUCE OR INCREASE IT FOR HOTMAIL AND YAHOO TO ACCEPT OUR MESSAGES? WHY ARE YOU SHOUTING? I have no real information as to why Hotmail and Yahoo were not accepting your mail, but apparently at one point I thought that reducing the number of Hotmail and/or Yahoo envelope recipients in a single message transaction might help and apparently, it did. Presumably, nothing at Hotmail and Yahoo has changed, so I wouldn't advise changing SMTP_MAX_RCPTS. What I would advise is before changing other things including hardware, that you try the Postfix changes recommended to you at http://mail.python.org/pipermail/mailman-users/2011-June/071671.html which you apparently never tried or at least never reported trying, and if that doesn't solve the problem completely, that you try the patch in the attached throttle.patch.txt file. The settings in this patch will limit Mailman to send to 2000 recipients per minute which will spread your 2+ million recipients over about 17 hours and will be as effective as making many small lists. Note that contrary to your assertion at http://mail.python.org/pipermail/mailman-users/2011-June/071720.html that the patch is not stable and not tested before, I never said it was not tested. The FAQ says it has been only minimally tested and is not guaranteed to work. What that means is it hasn't seen significant use and it may not solve your problem, but it is almost certainly more stable than rebooting the server every hour. And if you do try it and it does work, we can remove the caveats, and if it doesn't solve th problem, at least we'll know that. Free, open source software is a community effort, not a one-way street. -- Mark Sapiro m...@msapiro.netThe highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Server Configuration..
Khalil Abbas wrote:
lol I'm not shouting, just too lazy to hit caps :)
Regardless of why it happened, it comes across to others as shouting.
well I bought this server:
RAM 1x 4 GB DDR2-RAM ECC
HDD 2x 500 GB SATA II-HDD 7.200 rpm
Barebone 1x Fujitsu PRIMERGY RX100 S5
CPU 1x Intel Xeon X3320 Quadcore
I did set the SMTP_MAX_RCPT to 5 .. then I created a list of 10,000 members,
n it delivered them in like 5-6 minutes ..
there was about 500 yahoo addresses and yahoo deferred a few messages..
luckily Hotmail accepted all 5500 addresses..
is that good?
It's not terrific. At that rate, assuming everything had completely
left your MTA in that time, it would take between 17 and 20 hours to
deliver to 2 million recipients. Is that acceptable?
Also, it is not clear that just because Hotmail accepted all 5500
addresses out of 10,000, that they will accept the next 55,000 out of
the next 100,000 (if there are that many). Also, once Yahoo starts
deferring, their deferral rate will increase if you keep sending more,
and this will add to the load on your MTA.
do you think getting 2 or 3 or even 4 servers like this one and splitting
the 2 million users over them is better?
I can't say. Perhaps the bottleneck is the bandwidth of your internet
connection. If so, increasinfg the ability to push more data faster
won't help at all.
about the throttle patch, ok I confess, I AM IGNORANT! I have no idea how to
install the patch :)
I have mailman 2.1.9 by the way ..
The throttle.patch.txt file attached to this message contains a 2.1.9
version of the patch. The only difference is the line number of the
last portion.
To install it, save the file and cd to your mailman installation and
give the command
patch Mailman/Handlers/SMTPDirect.py /path/to/throttle.patch.txt
and then restart Mailman.
If anything goes wrong, you can remove the patch with the command
patch -R Mailman/Handlers/SMTPDirect.py /path/to/throttle.patch.txt
followed by another restart of Mailman.
--
Mark Sapiro m...@msapiro.netThe highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan
--- SMTPDirect.py.orig 2011-06-25 17:39:07.859375000 -0700
+++ SMTPDirect.py 2011-06-25 17:43:04.078125000 -0700
@@ -52,6 +52,37 @@
True = 1
False = 0
+# Throttling settings. Do not send to more than THROTTLE_LIMIT recipients
+# within THROTTLE_TIME seconds. Set THROTTLE_LIMIT to 0 to not throttle.
+THROTTLE_TIME = 60
+THROTTLE_LIMIT = 2000
+THROTTLE_DATA = []
+
+# check throttling. Doesn't return until total recipients in THROTTLE_TIME
+# = THROTTLE_LIMIT.
+#
+def check_throttle(count):
+global THROTTLE_DATA
+if THROTTLE_LIMIT = 0:
+return
+if count = 0 or count THROTTLE_LIMIT:
+syslog('error', 'check_throttle: count=%d; THROTTLE_LIMIT=%d',
+ count, THROTTLE_LIMIT)
+return
+now = time.time()
+total = 0
+THROTTLE_DATA = [(num, tim) for (num, tim) in THROTTLE_DATA
+ if tim now - THROTTLE_TIME]
+for num, tim in THROTTLE_DATA:
+total += num
+total += count
+if total THROTTLE_LIMIT:
+time.sleep(THROTTLE_TIME - now + THROTTLE_DATA[0][1])
+check_throttle(count)
+else:
+THROTTLE_DATA.append((count, now))
+return
+
# Manage a connection to the SMTP server
@@ -118,10 +149,14 @@
chunks = [[recip] for recip in recips]
msgdata['personalize'] = 1
deliveryfunc = verpdeliver
-elif mm_cfg.SMTP_MAX_RCPTS = 0:
+elif mm_cfg.SMTP_MAX_RCPTS = 0 and (THROTTLE_LIMIT =0 or
+ THROTTLE_LIMIT = len(recips)):
chunks = [recips]
else:
-chunks = chunkify(recips, mm_cfg.SMTP_MAX_RCPTS)
+if THROTTLE_LIMIT = 0 or THROTTLE_LIMIT mm_cfg.SMTP_MAX_RCPTS:
+chunks = chunkify(recips, mm_cfg.SMTP_MAX_RCPTS)
+else:
+chunks = chunkify(recips, THROTTLE_LIMIT)
# See if this is an unshunted message for which some were undelivered
if msgdata.has_key('undelivered'):
chunks = msgdata['undelivered']
@@ -366,6 +401,8 @@
msgid = msg['message-id']
try:
# Send the message
+# but wait if necessary for throttling
+check_throttle(len(recips))
refused = conn.sendmail(envsender, recips, msgtext)
except smtplib.SMTPRecipientsRefused, e:
syslog('smtp-failure', 'All recipients refused: %s, msgid: %s',
--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe:
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
