Re: [Mailman-Users] Server Configuration..
On 06/24/2011 09:10 PM, Khalil Abbas wrote: Hi, we are a local TV station newspaper having a mailing list with 2+ Million Subscribers.. what is the best server configuration (CPU, RAM) that can handle this list? taking in consideration that only 1 daily email is sent to the subscribers.. Thank you .. I am sure others can answer better, but budget is the primary (and missing) consideration in this equation. The fastest and most CPU cores and as much memory as possible in a 64 bit system is the answer. If you MUST sacrifice cpu or memory, then balance as well as possible, and let more memory win over more CPU speed. -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
[Mailman-Users] Mailman, Postfix virtual domains
Hi all. I have Centos 5.5 server with Postfix, configured to work in AD, and i want to add the mailman aliases: main.cf: virtual_alias_maps = ldap:/etc/postfix/ldap-groups.cf, ldap:/etc/postfix/ldap-groups2.cf, hash:/etc/postfix/valias /etc/postfix/aliases looks like: mailman@pmz.local |/usr/lib/mailman/mail/mailman post mailman But i recieved this: |/usr/lib/mailman/mail/mailman post mailman@melchior.kolos.local (expanded from mailman@pmz.local): unknown user: |/usr/lib/mailman/mail/mailman post mailman How can i connect Mailman to Postfix? P.S. Sorry for my English :) -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Mailman, Postfix virtual domains
On 6/24/2011 6:29 AM, Aranel Doriath wrote: Hi all. I have Centos 5.5 server with Postfix, configured to work in AD, and i want to add the mailman aliases: main.cf: virtual_alias_maps = ldap:/etc/postfix/ldap-groups.cf, ldap:/etc/postfix/ldap-groups2.cf, hash:/etc/postfix/valias /etc/postfix/aliases looks like: mailman@pmz.local |/usr/lib/mailman/mail/mailman post mailman This alias is not correct. You should consider Mailman/Postfix integration as discussed at http://www.list.org/mailman-install/node12.html. In any case, the left part of the alias (in alias_maps) should be a local_part only and terminated with a colon as mailman:|/usr/lib/mailman/mail/mailman post mailman If you need virtual mappings (virtual_alias_maps), those entries look like mailman@pmz.local mailman and map the virtual domain address mailman@pmz.local to the local address mailman which is in turn mapped to the pipe by alias_maps. But i recieved this: |/usr/lib/mailman/mail/mailman post mailman@melchior.kolos.local (expanded from mailman@pmz.local): unknown user: |/usr/lib/mailman/mail/mailman post mailman How can i connect Mailman to Postfix? It is hard to say for sure without at least the output of 'postconf -n', but it looks like you may have a file containing aliases that belongs in alias_maps included in virtual_alias_maps instead or in addition. There are many threads on this subject in the archives of this list. See the FAQ at http://wiki.list.org/x/OIA9 for information on searching the archives. -- Mark Sapiro m...@msapiro.netThe highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Server Configuration..
Khalil Abbas wrote: we are a local TV station newspaper having a mailing list with 2+ Million Subscribers.. what is the best server configuration (CPU, RAM) that can handle this list? taking in consideration that only 1 daily email is sent to the subscribers.. Consider that likely much more than half the impact is going to be from the MTA rather than Mailman. Perhaps you should be asking on a list specific to your MTA. I would suggest at a minimum that you could reduce your Mailman impact by breaking up your list into perhaps 240 roughly equal sub-lists and sending the post to one list at a time at 6 minute intervals. Then if your system can complete delivery of the post from one sub-list to its 8500+- members in under 6 minutes, you should be OK. -- Mark Sapiro m...@msapiro.netThe highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Server Configuration..
SMTP_MAX_RCPTS =5 WOULD BE OK? OR SHOULD I REDUCE OR INCREASE IT FOR HOTMAIL AND YAHOO TO ACCEPT OUR MESSAGES? -Original Message- From: Mark Sapiro Sent: Saturday, June 25, 2011 8:39 PM To: Khalil Abbas ; mailman-users@python.org Subject: Re: [Mailman-Users] Server Configuration.. Khalil Abbas wrote: we are a local TV station newspaper having a mailing list with 2+ Million Subscribers.. what is the best server configuration (CPU, RAM) that can handle this list? taking in consideration that only 1 daily email is sent to the subscribers.. Consider that likely much more than half the impact is going to be from the MTA rather than Mailman. Perhaps you should be asking on a list specific to your MTA. I would suggest at a minimum that you could reduce your Mailman impact by breaking up your list into perhaps 240 roughly equal sub-lists and sending the post to one list at a time at 6 minute intervals. Then if your system can complete delivery of the post from one sub-list to its 8500+- members in under 6 minutes, you should be OK. -- Mark Sapiro m...@msapiro.netThe highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Server Configuration..
On 6/25/2011 11:15 AM, Khalil Abbas wrote: SMTP_MAX_RCPTS =5 WOULD BE OK? OR SHOULD I REDUCE OR INCREASE IT FOR HOTMAIL AND YAHOO TO ACCEPT OUR MESSAGES? Have you read the FAQ sections on performance and searched the archive for discussions? There have been a few about this. Can your ISP deal with the load? Do you have appropriate SPF/etc set up. IP addresses that aren't likely to be blacklisted? Server performance is only half the battle. z! -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Server Configuration..
Khalil Abbas wrote: SMTP_MAX_RCPTS =5 WOULD BE OK? OR SHOULD I REDUCE OR INCREASE IT FOR HOTMAIL AND YAHOO TO ACCEPT OUR MESSAGES? WHY ARE YOU SHOUTING? I have no real information as to why Hotmail and Yahoo were not accepting your mail, but apparently at one point I thought that reducing the number of Hotmail and/or Yahoo envelope recipients in a single message transaction might help and apparently, it did. Presumably, nothing at Hotmail and Yahoo has changed, so I wouldn't advise changing SMTP_MAX_RCPTS. What I would advise is before changing other things including hardware, that you try the Postfix changes recommended to you at http://mail.python.org/pipermail/mailman-users/2011-June/071671.html which you apparently never tried or at least never reported trying, and if that doesn't solve the problem completely, that you try the patch in the attached throttle.patch.txt file. The settings in this patch will limit Mailman to send to 2000 recipients per minute which will spread your 2+ million recipients over about 17 hours and will be as effective as making many small lists. Note that contrary to your assertion at http://mail.python.org/pipermail/mailman-users/2011-June/071720.html that the patch is not stable and not tested before, I never said it was not tested. The FAQ says it has been only minimally tested and is not guaranteed to work. What that means is it hasn't seen significant use and it may not solve your problem, but it is almost certainly more stable than rebooting the server every hour. And if you do try it and it does work, we can remove the caveats, and if it doesn't solve th problem, at least we'll know that. Free, open source software is a community effort, not a one-way street. -- Mark Sapiro m...@msapiro.netThe highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Server Configuration..
Mark Sapiro wrote:
What I would advise is ... that you try the
patch in the attached throttle.patch.txt file.
Sorry, I forgot the attachment. Here it is.
--
Mark Sapiro m...@msapiro.netThe highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan
--- test-mailman-2.1/Mailman/Handlers/SMTPDirect.py 2010-07-03
14:12:22.0 -0700
+++ test-mailman/Mailman/Handlers/SMTPDirect.py 2011-02-04 19:29:49.234375000
-0800
@@ -52,6 +52,37 @@
True = 1
False = 0
+# Throttling settings. Do not send to more than THROTTLE_LIMIT recipients
+# within THROTTLE_TIME seconds. Set THROTTLE_LIMIT to 0 to not throttle.
+THROTTLE_TIME = 60
+THROTTLE_LIMIT = 2000
+THROTTLE_DATA = []
+
+# check throttling. Doesn't return until total recipients in THROTTLE_TIME
+# = THROTTLE_LIMIT.
+#
+def check_throttle(count):
+global THROTTLE_DATA
+if THROTTLE_LIMIT = 0:
+return
+if count = 0 or count THROTTLE_LIMIT:
+syslog('error', 'check_throttle: count=%d; THROTTLE_LIMIT=%d',
+ count, THROTTLE_LIMIT)
+return
+now = time.time()
+total = 0
+THROTTLE_DATA = [(num, tim) for (num, tim) in THROTTLE_DATA
+ if tim now - THROTTLE_TIME]
+for num, tim in THROTTLE_DATA:
+total += num
+total += count
+if total THROTTLE_LIMIT:
+time.sleep(THROTTLE_TIME - now + THROTTLE_DATA[0][1])
+check_throttle(count)
+else:
+THROTTLE_DATA.append((count, now))
+return
+
# Manage a connection to the SMTP server
@@ -118,10 +149,14 @@
chunks = [[recip] for recip in recips]
msgdata['personalize'] = 1
deliveryfunc = verpdeliver
-elif mm_cfg.SMTP_MAX_RCPTS = 0:
+elif mm_cfg.SMTP_MAX_RCPTS = 0 and (THROTTLE_LIMIT =0 or
+ THROTTLE_LIMIT = len(recips)):
chunks = [recips]
else:
-chunks = chunkify(recips, mm_cfg.SMTP_MAX_RCPTS)
+if THROTTLE_LIMIT = 0 or THROTTLE_LIMIT mm_cfg.SMTP_MAX_RCPTS:
+chunks = chunkify(recips, mm_cfg.SMTP_MAX_RCPTS)
+else:
+chunks = chunkify(recips, THROTTLE_LIMIT)
# See if this is an unshunted message for which some were undelivered
if msgdata.has_key('undelivered'):
chunks = msgdata['undelivered']
@@ -384,6 +419,8 @@
msgid = msg['message-id']
try:
# Send the message
+# but wait if necessary for throttling
+check_throttle(len(recips))
refused = conn.sendmail(envsender, recips, msgtext)
except smtplib.SMTPRecipientsRefused, e:
syslog('smtp-failure', 'All recipients refused: %s, msgid: %s',
--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe:
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Server Configuration..
lol I'm not shouting, just too lazy to hit caps :) well I bought this server: RAM 1x 4 GB DDR2-RAM ECC HDD 2x 500 GB SATA II-HDD 7.200 rpm Barebone 1x Fujitsu PRIMERGY RX100 S5 CPU 1x Intel Xeon X3320 Quadcore I did set the SMTP_MAX_RCPT to 5 .. then I created a list of 10,000 members, n it delivered them in like 5-6 minutes .. there was about 500 yahoo addresses and yahoo deferred a few messages.. luckily Hotmail accepted all 5500 addresses.. is that good? do you think getting 2 or 3 or even 4 servers like this one and splitting the 2 million users over them is better? about the throttle patch, ok I confess, I AM IGNORANT! I have no idea how to install the patch :) I have mailman 2.1.9 by the way .. Thanks .. -Original Message- From: Mark Sapiro Sent: Saturday, June 25, 2011 10:40 PM To: Khalil Abbas ; mailman-users@python.org Subject: Re: [Mailman-Users] Server Configuration.. Khalil Abbas wrote: SMTP_MAX_RCPTS =5 WOULD BE OK? OR SHOULD I REDUCE OR INCREASE IT FOR HOTMAIL AND YAHOO TO ACCEPT OUR MESSAGES? WHY ARE YOU SHOUTING? I have no real information as to why Hotmail and Yahoo were not accepting your mail, but apparently at one point I thought that reducing the number of Hotmail and/or Yahoo envelope recipients in a single message transaction might help and apparently, it did. Presumably, nothing at Hotmail and Yahoo has changed, so I wouldn't advise changing SMTP_MAX_RCPTS. What I would advise is before changing other things including hardware, that you try the Postfix changes recommended to you at http://mail.python.org/pipermail/mailman-users/2011-June/071671.html which you apparently never tried or at least never reported trying, and if that doesn't solve the problem completely, that you try the patch in the attached throttle.patch.txt file. The settings in this patch will limit Mailman to send to 2000 recipients per minute which will spread your 2+ million recipients over about 17 hours and will be as effective as making many small lists. Note that contrary to your assertion at http://mail.python.org/pipermail/mailman-users/2011-June/071720.html that the patch is not stable and not tested before, I never said it was not tested. The FAQ says it has been only minimally tested and is not guaranteed to work. What that means is it hasn't seen significant use and it may not solve your problem, but it is almost certainly more stable than rebooting the server every hour. And if you do try it and it does work, we can remove the caveats, and if it doesn't solve th problem, at least we'll know that. Free, open source software is a community effort, not a one-way street. -- Mark Sapiro m...@msapiro.netThe highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Server Configuration..
Khalil Abbas wrote:
lol I'm not shouting, just too lazy to hit caps :)
Regardless of why it happened, it comes across to others as shouting.
well I bought this server:
RAM 1x 4 GB DDR2-RAM ECC
HDD 2x 500 GB SATA II-HDD 7.200 rpm
Barebone 1x Fujitsu PRIMERGY RX100 S5
CPU 1x Intel Xeon X3320 Quadcore
I did set the SMTP_MAX_RCPT to 5 .. then I created a list of 10,000 members,
n it delivered them in like 5-6 minutes ..
there was about 500 yahoo addresses and yahoo deferred a few messages..
luckily Hotmail accepted all 5500 addresses..
is that good?
It's not terrific. At that rate, assuming everything had completely
left your MTA in that time, it would take between 17 and 20 hours to
deliver to 2 million recipients. Is that acceptable?
Also, it is not clear that just because Hotmail accepted all 5500
addresses out of 10,000, that they will accept the next 55,000 out of
the next 100,000 (if there are that many). Also, once Yahoo starts
deferring, their deferral rate will increase if you keep sending more,
and this will add to the load on your MTA.
do you think getting 2 or 3 or even 4 servers like this one and splitting
the 2 million users over them is better?
I can't say. Perhaps the bottleneck is the bandwidth of your internet
connection. If so, increasinfg the ability to push more data faster
won't help at all.
about the throttle patch, ok I confess, I AM IGNORANT! I have no idea how to
install the patch :)
I have mailman 2.1.9 by the way ..
The throttle.patch.txt file attached to this message contains a 2.1.9
version of the patch. The only difference is the line number of the
last portion.
To install it, save the file and cd to your mailman installation and
give the command
patch Mailman/Handlers/SMTPDirect.py /path/to/throttle.patch.txt
and then restart Mailman.
If anything goes wrong, you can remove the patch with the command
patch -R Mailman/Handlers/SMTPDirect.py /path/to/throttle.patch.txt
followed by another restart of Mailman.
--
Mark Sapiro m...@msapiro.netThe highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan
--- SMTPDirect.py.orig 2011-06-25 17:39:07.859375000 -0700
+++ SMTPDirect.py 2011-06-25 17:43:04.078125000 -0700
@@ -52,6 +52,37 @@
True = 1
False = 0
+# Throttling settings. Do not send to more than THROTTLE_LIMIT recipients
+# within THROTTLE_TIME seconds. Set THROTTLE_LIMIT to 0 to not throttle.
+THROTTLE_TIME = 60
+THROTTLE_LIMIT = 2000
+THROTTLE_DATA = []
+
+# check throttling. Doesn't return until total recipients in THROTTLE_TIME
+# = THROTTLE_LIMIT.
+#
+def check_throttle(count):
+global THROTTLE_DATA
+if THROTTLE_LIMIT = 0:
+return
+if count = 0 or count THROTTLE_LIMIT:
+syslog('error', 'check_throttle: count=%d; THROTTLE_LIMIT=%d',
+ count, THROTTLE_LIMIT)
+return
+now = time.time()
+total = 0
+THROTTLE_DATA = [(num, tim) for (num, tim) in THROTTLE_DATA
+ if tim now - THROTTLE_TIME]
+for num, tim in THROTTLE_DATA:
+total += num
+total += count
+if total THROTTLE_LIMIT:
+time.sleep(THROTTLE_TIME - now + THROTTLE_DATA[0][1])
+check_throttle(count)
+else:
+THROTTLE_DATA.append((count, now))
+return
+
# Manage a connection to the SMTP server
@@ -118,10 +149,14 @@
chunks = [[recip] for recip in recips]
msgdata['personalize'] = 1
deliveryfunc = verpdeliver
-elif mm_cfg.SMTP_MAX_RCPTS = 0:
+elif mm_cfg.SMTP_MAX_RCPTS = 0 and (THROTTLE_LIMIT =0 or
+ THROTTLE_LIMIT = len(recips)):
chunks = [recips]
else:
-chunks = chunkify(recips, mm_cfg.SMTP_MAX_RCPTS)
+if THROTTLE_LIMIT = 0 or THROTTLE_LIMIT mm_cfg.SMTP_MAX_RCPTS:
+chunks = chunkify(recips, mm_cfg.SMTP_MAX_RCPTS)
+else:
+chunks = chunkify(recips, THROTTLE_LIMIT)
# See if this is an unshunted message for which some were undelivered
if msgdata.has_key('undelivered'):
chunks = msgdata['undelivered']
@@ -366,6 +401,8 @@
msgid = msg['message-id']
try:
# Send the message
+# but wait if necessary for throttling
+check_throttle(len(recips))
refused = conn.sendmail(envsender, recips, msgtext)
except smtplib.SMTPRecipientsRefused, e:
syslog('smtp-failure', 'All recipients refused: %s, msgid: %s',
--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe:
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
