Re: [Mailman-Users] Umbrella List Spam Protection
Links at Momsview wrote: I currently have a VERY large ( 100k) announce only list (always moderated) running mailman 2.1 under Postfix. For performance reasons I'm considering splitting this list into 26 sublists based on the first letter of each subscriber email. I would then use an umbrella list to send to all of the sublists at once. As I understand it, unless I want to separately approve the message for the 26 sublists I need to allow at least one email address to post unmoderated on ALL of the sublists. This is very scary since spammers currently try to post on my current large list almost daily. Before you go down this road, be sure you have done everything possible in terms of selecting and configuring your outgoing mail server to handle the volume. I suspect that handling the outgoing volume will be easier and more effective when you direct your attention towards your mail server, than setting up sublists on the list server. FAQ entry 1.15 has more on this topic, and directs you to related topics: http://www.python.org/cgi-bin/faqw-mm.py?req=all#1.15 HTH jc -- Mailman-Users mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ This message was sent to: archive@jab.org Unsubscribe or change your options at http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Umbrella List Spam Protection
At 12:11 12/02/2003, Links at Momsview wrote: I currently have a VERY large ( 100k) announce only list (always moderated) running mailman 2.1 under Postfix. For performance reasons I'm considering splitting this list into 26 sublists based on the first letter of each subscriber email. I would then use an umbrella list to send to all of the sublists at once. As I understand it, unless I want to separately approve the message for the 26 sublists I need to allow at least one email address to post unmoderated on ALL of the sublists. This is very scary since spammers currently try to post on my current large list almost daily. (1) How do I protect the sublists from spammers who spoof the from address in their emails? (2) Can I protect these sublists from ALL emails other than those from localhost and my umbrella list? (3) Are there any other methods that people use for Umbrella announce only lists? One other method I've been thinking about is using a grep expression to define a sender address that is only valid for one day each month. Any idea how I could do this? I would have thought the following arrangement would suit your needs: 1. Set up the umbrella list the same as you have your current all-in-one list in terms of privacy, moderation etc 2. The trick I think will work is to only allow postings to the sublists that have first gone to the umbrella list so that in most all respects the sublists do not have to protect themselves. 3. On the Privacy-Recipients page of the admin web GUI for the sublists set the acceptable_aliases field to contain a regex that only matches your umbrella list. 4. On the Privacy-Spam filters page of the admin web GUI for the sublists set the bounce_matching_headers fields to contain the following: to: listname@your.mailman.sever.com cc: listname@your.mailman.sever.com In essence sublists are going to allow through and distribute messages that have been sent to the umbrella list but hold messages sent direct to themselves. Because the rules are being applied to posts by the umbrella lists we do not have to worry about having tight rules for the sublists as what reaches them has been proved acceptable mail via the umbrella list. You probably want to tidy up a bunch of loose ends so that only the umbrella list adds footers and prefixs to the subject line and such, but I think this will work as you want. I suspect you may want to: 1. add to the list of non-members who are allowed to post to each sublist the addresses of those you are allowing to post to the umbrella list. 2. set the following on the Privacy-Senders page of the admin web GUI for the sublists: a. set the generic_nonmember_action to Discard or Reject b. forward_auto_discards to No This will mean that most attempts to post to the sublists will be discarded/rejected automatically. If that is not acceptable because the list of allowed posters is too long or whatever, then you will get notifications of held mail when posts are made directly to the sublists. Haven't figured out how to stop that (yet). Be interested to hear what your final solution is. -- Mailman-Users mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ This message was sent to: archive@jab.org Unsubscribe or change your options at http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Umbrella List Spam Protection
LaM == Links at Momsview [EMAIL PROTECTED] writes: LaM I currently have a VERY large ( 100k) announce only list (always LaM moderated) running mailman 2.1 under Postfix. It seems to me if you turn off personalization, turn off VERP, and turn off anything else that would cause individual messages to be sent, postfix would handle this in the most efficient way given a large number of recipients per message. Making sub-lists would just cause a lot more work, I suspect. -- Mailman-Users mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ This message was sent to: archive@jab.org Unsubscribe or change your options at http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Umbrella List Spam Protection
JC, Thanks for the suggestion and the link. This is what I'm seeing with my single list of 140,000 subs (1) MTA performance is not a bottleneck OutgoingRunner is sending to Postfix at between 15-30 emails /sec. Since the mail queue never grows beyond 1000 (from failed deliveries) I'm assuming that Mailman (or the I/O to /var/log/maillog) is the bottleneck limiting performance. In any case I'm more than satisfied with 900-1800 emails/minute. (2) Excessive Memory usage BounceRunner, OutgoingRunner, CommandRunner, ArchRunner, IncomingRunner and VirginRunner are EACH using 120-135 Megabytes of Memory. That's over 700Mbytes RAM just to load these processes (and periodically trigger swap on my 1.5GB 2GHZ P4). I've been able to track the memory usage to the use of a Python dictionary to hold the mailing list (the extra memory required for the hash space probably results in 500% memory overhead compared to a Python list structure). (3) Excessive Processing Time BounceRunner was processing over 80 bounces per minute peak with a list size of 48,000 subs. That went to 40 bounces per minute peak at 89,000 subs and 25 per minute at 138,000. I believe this is due to the CPU time and disk I/O required to write out 17 megabytes of data for every bounce (at 142,000 subs). CommandRunner is showing a similar drop in performance probably due to the same reasons If there is another approach I should take PLEASE PLEASE let me know. But from what I'm seeing, short of re-writing a lot of code, splitting the list is my only hope of reducing both the memory usage and the processing time. John - Original Message - From: JC Dill [EMAIL PROTECTED] To: Links at Momsview [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Wednesday, February 12, 2003 3:36 PM Subject: Re: [Mailman-Users] Umbrella List Spam Protection Links at Momsview wrote: I currently have a VERY large ( 100k) announce only list (always moderated) running mailman 2.1 under Postfix. For performance reasons I'm considering splitting this list into 26 sublists based on the first letter of each subscriber email. I would then use an umbrella list to send to all of the sublists at once. As I understand it, unless I want to separately approve the message for the 26 sublists I need to allow at least one email address to post unmoderated on ALL of the sublists. This is very scary since spammers currently try to post on my current large list almost daily. Before you go down this road, be sure you have done everything possible in terms of selecting and configuring your outgoing mail server to handle the volume. I suspect that handling the outgoing volume will be easier and more effective when you direct your attention towards your mail server, than setting up sublists on the list server. FAQ entry 1.15 has more on this topic, and directs you to related topics: http://www.python.org/cgi-bin/faqw-mm.py?req=all#1.15 HTH jc -- Mailman-Users mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ This message was sent to: archive@jab.org Unsubscribe or change your options at http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Umbrella List Spam Protection
On Wed, 12 Feb 2003 22:23:51 -0500 Links at Momsview [EMAIL PROTECTED] wrote: Thanks for the suggestion and the link. This is what I'm seeing with my single list of 140,000 subs Urk, I've not played with 2.1 much at all yet (too buried in Zope). I'd thought that 2.1 lost the in-memory dictionary for the entire subscriber base. Damn. It does sound like you are RAM starved. -- J C Lawrence -(*)Satan, oscillate my metallic sonatas. [EMAIL PROTECTED] He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live. -- Mailman-Users mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ This message was sent to: archive@jab.org Unsubscribe or change your options at http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Umbrella List Spam Protection
Richard, Thank you very much for the detailed reply. Please let me know if I understood what you are suggesting (1) By having the sublists ONLY accept messages with a To: [EMAIL PROTECTED] NO ONE can ever directly post to the sublists by directly sending an email to my server since ANY email of this form would end up going to the umbrella list first. (2) By setting up Spam filters on each sublist to detect: to: sublist-name@myserver.com OR cc: sublist-name@myserver.com ANY emails sent directly to the sublist will be HELD. (3) Then set up a list of acceptable non-members to post to the list (on both the umbrella and the sublists) AND require moderation on the umbrella for this address (but no moderation on the sublists). (4) An email sent from an acceptable non-member address to the Umbrella list will be held for moderation. Once approved it will be forwarded to the sublists and mailed out. (5)If someone attempts to spoof this acceptable non-member address and send to the sublists the SPAM filters will match on the sublist-name in the to: or cc: fields and HOLD the message. Questions: It appears that (5) is critical since the non-member address on the sublist is set to allow posting without moderation. Could you post the exact form of this spam filter regex string. Do I need several different formats to detect possible variations? How do I set the General Options (on the Umbrella and sublists) to hide the acceptable non-member address from the mail headers and allow the post to pass properly between the umbrella and sublists? Any idea how I can easily create my 26 sublists without a lot of GUI clicks? Thank you so much in advance for all your help, John 2. The trick I think will work is to only allow postings to the sublists that have first gone to the umbrella list so that in most all respects the sublists do not have to protect themselves. 3. On the Privacy-Recipients page of the admin web GUI for the sublists set the acceptable_aliases field to contain a regex that only matches your umbrella list. 4. On the Privacy-Spam filters page of the admin web GUI for the sublists set the bounce_matching_headers fields to contain the following: to: listname@your.mailman.sever.com cc: listname@your.mailman.sever.com In essence sublists are going to allow through and distribute messages that have been sent to the umbrella list but hold messages sent direct to themselves. Because the rules are being applied to posts by the umbrella lists we do not have to worry about having tight rules for the sublists as what reaches them has been proved acceptable mail via the umbrella list. -- Mailman-Users mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ This message was sent to: archive@jab.org Unsubscribe or change your options at http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
