Re: [Mailman-Users] Big Lists
On Tue, 18 Dec 2001 12:39:22 -0800 Dan Wilder <[EMAIL PROTECTED]> wrote: > Maybe I'm being stoopid or something. Nahh, its just a confusion between the default phraseology under Sendmail being different from the phraseology under Postfix. > Supposing I have a list server, which I wish to deliver directly > to the subscribers, rather than making use of a presumably bigger, > better connected, or faster smarthost. Then leave disable_dns_lookups set to no. You may also want to ensure that the various SPAM and header checks are not enabled. -- J C Lawrence -(*)Satan, oscillate my metallic sonatas. [EMAIL PROTECTED] He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live. -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] Big Lists
On Tue, Dec 18, 2001 at 12:02:07PM -0800, J C Lawrence wrote: > On Tue, 18 Dec 2001 10:39:39 -0800 > Chuq Von Rospach <[EMAIL PROTECTED]> wrote: > > On 12/18/01 5:06 AM, "Tass" <[EMAIL PROTECTED]> wrote: > > >>> disable_dns_lookups = yes > > >> I have found this to be no real change. On a PIII-500 with 512M > >> I am getting 100K-120K/hr on a T1 connection with avg message of > >> 6.1K. > > > Then I'd guess you have a fairly small group of domains you deal > > with. The wider your subscribers wander, especially if you cross > > continents, the more you'll see issues with slow DNS. Better to > > turn it off and let the Mta deal with it asynchronously. > > > Uhh, Chuq, from the man page: > >disable_dns_lookups > Disable DNS lookups. This means that mail must be > forwarded via a smart relay host. Maybe I'm being stoopid or something. Supposing I have a list server, which I wish to deliver directly to the subscribers, rather than making use of a presumably bigger, better connected, or faster smarthost. Then I want to do precisely what I've done: that is, leave disable_dns_lookups = no the default? Or have I got something backward? -- - Dan Wilder <[EMAIL PROTECTED]> Technical Manager & Editor SSC, Inc. P.O. Box 55549 Phone: 206-782-8808 Seattle, WA 98155-0549URL http://embedded.linuxjournal.com/ - -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] Big Lists
On Tue, 18 Dec 2001 10:39:39 -0800 Chuq Von Rospach <[EMAIL PROTECTED]> wrote: > On 12/18/01 5:06 AM, "Tass" <[EMAIL PROTECTED]> wrote: >>> disable_dns_lookups = yes >> I have found this to be no real change. On a PIII-500 with 512M >> I am getting 100K-120K/hr on a T1 connection with avg message of >> 6.1K. > Then I'd guess you have a fairly small group of domains you deal > with. The wider your subscribers wander, especially if you cross > continents, the more you'll see issues with slow DNS. Better to > turn it off and let the Mta deal with it asynchronously. Uhh, Chuq, from the man page: disable_dns_lookups Disable DNS lookups. This means that mail must be forwarded via a smart relay host. AFAICT Postfix will only do DNS verifies if you set one the (E)HELO/MAIL FROM: etc checks as detailed here: http://www.postfix.org/uce.html#smtpd_helo_required And unless you set that, Postfix does not DNS verifies until attempted delivery time. -- J C Lawrence -(*)Satan, oscillate my metallic sonatas. [EMAIL PROTECTED] He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live. -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] Big Lists [Postfix]
I'd like to emphasize that YMMV. As Chuq notes, don't take my numbers or anybody else's and apply them blindly. What fits one site will fit badly on others. Change one thing at a time, in some orderly way that lets you evaluate impact of changes. Important things I can think of immediately that will vary by site include: -- Bandwidth to internet -- Size of email postings -- Number of postings -- Number of recipients -- Number of recipient domains -- Ease and reliability of reaching recipient domains and their DNS -- Amount of RAM, processer speed, disk speed in mailserver -- Other load the mailserver may be under -- How fast the list mail needs to be delivered -- Proximity of nearest DNS cache All of these may impact the optimum tuning. On Tue, Dec 18, 2001 at 10:39:39AM -0800, Chuq Von Rospach wrote: > On 12/18/01 5:06 AM, "Tass" <[EMAIL PROTECTED]> wrote: > > >> default_process_limit = 150 > > > > If you have 512M of Ram set it to 200, it will give you a lot of room. > > Maybe. Maybe not. > > One of the things you need to do when setting up your MTA is figure out what > your network can take. It makes no sense (in fact, it hurts throughput > significantly) if you have more processes attempting to send mail than your > network can handle. > > If you're on a 384K DSL, you have to tune much different than if you're on > an T1. And you're gated by your slowest link, not your closest one. > > One of the things you need to do, then, is try increasingly large numbers of > parallel delivery processes, and watch your network patterns. What I usually > do is look for when dropped packets and/or retries start going up. Once you > get there, you've basically saturated your slowest network link, bceause > stuff is disappearing into the void. Once that happens, your overall > throughput will start going down, because your processes will be fighting > for the network, losing packets, retrying, waiting for each other's packets > to get out of the way, and generally creating minor chaos at the TCP layer. > > So what I do is I ramp up delivery until this point happens, then back it > off by about 10%. If (as with the SSH guys a couple of days ago) you want to > reserve more bandwidth, back it off more. > > A really rough way to estimate this, if you don't want to get nerdy, is to > use "ping". Ping, for instance, the router on the far side of your WAN link, > and wait for packets to start dropping. That'll tell you when that link > fills up. > > If you're seeing significant collisions, retries or dropped packets, you're > overfilling your network and hurting performance. Better 50 mail processes > cooperating than 60 getting in each other's way. And I think it's best to > manage this by rate-limiting the MTA, which you can test and tune fairly > easily. > > >> disable_dns_lookups = yes > > > > I have found this to be no real change. > > On a PIII-500 with 512M I am getting 100K-120K/hr on a T1 connection with > > avg message of 6.1K. > > Then I'd guess you have a fairly small group of domains you deal with. The > wider your subscribers wander, especially if you cross continents, the more > you'll see issues with slow DNS. Better to turn it off and let the Mta deal > with it asynchronously. > > > If most of your email is going to a few domains that you know are good at > > handling email you can also set > > > > default_destination_concurrency_limit = 15 > > As long as you don't flood them and piss them off. Be careful over-ramping > to a single domain. > > Yes. I've got default_destination_concurrency_limit = 5 and also smtp_destination_recipient_limit = 10 in postfix/main.cf because of piss-off problems with a few recipient domains. These don't seem to affect our overall delivery rate much, because no one domain accounts for more than a few percent of total subscribers. -- - Dan Wilder <[EMAIL PROTECTED]> Technical Manager & Editor SSC, Inc. P.O. Box 55549 Phone: 206-782-8808 Seattle, WA 98155-0549URL http://embedded.linuxjournal.com/ - -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] Big Lists
On 12/18/01 5:06 AM, "Tass" <[EMAIL PROTECTED]> wrote: >> default_process_limit = 150 > > If you have 512M of Ram set it to 200, it will give you a lot of room. Maybe. Maybe not. One of the things you need to do when setting up your MTA is figure out what your network can take. It makes no sense (in fact, it hurts throughput significantly) if you have more processes attempting to send mail than your network can handle. If you're on a 384K DSL, you have to tune much different than if you're on an T1. And you're gated by your slowest link, not your closest one. One of the things you need to do, then, is try increasingly large numbers of parallel delivery processes, and watch your network patterns. What I usually do is look for when dropped packets and/or retries start going up. Once you get there, you've basically saturated your slowest network link, bceause stuff is disappearing into the void. Once that happens, your overall throughput will start going down, because your processes will be fighting for the network, losing packets, retrying, waiting for each other's packets to get out of the way, and generally creating minor chaos at the TCP layer. So what I do is I ramp up delivery until this point happens, then back it off by about 10%. If (as with the SSH guys a couple of days ago) you want to reserve more bandwidth, back it off more. A really rough way to estimate this, if you don't want to get nerdy, is to use "ping". Ping, for instance, the router on the far side of your WAN link, and wait for packets to start dropping. That'll tell you when that link fills up. If you're seeing significant collisions, retries or dropped packets, you're overfilling your network and hurting performance. Better 50 mail processes cooperating than 60 getting in each other's way. And I think it's best to manage this by rate-limiting the MTA, which you can test and tune fairly easily. >> disable_dns_lookups = yes > > I have found this to be no real change. > On a PIII-500 with 512M I am getting 100K-120K/hr on a T1 connection with > avg message of 6.1K. Then I'd guess you have a fairly small group of domains you deal with. The wider your subscribers wander, especially if you cross continents, the more you'll see issues with slow DNS. Better to turn it off and let the Mta deal with it asynchronously. > If most of your email is going to a few domains that you know are good at > handling email you can also set > > default_destination_concurrency_limit = 15 As long as you don't flood them and piss them off. Be careful over-ramping to a single domain. -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] Big Lists
At 06:01 PM 12/17/01 -0800, Dan Wilder wrote: ... >With these changes in place, a K6-350 with 128M RAM delivers >messages averaging perhaps 3K over a medium-speed DSL, >without entirely saturating the DSL, at load average below >2.0, without impinging on swap. It'll reach 20,000 >recipients in a couple of hours. > >Another common recommendation with Postfix is to set > >disable_dns_lookups = yes > >a measure others have reported favorably on, but which >I have not yet tried. I fear I'm missing something obvious... I'm not sure I follow the "disable_dns_lookups = yes" recommendation I've seen posted here a few times. Is that assuming that all mail is delivered to some "smart host" that then does the MX lookups? Obviously, the MX must be looked up someplace. (But then why not set SMTPHOST to point to the smart host and deliver directly from Mailman to the smarthost?) Or is the idea to get the mail out of mailman as soon as possible? Hum, coffee is tickling memory cells... Seems like I remember with sendmail a way to say "don't do DNS lookups during SMTP session" (but obviously lookup MX for delivery). But that's seems different than Postfix's disable_dns_lookups: disable_dns_lookups Disable DNS lookups. This means that mail must be forwarded via a smart relay host. Thanks, Bill Moseley mailto:[EMAIL PROTECTED] -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] Big Lists
On Mon, 17 Dec 2001, Dan Wilder wrote: > On Mon, Dec 17, 2001 at 05:22:37PM -0800, Bill Moseley wrote: > To get the number of SMTP processes up, change "default_process_limit" > in main.cf. > > default_process_limit = 150 If you have 512M of Ram set it to 200, it will give you a lot of room. > There's an active message queue which also became a sticky wicket > for us. Two limits which affect this, and the values I arrived > at by experiment are: > > qmgr_message_active_limit = 4 > qmgr_message_recipient_limit = 4 > > Default on both of these is 1000, as of the Postfix version I initially > installed. The comments in the config file say: Unless newer versions have been fixed, there is a max of 10K or 50K, depending on source version. I had to edit teh source and recompile to get it up to the 75K+ range. > disable_dns_lookups = yes I have found this to be no real change. On a PIII-500 with 512M I am getting 100K-120K/hr on a T1 connection with avg message of 6.1K. A good thing to do is to have hash_queue_names = incoming, defer, deferred active bounce flush hash_queue_depth = 2 the hashed structure gave us an increase of 20% in soeed. If most of your email is going to a few domains that you know are good at handling email you can also set default_destination_concurrency_limit = 15 -- |Tass Chapman| [EMAIL PROTECTED] : PGP key @ pgpkeys.mit.edu:11371 | | http://www.kenderhome.com : KEYID: 0x5BC152A1 | |ICQ UIN:394570| -- -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] Big Lists
Dan, Have stolen your messages almost verbatim and put them into the postfix tuning part of the FAQ. [I'm currently aiming to get content on there - someone or even me can clean these up later] http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq06.004.htp -- [ Nigel Metheringham [EMAIL PROTECTED] ] [ Phone: +44 1423 85 Fax +44 1423 858866 ] [ - Comments in this message are my own and not ITO opinion/policy - ] -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] Big Lists
On Mon, 17 Dec 2001 20:33:23 -0800 Dan Wilder <[EMAIL PROTECTED]> wrote: > On Mon, Dec 17, 2001 at 06:39:54PM -0800, J C Lawrence wrote: >> On Mon, 17 Dec 2001 18:01:56 -0800 Dan Wilder <[EMAIL PROTECTED]> >> wrote: >> Without those changes in place (more exctly, using the >> Debian/Linux defaults) a Dual PII-333 on the end of a Tier 2 T3 >> has regularly sustained just under 1,400 deliveries (to target >> MX) per minute for me. While I've not maintained that level for >> more than single digit minutes (not enough traffic) that sums to >> 80K messages per hour. I've been happy enough with those numbers >> for the box in question that I haven't looked further. > I began to see signs of thrashing some place between a couple of > thousand recipients, and 10,000. There was what appeared to be a > phase transition of sorts, above which waits began to predominate > and I believe delivery rate may have even gone down. The largest load I've thrown on that box was 10K spool entries with 50K RCPT TOs. A more typical load is 3K spool entries and 15K RCPT TOs from a single qrunner dump. (Yeah, I do 5 per on RCPT TOs and I hand moderate in messages in batches) At those levels I've not noticed any thrashing. > I'm not getting anything like 1400 deliveries/minute (I don't > think). A typical average for me (given that the box also does web duty and a bunch of other bits) is ~700 per minute (40K per hour), but that's when the spool is well stuffed with dead mail, PostgresQL is getting hit via Apache from other quarters, etc. 1,400/minute is more typical when running off a fairly clean slate with a pre-stuffed DNS cache. > I'll try measuring delivery rate during this week's large > announce-lists posting, then next week set disable_dns_lookups and > see what change there may be. Cool. BTW: I like David Schweikert's MailGraph tool for Postfix stat collection/graphing . Quite sweet if rather CPU intensive. http://people.ee.ethz.ch/~dws/software/mailgraph/ > The DSL does get to be pretty much of a nuisance to run vi on over > an ssh connection, when the list mail is going out. Hehn. I tend to SSH in to the firewall, then SSH to the target box with X11 forwarding turned on all the way so I can run XEmacs on the target and have it display locally... -- J C Lawrence -(*)Satan, oscillate my metallic sonatas. [EMAIL PROTECTED] He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live. -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] Big Lists
On Mon, 17 Dec 2001 21:05:19 -0800 (PST) alex wetmore <[EMAIL PROTECTED]> wrote: > My Mailman configuration uses Unix for running Mailman, and > Windows 2000 SMTP Server for outbound email. I doubt many (if > any) other people are using this configuration, but if people are > interested in more detail on configuring Windows 2000 SMTP for > maximum performance I can help out quite a bit. Why not add that data to the FAQ? -- J C Lawrence -(*)Satan, oscillate my metallic sonatas. [EMAIL PROTECTED] He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live. -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] Big Lists
On Mon, 17 Dec 2001, Dan Wilder wrote: > The DSL does get to be pretty much of a nuisance to run vi on > over an ssh connection, when the list mail is going out. I used to have issues with this, but my solution was to turn on dummynet (http://cs.baylor.edu/~donahoo/tools/dummy/) on my firewall and limit SMTP to use half of my outbound bandwidth (I get 256kbps, SMTP gets 128kbps of that). Interactive sessions are a lot more pleasant to work with now. I know a lot more about Windows SMTP server than Postfix, but outbound delivery rate on machines is generally limited by the number of io's per second that a disk can do. If you want to saturate a fast link with small messages you'll probably need to stripe your queue/spool directory across multiple disks. Performance on the Windows SMTP stack will also greatly improve if you increase the rcpt to batching that Mailman uses. On many Unix MTAs that will slow you down, because the MTA can't send the same message to multiple destinations at the same time. My Mailman configuration uses Unix for running Mailman, and Windows 2000 SMTP Server for outbound email. I doubt many (if any) other people are using this configuration, but if people are interested in more detail on configuring Windows 2000 SMTP for maximum performance I can help out quite a bit. alex -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] Big Lists
On Mon, Dec 17, 2001 at 06:39:54PM -0800, J C Lawrence wrote: > On Mon, 17 Dec 2001 18:01:56 -0800 > Dan Wilder <[EMAIL PROTECTED]> wrote: > > > With these changes in place, a K6-350 with 128M RAM delivers > > messages averaging perhaps 3K over a medium-speed DSL, without > > entirely saturating the DSL, at load average below 2.0, without > > impinging on swap. It'll reach 20,000 recipients in a couple of > > hours. > > Without those changes in place (more exctly, using the Debian/Linux > defaults) a Dual PII-333 on the end of a Tier 2 T3 has regularly > sustained just under 1,400 deliveries (to target MX) per minute for > me. While I've not maintained that level for more than single digit > minutes (not enough traffic) that sums to 80K messages per hour. > I've been happy enough with those numbers for the box in question > that I haven't looked further. I began to see signs of thrashing some place between a couple of thousand recipients, and 10,000. There was what appeared to be a phase transition of sorts, above which waits began to predominate and I believe delivery rate may have even gone down. I'm not getting anything like 1400 deliveries/minute (I don't think). I'll try measuring delivery rate during this week's large announce-lists posting, then next week set disable_dns_lookups and see what change there may be. The DSL does get to be pretty much of a nuisance to run vi on over an ssh connection, when the list mail is going out. The local copy of BIND is something I'd neglected to mention. We use split DNS, with internal copies of BIND claiming to be definitive for our domain, forwarding queries with respect to other domains outside. The mailing list server does auxilliary duty as second internal MX server, so there's a copy of BIND running on it, as well as Postfix/Mailman. It looks to itself first for all name resolution. > And no, it doesn't even begin to saturate available b/w. > > > disable_dns_lookups = yes > > I do have this set. I also have a local copy of BIND 9 listening on > localhost only. > > > mailq > > FWVLIW I typically have between 1,500 and 2K messages in my queue on > that machine (my hobby box), all for slow/down MXes. > Things look a lot like that for a while after an announce list posting here, too. I only try to measure delivery rate while the list server is still running 150 smtp processes. -- - Dan Wilder <[EMAIL PROTECTED]> Technical Manager & Editor SSC, Inc. P.O. Box 55549 Phone: 206-782-8808 Seattle, WA 98155-0549URL http://embedded.linuxjournal.com/ - -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] Big Lists
On Mon, 17 Dec 2001 18:01:56 -0800 Dan Wilder <[EMAIL PROTECTED]> wrote: > With these changes in place, a K6-350 with 128M RAM delivers > messages averaging perhaps 3K over a medium-speed DSL, without > entirely saturating the DSL, at load average below 2.0, without > impinging on swap. It'll reach 20,000 recipients in a couple of > hours. Without those changes in place (more exctly, using the Debian/Linux defaults) a Dual PII-333 on the end of a Tier 2 T3 has regularly sustained just under 1,400 deliveries (to target MX) per minute for me. While I've not maintained that level for more than single digit minutes (not enough traffic) that sums to 80K messages per hour. I've been happy enough with those numbers for the box in question that I haven't looked further. And no, it doesn't even begin to saturate available b/w. > disable_dns_lookups = yes I do have this set. I also have a local copy of BIND 9 listening on localhost only. > mailq FWVLIW I typically have between 1,500 and 2K messages in my queue on that machine (my hobby box), all for slow/down MXes. -- J C Lawrence -(*)Satan, oscillate my metallic sonatas. [EMAIL PROTECTED] He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live. -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] Big Lists
On Mon, Dec 17, 2001 at 05:22:37PM -0800, Bill Moseley wrote: > At 04:26 PM 12/17/01 -0800, J C Lawrence wrote: > >On Mon, 17 Dec 2001 15:32:39 -0800 > >Bill Moseley <[EMAIL PROTECTED]> wrote: > > > >> Someone posted a few days or so ago asking about list sizes. Was > >> there any response to that query? > > > >Yup, and its already in the FAQ. > > http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq01.015.htp > > Now, anyone available to fill in the blanks for qmail or Postfix tuning? I'll try to start that ball rolling. As lists get larger, Postfix delivery with out-of-the box configuration really slows down. The bottlenecks I've found are queue length and number of SMTP processes, both of which default to values too small for large lists. I began noticing pretty severe rate limiting at about 10,000 deliveries on a list. To get the number of SMTP processes up, change "default_process_limit" in main.cf. default_process_limit = 150 gave results I could live with. There's an active message queue which also became a sticky wicket for us. Two limits which affect this, and the values I arrived at by experiment are: qmgr_message_active_limit = 4 qmgr_message_recipient_limit = 4 Default on both of these is 1000, as of the Postfix version I initially installed. The comments in the config file say: # The qmgr_message_active_limit parameter limits the number of # messages in the active queue. # The qmgr_message_recipient_limit parameter limits the number of # in-memory recipients. This parameter also limits the size of the # short-term, in-memory destination status cache. With these changes in place, a K6-350 with 128M RAM delivers messages averaging perhaps 3K over a medium-speed DSL, without entirely saturating the DSL, at load average below 2.0, without impinging on swap. It'll reach 20,000 recipients in a couple of hours. Another common recommendation with Postfix is to set disable_dns_lookups = yes a measure others have reported favorably on, but which I have not yet tried. After changing parameters in main.cf, run "postfix reload" then look at the process table to see if the postfix processes are running, and how many smtp processes are working at it. mailq gives a report of what's in process and is a great help in tuning. A crude index of what's out there is mailq | grep '^[A-Z0-9]' | wc -l and variations. -- - Dan Wilder <[EMAIL PROTECTED]> Technical Manager & Editor SSC, Inc. P.O. Box 55549 Phone: 206-782-8808 Seattle, WA 98155-0549URL http://embedded.linuxjournal.com/ - -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] Big Lists
At 04:26 PM 12/17/01 -0800, J C Lawrence wrote: >On Mon, 17 Dec 2001 15:32:39 -0800 >Bill Moseley <[EMAIL PROTECTED]> wrote: > >> Someone posted a few days or so ago asking about list sizes. Was >> there any response to that query? > >Yup, and its already in the FAQ. http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq01.015.htp Now, anyone available to fill in the blanks for qmail or Postfix tuning? > >-- >J C Lawrence >-(*)Satan, oscillate my metallic sonatas. >[EMAIL PROTECTED] He lived as a devil, eh? >http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live. > >-- >Mailman-Users maillist - [EMAIL PROTECTED] >http://mail.python.org/mailman/listinfo/mailman-users > > Bill Moseley mailto:[EMAIL PROTECTED] -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] Big Lists
On Mon, 17 Dec 2001 15:32:39 -0800 Bill Moseley <[EMAIL PROTECTED]> wrote: > Someone posted a few days or so ago asking about list sizes. Was > there any response to that query? Yup, and its already in the FAQ. -- J C Lawrence -(*)Satan, oscillate my metallic sonatas. [EMAIL PROTECTED] He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live. -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] Big Lists
On 12/17/01 3:32 PM, "Bill Moseley" <[EMAIL PROTECTED]> wrote: > I've got a list of about 11,000 currently on Solaris/Sendmail/listproc that > I'm thinking of moving to Linux/(qmail|Postfix)/mailman. Only one message > a week is sent. > > Anyone running a list that big on Mailman? Any special setup required? Yes. No. Make sure you have your MTA tuned well, especially with DNS lookups on accept turned off. And realize that it's going to take some time, and with mailman single-threaded in 2.0.x, while it's delivering, all the other lists will wait. But it works fine on my E250/sendamil box. -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] Big Lists
On Mon, 17 Dec 2001, Bill Moseley wrote: > Someone posted a few days or so ago asking about list sizes. Was there any > response to that query? > > I've got a list of about 11,000 currently on Solaris/Sendmail/listproc that > I'm thinking of moving to Linux/(qmail|Postfix)/mailman. Only one message > a week is sent. > > Anyone running a list that big on Mailman? Any special setup required? Sure, one of my lists is that big. Nothing fancy required. If you want it to go fast, make sure your mail server if set up to handle whatever sort of volume you are looking for. I think some folks here have -much- bigger lists ... -- Charlie Watts [EMAIL PROTECTED] Frontier Internet, Inc. http://www.frontier.net/ -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
[Mailman-Users] Big Lists
Someone posted a few days or so ago asking about list sizes. Was there any response to that query? I've got a list of about 11,000 currently on Solaris/Sendmail/listproc that I'm thinking of moving to Linux/(qmail|Postfix)/mailman. Only one message a week is sent. Anyone running a list that big on Mailman? Any special setup required? Bill Moseley mailto:[EMAIL PROTECTED] -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
Well, you guys are gonna kill me for this, but I seem to have solved my original problem. You may recall that I have a 10,000 member list which gets a 30K mailing each week, and many people were reporting missing issues. Um... it had nothing to do with the size of anything at all. Somebody managed to drop an address into the list that began with a colon (:). It seems that a colon is a control or routing character in majordomo, and that's what was screwing things up. Once I removed that offending address and sent out a test message, we got many replies saying the list is now working. This has turned out to be entirely off-topic, but I suppose there's a chance that Mailman would also choke on an address with a colon in it. Again, thanks for ALL the suggestions! -Eric (running for his foxhole!) -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
RE: [Mailman-Users] big lists, big messages
On May 14, 2001 at 09:54, Rabinowitz, Ari (Exchange) wrote: >Chuq, PMJI. >Thanks for all of your good ideas in this thread. I'm new to Postfix, and >Mailman. How would I defer DNS in Postfix? I have a few lists (read only) If I understand you, put one or both of these in main.cf: defer_transports=smtp #disables smtp transport until you issue sendmail -q disable_dns_lookups = yes #I'm not sure, but apparently postfix doesn't check with DNS during #SMTP dialogs. >with about 1200 addresses that send out only a few messages a day, at most. >I have MAX_RCPT_TO set to 500 so I would expect mailman to split the lists >into 3 messages, although it ends up splitting into 4 messages, two with >about 500 recipients each, and then two smaller lists instead of just one. > >I am using Postfix as the MTA and it seems to work well, but I notice from >the logs that it seems that Postfix only gets one message at a time from >Mailman. Is this because of my Mailman settings or my Postfix settings? >What controls this? I see something that may be similar. Mailman is batching messages as I want -- 5 or 10 at a time. Postfix sends to the upstream MTA one at a time. Is this what you're seeing? -- Satya. http://satya.virtualave.net/> US-bound grad students! For pre-apps, see http://quickapps.cjb.net/> Cap'n! Cap'n! the UART's will'nae take the speed! -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
RE: [Mailman-Users] big lists, big messages
Chuq, Thanks for all of your good ideas in this thread. I'm new to Postfix, and Mailman. How would I defer DNS in Postfix? I have a few lists (read only) with about 1200 addresses that send out only a few messages a day, at most. I have MAX_RCPT_TO set to 500 so I would expect mailman to split the lists into 3 messages, although it ends up splitting into 4 messages, two with about 500 recipients each, and then two smaller lists instead of just one. I am using Postfix as the MTA and it seems to work well, but I notice from the logs that it seems that Postfix only gets one message at a time from Mailman. Is this because of my Mailman settings or my Postfix settings? What controls this? Any suggestions as to speeding up delivery of the messages would be appreciated. Thanks, Ari Rabinowitz -Original Message- From: Chuq Von Rospach [mailto:[EMAIL PROTECTED]] Sent: Sunday, May 13, 2001 2:21 AM To: J C Lawrence Cc: Ian White; Mailman Users Group Subject: Re: [Mailman-Users] big lists, big messages On 5/12/01 10:51 PM, "J C Lawrence" <[EMAIL PROTECTED]> wrote: > I find this curious. I have MAX_RCPT_TO set to 5, and to broadcast > 30 messages to a subscriber base of 1,000 (ie 6,000 spool entries) > through qrunner to the MTA (postfix) on a dual PII-333 takes just > over 6 seconds once started. Admittedly that's an appreciable time, > but its also not that long a time in the lands of lock contention > and lock timeouts. It all comes down to how fast your MTA accepts messages. If you're running postfix with DNS deferred, you rock. If you're running sendmail with DNS on, it's a lot slower. So it's something you have to judge based on your own system. -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users *** Bear Stearns is not responsible for any recommendation, solicitation, offer or agreement or any information about any transaction, customer account or account activity contained in this communication. *** -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
Zowie! I seem to have started a major thread here! Thenks everyone for all the advice. I can't pretend to understand everything, since I'm not exactly an e-mail god, but I appreciate the suggestions and will try some. I too had considered going to an "abstract format" for the newsletter, but it looks like that might 1) not solve my problem, and 2) have troubles of its own. Would definitely mean more work for me, the webmaster. (My wife actually writes the newsletter.) I'll look into the use of other MTAs. I have a question in to my tech support people about that. I run a virtual server, and I'm not sure what is available in the way of alternative MTAs. I may be stuck with sendmail. We'll see. All that being said, I do think Mailman is going to be the way to go, just for its robust feature set and ease of use and configuration. I've already figured out how to place rotating ads in the headers and footers, and I appreciate the way it dynamically generates interface pages for users and list admins to configure their subscriptions and lists. Very cool! Again, thanks everyone for *all* the help! This is a really great list! -Eric Schmitz -- "So long, and thanks for all the fish." -Douglas Adams, 1952 - 2001 -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
On Sun, 13 May 2001, J C Lawrence wrote: > On Sun, 13 May 2001 16:16:44 -0700 (PDT) > alex wetmore <[EMAIL PROTECTED]> wrote: > > > On Sun, 13 May 2001, J C Lawrence wrote: > >> There's reason to keep the RCPT TO envelope reasonably small to > >> prevent triggering some ISP's SPAM traps. Specifically, this > >> appears to be one of the filter points that AOL uses (and of > >> course it then drops the caught mail silently without a bounce of > >> warning). > > > Do you have any details on AOL's spam filters? None of my AOL > > readers have complained about missing email. It looks like my > > largest list has 60 digested and 55 regular AOL members, so I'm > > guessing that their spam filter is set to a higher threshold than > > that. > > They appear to be using a multi-valued metric for their spam > detection -- and to change their metrics regularly. See: > > http://members.aol.com/adamkb/aol/mailfaq/dropped-mail.html#lists It doesn't sound like it makes a huge difference if they are batched up at once, or if they come very quickly on different messages. "If a mailing list server sends too much mail in too short a time to a number of AOL members, AOL will consider it questionable and delete it." alex -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
On Sun, 13 May 2001 16:16:44 -0700 (PDT) alex wetmore <[EMAIL PROTECTED]> wrote: > On Sun, 13 May 2001, J C Lawrence wrote: >> There's reason to keep the RCPT TO envelope reasonably small to >> prevent triggering some ISP's SPAM traps. Specifically, this >> appears to be one of the filter points that AOL uses (and of >> course it then drops the caught mail silently without a bounce of >> warning). > Do you have any details on AOL's spam filters? None of my AOL > readers have complained about missing email. It looks like my > largest list has 60 digested and 55 regular AOL members, so I'm > guessing that their spam filter is set to a higher threshold than > that. They appear to be using a multi-valued metric for their spam detection -- and to change their metrics regularly. See: http://members.aol.com/adamkb/aol/mailfaq/dropped-mail.html#lists -- J C Lawrence [EMAIL PROTECTED] -(*) http://www.kanga.nu/~claw/ The pressure to survive and rhetoric may make strange bedfellows -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
On Sun, 13 May 2001, J C Lawrence wrote: > There's reason to keep the RCPT TO envelope reasonably small to > prevent triggering some ISP's SPAM traps. Specifically, this > appears to be one of the filter points that AOL uses (and of course > it then drops the caught mail silently without a bounce of warning). Do you have any details on AOL's spam filters? None of my AOL readers have complained about missing email. It looks like my largest list has 60 digested and 55 regular AOL members, so I'm guessing that their spam filter is set to a higher threshold than that. alex -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
On Sun, 13 May 2001 08:46:57 -0700 (PDT) alex wetmore <[EMAIL PROTECTED]> wrote: > My mailman installation talks to the Windows 2000 SMTP MTA, where > there are no performance problems with having the batching set to > a very high number. Multiple queues can send out the same message > at the same time. Using high SMTP_MAX_RCPTS numbers results in a > lot less disk activity and also means that all of my mail going to > hotmail.com or aol.com can get sent in one transaction. There's reason to keep the RCPT TO envelope reasonably small to prevent triggering some ISP's SPAM traps. Specifically, this appears to be one of the filter points that AOL uses (and of course it then drops the caught mail silently without a bounce of warning). -- J C Lawrence [EMAIL PROTECTED] -(*) http://www.kanga.nu/~claw/ The pressure to survive and rhetoric may make strange bedfellows -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
On 5/13/01 1:21 AM, "Tib" <[EMAIL PROTECTED]> wrote: > If you're running on a 384k dsl then the only advertising is the stuff you put > up yourself because it's your server. Not true, but also not really relevant -- I was using it only as an example, since it's my home DSL. My big stuff is somewhere else with big pipes (> 7 DS-3s, last I looked) > I would think this would be a bit the opposite. Again -- you THINK. I've done. > What I have: > A personally built server (PIII 500, 256mb memory, 30g hd) > a 1.1 sdsl line > 5 fully hosted domains > 23 users (maybe a third of which are decently active for mail and web demands) > 5 email lists running on mailman, the largest of which is about 300 users. > mrtg to keep track of bandwidth usage > basic algebra Ah, thanks. You finally admit you don't know and haven't done. I've done lists up into the seven digit size. We've experimented with URLs and click-ins. If you're talking about 300 users max, it doesn't matter what you do. ANYTHING works. It doesn't scale, though. But believe what you want. Trust your algebra. I suggest you don't sell the service until you know for sure, though with real numbers. -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
On Sun, 13 May 2001 01:21:52 -0700 (PDT) tib <[EMAIL PROTECTED]> wrote: > On Sun, 13 May 2001, Chuq Von Rospach wrote: >> On 5/13/01 12:22 AM, "Tib" <[EMAIL PROTECTED]> wrote: >> If the piece of email is sponsored and has advertising, it's a >> HUGE problem. As was the original poster's note on this stuff. > If you're running on a 384k dsl then the only advertising is the > stuff you put up yourself because it's your server. Unsupoortable assumption. > I would think this would be a bit the opposite. For your mail > server to push anything it has to look up records on the domain > it's going to send to (which for 10k users I would assume to be a > lot of domains)... Which given a local cacheing name server with reasonably large and enforced minimum TTLs is a small overhead. > Again it would seem to me that the webserver would have an easier > time with the load than the mail server. Web server traffic is controllable, bandwidth profiles poorly (users complain), is unschedulable and tends to be bursty. Mail traffic is implicitly controllable, bandwidth profiles easily, is schedulable, and is as flat as you care to make it. -- J C Lawrence [EMAIL PROTECTED] -(*) http://www.kanga.nu/~claw/ The pressure to survive and rhetoric may make strange bedfellows -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
On Sat, 12 May 2001 23:19:15 -0700 Chuq Von Rospach <[EMAIL PROTECTED]> wrote: > On 5/12/01 10:43 PM, "J C Lawrence" <[EMAIL PROTECTED]> wrote: >> 3) If delivery failures are clogging your MTA queue and are >> noticably slowing delivery rates, you need to start thinking >> about reviewing your MTA configuration or using a different and >> more intelligent MTA. > MTA configuration is huge. Yes, and no. I posit that some MTA's ala Postfix and QMail do a good enough job that they do solve 80% of the problem, which leave only rather rare cases being interestingfrom a tuning viewpoint. > (Oh, and at times, you'll find you have to redo stuff, because > there are places where the paradigms change -- you scale to some > point, and then you have to rethink how you do things...) Yeah, I've been caught there a couple times. -- J C Lawrence [EMAIL PROTECTED] -(*) http://www.kanga.nu/~claw/ The pressure to survive and rhetoric may make strange bedfellows -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
On Sun, 13 May 2001 00:41:16 -0700 Chuq Von Rospach <[EMAIL PROTECTED]> wrote: > On 5/13/01 12:22 AM, "Tib" <[EMAIL PROTECTED]> wrote: >> Who has email that does not have web access at the time they get >> their email? > Not a huge number, but not zero. As wireless mobile becomes more > significant, it'll be a growing issue, not a shrinking one. Which of course means that your web page must now be WAP friendly and desktop browser friendly. Reading the web effectively on a connected Palm requires a rather different view on web design. >> The load of an entire batch (rough 300meg estimate) will also be >> spread out more over the course of a few days as everyone checks >> their mail and may or may not look at that message and cause it >> to draw on the url or click/paste it into a browser themselves. > No, it won't. There's a huge peak in the few hours after delivery, > which drops radically after that and stretches out over about ten > days or so. I wandered over to Akami a few months ago and looked at the graphs of attack time and decay rates for sites that hit Slashdot. The leading edge of the peak is typically measured in single digit minutes. The decay rates vary fairly widely depending on the time of posting, but fall into a small number of basic camps. >> It all boils down to a matter of how you want to use your server. > And whether you want to pay for peak load capacities on your web > server or the lower, spread out push capacities on your e-mail. There's also the possiblity of using your ISP's MTAs as smarthosts (which many ISPs actively encourage and some enforce/require). -- J C Lawrence [EMAIL PROTECTED] -(*) http://www.kanga.nu/~claw/ The pressure to survive and rhetoric may make strange bedfellows -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
On Sun, 13 May 2001 00:22:08 -0700 (PDT) tib <[EMAIL PROTECTED]> wrote: > On Sat, 12 May 2001, J C Lawrence wrote: >> Your math is off as it ignores RCPT-TO envelope size. > So throw in a few more bytes times 10k and it gets even bigger Chuq has already addressed this point thoroughly. >> Actually, list servers are generically disk IO bound with the >> primary factor in the disk IO being open/close/unlink time not >> read/write time. > I'm not quite the hardware guru I'd like to be yet - explain in > english please? MTAs are fundamentally disk IO bound. This is one of the reason that several large mail houses run their mail spools on silicon disks (ie hard drives which sit on SCSI busses, and which are made out of RAM). Within the bottleneck that is formed by disk IO, read/write time is a smaller percentage. open/close/delete occupy the lion's share of the IO time. Ergo, volume of data processed is really not that concerning -- its the NUMBER of data items (spool entries) processed that is concerning. >> This assumes of course that the audience has web access, and in >> particular has web access at the time and on the device they >> would normally read the messages. >> >> Example : It wouldn't work for me reading on the train on my >> laptop. > Who has email that does not have web access at the time they get > their email? I do. > Granted I suppose it's possible that you would download your mail > ahead of time before leaving home and then opening your laptop on > the train. The laptop has a Wavelan card. When its at home it auto-joins the home network and picks up the mail my server has spooled for it, and delivers the mail it has waiting for outbound. I then go to work, laptop in tow, and (generally) have no more 'net access from the laptop until I retunr home. (I'm still not happy with this setup, but it kinda works ATM). Given the number of others I see on the train reading mail I'm nowhere near alone in this (tho our methods will vary on how we got there -- mine is excessively complex). -- J C Lawrence [EMAIL PROTECTED] -(*) http://www.kanga.nu/~claw/ The pressure to survive and rhetoric may make strange bedfellows -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
On Sat, 12 May 2001 23:21:05 -0700 Chuq Von Rospach <[EMAIL PROTECTED]> wrote: > On 5/12/01 10:51 PM, "J C Lawrence" <[EMAIL PROTECTED]> wrote: >> I find this curious. I have MAX_RCPT_TO set to 5, and to >> broadcast 30 messages to a subscriber base of 1,000 (ie 6,000 >> spool entries) through qrunner to the MTA (postfix) on a dual >> PII-333 takes just over 6 seconds once started. Admittedly >> that's an appreciable time, but its also not that long a time in >> the lands of lock contention and lock timeouts. > It all comes down to how fast your MTA accepts messages. True, thus my query. > If you're running postfix with DNS deferred, you rock. If you're > running sendmail with DNS on, it's a lot slower. So it's something > you have to judge based on your own system. Aye, that's one of my more common performance refrains along with, "install and use a local cacheing name server!". Fair dunkum tho -- I'd forgotton the painful side effects of turning of DNS checks for Sendmail, and had thus ritely assumed that non-DNS-check was a default starting point. -- J C Lawrence [EMAIL PROTECTED] -(*) http://www.kanga.nu/~claw/ The pressure to survive and rhetoric may make strange bedfellows -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
On Sat, 12 May 2001, Chuq Von Rospach wrote: > On 5/12/01 6:52 PM, "Ian White" <[EMAIL PROTECTED]> wrote: > > Which variables should be changed? It looks like the following could use > > some changes: > > SMTP_MAX_RCPTS = 500 > > Between 5 and 10 - that should be set for any mailman installation. 500 is > way too high for reasonable performance. This blanket statement is not always correct. It strongly depends on the MTA that you are using. My mailman installation talks to the Windows 2000 SMTP MTA, where there are no performance problems with having the batching set to a very high number. Multiple queues can send out the same message at the same time. Using high SMTP_MAX_RCPTS numbers results in a lot less disk activity and also means that all of my mail going to hotmail.com or aol.com can get sent in one transaction. The statement above is true for many Unix MTAs where the queue runner can only send one message to one file at time. If you only give the MTA a couple of messages to party on then it will only be able to talk to a couple of remote servers at a time, so sending out a single message could take quite a while. What this really means is that you should understand how your MTA works and choose a SMTP_MAX_RCPTS that works well for your MTA. I just think that blanket statements such as "500 is way too high for reasonable performance" should be avoided, as you never know what MTA the admin is using. alex -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
On Sun, 13 May 2001, Chuq Von Rospach wrote: > On 5/13/01 12:22 AM, "Tib" <[EMAIL PROTECTED]> wrote: > If the piece of email is sponsored and has advertising, it's a HUGE problem. > As was the original poster's note on this stuff. If you're running on a 384k dsl then the only advertising is the stuff you put up yourself because it's your server. > Um, Tib -- I've actually done this and measured it. Have you? You ignored > pretty much all of the data in my message while responding to JC -- and you > still sound like you're guessing at this stuff. Have you actually run > servers in both configurations and measured response like I have? Because my > numbers say you're wrong. Well your numbers aren't my numbers, so of course they'll say I'm wrong, and if that's all you're intent on proving then feel free to flame/shoot down every one of my comments to me rather than wasting everyone elses time with it. > No, it won't. There's a huge peak in the few hours after delivery, which > drops radically after that and stretches out over about ten days or so. > > > It all boils down to a matter of how you want to use your server. > And whether you want to pay for peak load capacities on your web server or > the lower, spread out push capacities on your e-mail. I would think this would be a bit the opposite. For your mail server to push anything it has to look up records on the domain it's going to send to (which for 10k users I would assume to be a lot of domains), rather than just firing back a response to an ip for a data request (and will only lookup and put resolved host names into your log only if you tell it to do so). Again it would seem to me that the webserver would have an easier time with the load than the mail server. > Again, I have to ask. Have you actually done this with large lists? I'm > curious if your numbers disagree with mine, or if you don't have numbers to > back up what you're saying. I'd like to know if the data I've gathered may > or may not be typical. If you've done it, what size lists? What I have: A personally built server (PIII 500, 256mb memory, 30g hd) a 1.1 sdsl line 5 fully hosted domains 23 users (maybe a third of which are decently active for mail and web demands) 5 email lists running on mailman, the largest of which is about 300 users. mrtg to keep track of bandwidth usage basic algebra Do I run a newsletter like the original topic of this thread was about? Nope. Am I guessing a little about traffic and how it could happen if changed to the way I talked about? Yep. I say a little because I used to work for a company that was completely based off of this kind of newsletter and traffic, to the tune of about 8 million addresses, 200 mail servers, and about that many web servers. Dealing with systems on two vastly different scales (millions of users at my old company, and a handful at my own server - versus 10k towards the middle of the spectrum) turn experience into best guesses. I have never maxed out my bandwidth with list traffic, so my comments are /estimates/ of numbers and /estimates/ on behaviors of people based on myself as being 'joe average'. Tib -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
On 5/13/01 12:44 AM, "Roger B.A. Klorese" <[EMAIL PROTECTED]> wrote: > Do you expect wireless mobile NOT to have web access? Hell, I use my web > access when mobile much more than my email access. No, but I expect wireless mobile to have limitations on display -- not to the level that WAP hoses you (into basically unusability), but in other directions. -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
On Sun, 13 May 2001, Chuq Von Rospach wrote: > As wireless mobile becomes more significant, it'll be a growing issue, > not a shrinking one. Do you expect wireless mobile NOT to have web access? Hell, I use my web access when mobile much more than my email access. -- ROGER B.A. KLORESE [EMAIL PROTECTED] PO Box 14309San Francisco, CA 94114 "There is only one real blasphemy -- the refusal of joy!" -- Paul Rudnick -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
On 5/13/01 12:22 AM, "Tib" <[EMAIL PROTECTED]> wrote: > Who has email that does not have web access at the time they get their email? Not a huge number, but not zero. As wireless mobile becomes more significant, it'll be a growing issue, not a shrinking one. > True: users who have a bland interest in the article that is being > posted will probably not pull down the URL - what's the problem with this, > it's > saved bandwidth If the piece of email is sponsored and has advertising, it's a HUGE problem. As was the original poster's note on this stuff. > However it's now pulling on a webserver rather than > pushing through an email server, which depending on it's configuration may > have > as few as one outward connection at a time (which I'm not sure how rare /that/ > is, my original sendmail server only had 1 outbound, but my current qmail > setup > allows up to 50 outward smtp connections at once), and a webserver is designed > to be pulled on a lot. Most basic configurations will start anywhere from 10 > to > 20 instances at once, and the 'high bandwidth' demand won't be high at all if > you keep the presentation simple and clean and non-graphicy - just you like > you'd get in a 30k message that got pushed out to you. Plus, if you really > want > to get finicky, the manner of processing http requests churns out fewer > headers > and data than mail. Um, Tib -- I've actually done this and measured it. Have you? You ignored pretty much all of the data in my message while responding to JC -- and you still sound like you're guessing at this stuff. Have you actually run servers in both configurations and measured response like I have? Because my numbers say you're wrong. > The load of an entire batch (rough 300meg > estimate) will also be spread out more over the course of a few days as > everyone checks their mail and may or may not look at that message and cause > it > to draw on the url or click/paste it into a browser themselves. No, it won't. There's a huge peak in the few hours after delivery, which drops radically after that and stretches out over about ten days or so. > It all boils down to a matter of how you want to use your server. And whether you want to pay for peak load capacities on your web server or the lower, spread out push capacities on your e-mail. Again, I have to ask. Have you actually done this with large lists? I'm curious if your numbers disagree with mine, or if you don't have numbers to back up what you're saying. I'd like to know if the data I've gathered may or may not be typical. If you've done it, what size lists? -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
On Sat, 12 May 2001, J C Lawrence wrote: > Your math is off as it ignores RCPT-TO envelope size. So throw in a few more bytes times 10k and it gets even bigger > 1) If your messages are getting corrupted, AT ALL, you have far more > serious problems than how fast your system is able to deliver a list > broadcast. Something is fundamentally broken and that needs to be > fixed, now, before you start worrying about much else. > > 2) Transfer failures given a good MTA and reasonable choice of RCPT > TO bundle size should cause minimal problem in delivery rates for > the list broadcast. Empirical testing here, for my admittedly very > atypical membership/domain distribution suggests that between 5 and > 25 is my sweet spot under Postfix. Chuq IIRC has found for his > locad under Sendmail that somewhere in the 30 range is his sweet > pot. Vour mileage will vary. > > 3) If delivery failures are clogging your MTA queue and are > noticably slowing delivery rates, you need to start thinking about > reviewing your MTA configuration or using a different and more > intelligent MTA. Point > Actually, list servers are generically disk IO bound with the > primary factor in the disk IO being open/close/unlink time not > read/write time. I'm not quite the hardware guru I'd like to be yet - explain in english please? > This assumes of course that the audience has web access, and in > particular has web access at the time and on the device they would > normally read the messages. > > Example : It wouldn't work for me reading on the train on my > laptop. Who has email that does not have web access at the time they get their email? Granted I suppose it's possible that you would download your mail ahead of time before leaving home and then opening your laptop on the train. > Result? > > I get to read what I'm interested in on the list, and any time I > post to the list, I get to see everything on that thread until it > dies. Meanwhile the rest of the list passes me silently by. I > can of course go read the main list folder any time I want, which > I do periodically to update the key word lists -- but usually its > enough to just read -interesting. > > That sort of autmation would be simply impossible with the web-based > distribution you describe. The load shift is a valid concern, and does change from a 'push' to a 'pull' format. True: users who have a bland interest in the article that is being posted will probably not pull down the URL - what's the problem with this, it's saved bandwidth. Anyone who is /actually/ interested and can take the time to read/parse through 30k worth of email will take an extra second to click on a link or do what not to pull up what they want. And I do understand that this means that a person may actually pull on that link a number of times on different occasions. However it's now pulling on a webserver rather than pushing through an email server, which depending on it's configuration may have as few as one outward connection at a time (which I'm not sure how rare /that/ is, my original sendmail server only had 1 outbound, but my current qmail setup allows up to 50 outward smtp connections at once), and a webserver is designed to be pulled on a lot. Most basic configurations will start anywhere from 10 to 20 instances at once, and the 'high bandwidth' demand won't be high at all if you keep the presentation simple and clean and non-graphicy - just you like you'd get in a 30k message that got pushed out to you. Plus, if you really want to get finicky, the manner of processing http requests churns out fewer headers and data than mail. Also, unless 100% of your user base is actually reading the article, you're going to be saving bandwidth (some people may open an article more than just once, but most will read it once and be done, in which case you will come out a little bit ahead instead of a lot). The load of an entire batch (rough 300meg estimate) will also be spread out more over the course of a few days as everyone checks their mail and may or may not look at that message and cause it to draw on the url or click/paste it into a browser themselves. It all boils down to a matter of how you want to use your server. There are as many good points for as against everything I said and have been responded to with, however all the items presented have not really made an effect on my point of view. There are two sides to this debate, the client side (where the rebuttal for my view came from), and the server side (which I was presenting). If you want the best of both worlds, perhaps consider doing similar to many newspapers have done with their web presence/maling list user base. Send out a smaller email message which briefs the content of the full article and then present a link at the end which lets users get an idea of the article rather than flying blind on whether or not to follow the link or not. Cut a 30k message to 10k users down to maybe 5k in this way and y
Re: [Mailman-Users] big lists, big messages
On 5/12/01 10:51 PM, "J C Lawrence" <[EMAIL PROTECTED]> wrote: > I find this curious. I have MAX_RCPT_TO set to 5, and to broadcast > 30 messages to a subscriber base of 1,000 (ie 6,000 spool entries) > through qrunner to the MTA (postfix) on a dual PII-333 takes just > over 6 seconds once started. Admittedly that's an appreciable time, > but its also not that long a time in the lands of lock contention > and lock timeouts. It all comes down to how fast your MTA accepts messages. If you're running postfix with DNS deferred, you rock. If you're running sendmail with DNS on, it's a lot slower. So it's something you have to judge based on your own system. -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
On 5/12/01 10:43 PM, "J C Lawrence" <[EMAIL PROTECTED]> wrote: > 1) If your messages are getting corrupted, AT ALL, you have far more > serious problems than how fast your system is able to deliver a list > broadcast. Yeah. TCP guarantees the data is good. You basically can't get corruption unless one side or the other is broken. > Chuq IIRC has found for his > locad under Sendmail that somewhere in the 30 range is his sweet > pot. Vour mileage will vary. I use ten these days with good results. > 3) If delivery failures are clogging your MTA queue and are > noticably slowing delivery rates, you need to start thinking about > reviewing your MTA configuration or using a different and more > intelligent MTA. MTA configuration is huge. And it's a big black art. And not all of what makes for good delivery is obvious. You have to sit down and get into the system to your elbows, as you grow. (Oh, and at times, you'll find you have to redo stuff, because there are places where the paradigms change -- you scale to some point, and then you have to rethink how you do things...) -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
On Sat, 12 May 2001 20:36:37 -0700 Chuq Von Rospach <[EMAIL PROTECTED]> wrote: > QRUNNER_LOCK_LIFETIME be longer than however long it takes to > deliver your really large message, or the system will assume the > lock is dead and break it. You don't want it too large, because if > the system does lock for some reason, it'll take a HUGE time to > recover, so it takes some experimentation. > If this value is too small, it's VERY possible for qrunner to be > delivering your newsletter, for the lock to time out, and for > another qrunner to start up break the lock as dead, and start > sending your newsletter out again -- you send out random numbers > of duplicates until you figure it out or happen to get in below > the lock lifetime I find this curious. I have MAX_RCPT_TO set to 5, and to broadcast 30 messages to a subscriber base of 1,000 (ie 6,000 spool entries) through qrunner to the MTA (postfix) on a dual PII-333 takes just over 6 seconds once started. Admittedly that's an appreciable time, but its also not that long a time in the lands of lock contention and lock timeouts. -- J C Lawrence [EMAIL PROTECTED] -(*) http://www.kanga.nu/~claw/ The pressure to survive and rhetoric may make strange bedfellows -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
On Sat, 12 May 2001 19:20:21 -0700 (PDT) tib <[EMAIL PROTECTED]> wrote: > To take another approach, mail out a link to the newsletter rather > than the ENTIRE newsletter to each person. Do the math; if you're > mailing out a letter that's 30k, to 10,000 users. that's gonna be > 300 megs of data that's getting pumped through your system... Your math is off as it ignores RCPT-TO envelope size. > ... on a weekly basis, with each one having a possiblity of > becoming corrupt or having other failures in transfer. 1) If your messages are getting corrupted, AT ALL, you have far more serious problems than how fast your system is able to deliver a list broadcast. Something is fundamentally broken and that needs to be fixed, now, before you start worrying about much else. 2) Transfer failures given a good MTA and reasonable choice of RCPT TO bundle size should cause minimal problem in delivery rates for the list broadcast. Empirical testing here, for my admittedly very atypical membership/domain distribution suggests that between 5 and 25 is my sweet spot under Postfix. Chuq IIRC has found for his locad under Sendmail that somewhere in the 30 range is his sweet pot. Vour mileage will vary. 3) If delivery failures are clogging your MTA queue and are noticably slowing delivery rates, you need to start thinking about reviewing your MTA configuration or using a different and more intelligent MTA. > What about turning the newsletter into a webarticle that you post > on the net somewhere and send out just the link to all those 10k > subscribers. First of all it'll cut your data output through the > mailserver IMMENSELY, a 1k message that goes out rather than 30k > will only end up being 10 megs rather than 300. Actually, list servers are generically disk IO bound with the primary factor in the disk IO being open/close/unlink time not read/write time. > Second, the time it takes to SEND that batch of messages will be > drasticly reduced. And last, if you have to make any changes to > the message or find a critical editing error AFTER it's out, you > can correct it in one place (the single web page) rather than > having to mail out an error correction message to those 10k people > all over again. This assumes of course that the audience has web access, and in particular has web access at the time and on the device they would normally read the messages. Example : It wouldn't work for me reading on the train on my laptop. > Downside? You cut off a potentially significant percentage of your audience. You make third part archiving and disconnected analysis and review of your material more difficult. You make automated processing of your content (ie email driven automation systems) much more difficult. You turn a low bandwidth, disconnected, permanent operation (to the user, the mail arrives asynchronously with his other operations), into a high bandwidth connected operation which is explicitly transitory (close the browser and its gone). There are reasons I don't subscribe to lists as you describe above. > I don't really see any big ones. I subscribe to something over 130 lists at this point. I don't read them all, and I don't even try and keep up with many of them. However, I subscribe to them as I'm interested in something about them, so I *do* want to participate to at least some degree. So I automate and reduce the problem. Procmail appropirate files all list messages into per-list folders. As it files each message it also compares each message against a list of key words for hat list, and if the message matches, it also files a copy of the message in a folder called -interesting. I then read -interesting. Should I reply to a message on the list, or send a message to the list (from -interesting or wherever), procmail will notice when it receives that message and goes to file in the appropriate folders. Noticing that it is from me, it logs the MessageID into a DB. Any future messages for that list are also checked for that MessageID in their References: and In-Reply-To: headers, and if there's a match, that message's MessageID is dropped into the DB and a copy of it is dropped into -interesting. Result? I get to read what I'm interested in on the list, and any time I post to the list, I get to see everything on that thread until it dies. Meanwhile the rest of the list passes me silently by. I can of course go read the main list folder any time I want, which I do periodically to update the key word lists -- but usually its enough to just read -interesting. That sort of autmation would be simply impossible with the web-based distribution you describe. -- J C Lawrence [EMAIL PROTECTED] -(*) http://www.kanga.nu/~claw/ The pressure to survive and rhetoric may make strange bedfellows -- Mailman-Users maillist - [EMAI
Re: [Mailman-Users] big lists, big messages
On 5/12/01 7:20 PM, "Tib" <[EMAIL PROTECTED]> wrote: > To take another approach, mail out a link to the newsletter rather than the > ENTIRE newsletter to each person. Do the math; Your math is wrong, though. > if you're mailing out a letter > that's 30k, to 10,000 users. that's gonna be 300 megs of data that's getting > pumped through your system, on a weekly basis, with each one having a > possiblity of becoming corrupt or having other failures in transfer. > First of all it'll cut > your data output through the mailserver IMMENSELY, a 1k message that goes out > rather than 30k will only end up being 10 megs rather than 300. Second, the > time it takes to SEND that batch of messages will be drasticly reduced. There are any number of problems with this. First, if you have a measley 384K DSL connection, you can send out 300 megs of data in about 2.5 hours. It's just not a big deal from the start. Second, you're ignoring the economies of scale that comes from SMTP putting this together. If you set your SMTP_MAX_RCPTS to 10, which is still a fairly small number, then any domains that have 2 or more subscribers get lumped together, and the newsletter gets sent once to each domain. If you have more than ten subscribers to a domain, it gets sent once for every ten subscribers, a 90% reduction. If the list is at all typical, you'll be sending 25% or more less than a simple mathematical multiplication might assume. Third, you ignore the overhead of the SMTP protocol -- there's control data that goes along with the content data. Not much, but it'll add to the time somewhat. Fourth, and most important -- you're ignoring the overhead imposed by SMTP and the acceptance speed of the client systems. Those are significant, and tend to make the actual size of the message (within reason) not all that significant. For a non-optimized system, I'd say the difference in delivery time of 10,000 subscribers for a 2-3K message and a 30K message is probably double, not the 10X or more you'd imply by simply looking at the size difference. If you have a fast network connection and figure out how to optimize your delivery setup, then for small lists like this, your primary delay vector is the speed the other side is willing to accept it. On my big server, where 40,000 is my smallest list, I literally can't get the system up to speed before it's delivered, and for something that size, my big limiter is the speed other systems respond to my delivery request. And there are logistic problems to your suggestion -- if you send a URL to the users instead of the message, the number of eyeballs that person's advertisers will get plummets by a huge fraction. Send URLs, and many users won't bother much of the time. It's a really bad idea for most environments, especially if they're doing newsletter or marketing oriented stuff. > And > last, if you have to make any changes to the message or find a critical > editing > error AFTER it's out, A legitimate issue, but one which you mostly deal with by making sure you edit the thing properly the first time. Sometimes URLs change, but there are ways to deal with that (but I can't talk about how I did it, at least not right now). > Downside? I don't really see any big ones. This type of change would only > require that you have a web server resource to point this at. Downsides: 1) you're forcing the user to make an action to read the newsletter. If you give users an excuse, they'll take it. Your actual readership will plummet, not good for the advertisers. Any little hassle is still a hassle. 2) it requires having a web server and a network connection that can handle the PEAK load when everyone clicks the URL at the same time after you deliver it. You'll see 50-70% of those users click through in the first 90 minutes after delivery of the URL, so you probably need a faster link than you'd need to deliver the actual message, since delivery of a piece of e-mail isn't peak-load sensitive (the receiver doesn't care all that much if they get it now or in two hours from now) -- but web stuff is (if they get that URL now, and the server is overloaded, you're toast). In fact, if you start doing newsletters with a lot of clickthrough stuff to a central server, I've found it's best to SLOW DOWN delivery to spread out the load on your web server. 3) remember all that bandwidth you thought you saved by going with the small piece of e-mail? You just gave it all back (and probably more) via the web server. But instead of it going out in a controlled manner because you're pushing email, it's going out in a demand manner, because now they're pulling it on THEIR schedule, not yours. Which means you need a faster net link and bigger web server to handle the peak load, because the users won't wait. That kind of delivery model is a false economy. That content is going to go out the network somewhere, sometime. Send it out by e-mail, it goes out on your schedule. Send out links and wait for web clicks,
Re: [Mailman-Users] big lists, big messages
On 5/12/01 6:52 PM, "Ian White" <[EMAIL PROTECTED]> wrote: > Which variables should be changed? It looks like the following could use > some changes: > SMTP_MAX_RCPTS = 500 Between 5 and 10 - that should be set for any mailman installation. 500 is way too high for reasonable performance. > MAX_DELIVERY_THREADS = 0 Zero. That's experimental and has been found not to be overly useful, because the threads lock each other up. It'd need to be redone as processes. Make sure you have: DELIVERY_MODULE = 'SMTPDirect' Not sendmail QRUNNER_LOCK_LIFETIME = hours(10) QRUNNER_PROCESS_LIFETIME = minutes(15) QRUNNER_MAX_MESSAGES = 300 I set QRUNNER_LOCK_LIFETIME = hours(20) QRUNNER_PROCESS_LIFETIME = hours(2) QRUNNER_MAX_MESSAGES = 50 What's important for a large list is that QRUNNER_LOCK_LIFETIME be longer than however long it takes to deliver your really large message, or the system will assume the lock is dead and break it. You don't want it too large, because if the system does lock for some reason, it'll take a HUGE time to recover, so it takes some experimentation. If this value is too small, it's VERY possible for qrunner to be delivering your newsletter, for the lock to time out, and for another qrunner to start up break the lock as dead, and start sending your newsletter out again -- you send out random numbers of duplicates until you figure it out or happen to get in below the lock lifetime -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
To take another approach, mail out a link to the newsletter rather than the ENTIRE newsletter to each person. Do the math; if you're mailing out a letter that's 30k, to 10,000 users. that's gonna be 300 megs of data that's getting pumped through your system, on a weekly basis, with each one having a possiblity of becoming corrupt or having other failures in transfer. What about turning the newsletter into a webarticle that you post on the net somewhere and send out just the link to all those 10k subscribers. First of all it'll cut your data output through the mailserver IMMENSELY, a 1k message that goes out rather than 30k will only end up being 10 megs rather than 300. Second, the time it takes to SEND that batch of messages will be drasticly reduced. And last, if you have to make any changes to the message or find a critical editing error AFTER it's out, you can correct it in one place (the single web page) rather than having to mail out an error correction message to those 10k people all over again. Downside? I don't really see any big ones. This type of change would only require that you have a web server resource to point this at. A slight negative would be that depending on the email client the person was using, it would either load the url into the mail reader and be a transparent change to your users (say with outlook or express or even netscape mail) or your users would have to copy/paste the url into a browser (pine and other command line clients and possibly eudora and others). It's up to you, but this is just my two cents. On Sat, 12 May 2001, J C Lawrence wrote: > On Sat, 12 May 2001 16:32:05 -0500 > Eric Schmitz <[EMAIL PROTECTED]> wrote: > > > Does anyone have any experience in using Mailman with very large > > lists? I have a newsletter that goes out to 10,000 people each > > week, and each issue is about 30K. Majordomo seems to be tripping > > over itself, and many of my subscribers are not receiving their > > newsletter. This is Very Bad, especially when the advertisers find > > out! :-( > > Not to downplay mailman, this is unlikely to be a Majordomo problem, > and it quite likely to be a question of MTA choice and > configuration. > > -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
On Sat, 12 May 2001, Chuq Von Rospach wrote: > I ran lists larger than that on majordomo. You can run them on mailman, > also, but with 2.0, you need to set things up properly -- out of the box, > you may well have some problems with mailman and lists of that size. The two > issues are the single-threaded delivery (which means nothing else happens > while it's delivering that list, which can be a problem if the server does a > lot of different lists) and the length it takes to deliver, which can cause > qrunner locks to time out, which can lead to duplicated delivery and chaos > -- easy to fix by tweaking a few variables, if you know to look for it. Which variables should be changed? It looks like the following could use some changes: SMTP_MAX_RCPTS = 500 MAX_DELIVERY_THREADS = 0 and the lock management defaults. Any suggestions for tuning these other than what is listed in the Defaults.py file? Ian -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
On 5/12/01 2:32 PM, "Eric Schmitz" <[EMAIL PROTECTED]> wrote: > Majordomo seems to be tripping over itself, and many > of my subscribers are not receiving their newsletter. This is Very Bad, > especially when the advertisers find out! :-( I agree with JC -- make sure you know what's failing before you change list servers, because if some other part of the system is broken, you don't want to fix the wrong thing. I ran lists larger than that on majordomo. You can run them on mailman, also, but with 2.0, you need to set things up properly -- out of the box, you may well have some problems with mailman and lists of that size. The two issues are the single-threaded delivery (which means nothing else happens while it's delivering that list, which can be a problem if the server does a lot of different lists) and the length it takes to deliver, which can cause qrunner locks to time out, which can lead to duplicated delivery and chaos -- easy to fix by tweaking a few variables, if you know to look for it. But make sure you know what the problem is first. I doubt it's majordomo, if it's set up right. -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
Re: [Mailman-Users] big lists, big messages
On Sat, 12 May 2001 16:32:05 -0500 Eric Schmitz <[EMAIL PROTECTED]> wrote: > Does anyone have any experience in using Mailman with very large > lists? I have a newsletter that goes out to 10,000 people each > week, and each issue is about 30K. Majordomo seems to be tripping > over itself, and many of my subscribers are not receiving their > newsletter. This is Very Bad, especially when the advertisers find > out! :-( Not to downplay mailman, this is unlikely to be a Majordomo problem, and it quite likely to be a question of MTA choice and configuration. -- J C Lawrence [EMAIL PROTECTED] -(*) http://www.kanga.nu/~claw/ The pressure to survive and rhetoric may make strange bedfellows -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
[Mailman-Users] big lists, big messages
Does anyone have any experience in using Mailman with very large lists? I have a newsletter that goes out to 10,000 people each week, and each issue is about 30K. Majordomo seems to be tripping over itself, and many of my subscribers are not receiving their newsletter. This is Very Bad, especially when the advertisers find out! :-( I'm thinking of trying the same list through Mailman. Any thoughts? -Eric -- Mailman-Users maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users
