At 10:40 AM -0800 2005-02-08, Cabel Sasser wrote:
I'll get right to the point: one of our Mailman lists currently has 121,047 members. It uses full personalization and auto bounce processing. I'm sure you know where this is headed... :)
Cool! That may be the largest Mailman-hosted currently operating single mailing list, at least the largest we've heard of so far. I would imagine that lists.apple.com probably does more traffic on a daily basis, but so far as I know, it doesn't have any individual lists that are anywhere near that large.
And from the company that created Audion, Unison, CandyBar, Stattoo, the amazingly funny "USA vs. Japan Food" page, and so much more. Very, very cool.
My problem is not with the mailing, but seemingly rather with the user management.
Once you get to this kind of size, the web administration tools start to fail. To make things work, I imagine you're going to have to go to command-line administration techniques. Most of what you might want to do should be able to be done directly from the command-line using existing scripts and programs, but you may need a few additional tools to help round out the mix.
I recall hearing not too long ago that the folks who run the Mailman-hosted mailing lists for FreeBSD.org have gotten to the point where they don't even bother with the web interface any more. However, while they have a very large mailing list server (by Mailman standards), their largest list is probably around 15,000 subscribers. They just have a much higher traffic load, sometimes a hundred messages or more on lists with tens of thousands of subscribers.
Your situation is likely to share some problems with theirs, but you might also have some unique issues.
1. Should I split the list into a series of smaller lists? What's the best way to do that easily?
This is the first that I have personally heard of a single list being this large. I would imagine that splitting things out into multiple sub-lists with a parent umbrella list, would be a pretty big help. It shouldn't be too hard to use the command-line tool list_members to get a list of your subscribers to the list, and then to split that up unto multiple smaller chunks.
It will take a bit more work to copy out all their preferences settings so that you can replicate those on the new sub-list.
2. Is there any way to "optimize" this database, other than throwing more memory at the machine, etc.?
Optimizing Python pickles? No, not that I know of.
3. Any general advice for handling ridiculously large lists?
You've already done a lot of homework, but see also FAQ 1.24. It's not going to add that much to your knowledge base, but you might as well round out the list.
Disk and memory are going to be your biggest constraints. If you can set up a RAID 1+0 filesystem (preferably using an XServe RAID array), plus throw as much memory at the machine as you can, that will get you most of the performance gain you're likely to see in terms of hardware tuning.
I've seen in the FAQ (1.15) that The Guardian ran a list with 147,000 subscribers, and I'm wondering how it was done! I've read performance tuning (4.11) but that seems to focus on the mail _delivery_, which is generally working just fine, not the user management -- think bursting Python pickles.
Most of the "large list" issues that we have dealt with so far have been with relation to servers with more lists, each of which is moderately large, but the machine has a high level of incoming traffic to those lists. Do the math, and you find that they are doing very high delivery volumes, relatively speaking.
The Web administration issues tend to be less common, and therefore they've gotten less attention.
-- Brad Knowles, <[EMAIL PROTECTED]>
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755SAGE member since 1995. See <http://www.sage.org/> for more info. ------------------------------------------------------ Mailman-Users mailing list [email protected] http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
