Re: [Mailman-Users] Performance issues with 12K subscribers and 14K subscribers and personalized footers.

[Mark Sapiro]

On 10/01/2013 01:05 PM, Joseph Brennan wrote:
> 
> Worth noting, Mailman is a lot faster if you don't send to 1 at a time.


Often true, but the subject of this thread included "personalized
footers" which require sending separately to each recipient.

Also, what should be of concern is not only the time to deliver all the
messages from Mailman to the MTA, but also the total time from the start
of Mailman's delivery to the MTA until the MTA has finished delivering
to the recipient's MXs. I suspect this latter time is not so
dramatically impacted by Mailman's VERP or personalization.

-- 
Mark Sapiro The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Performance issues with 12K subscribers and 14K subscribers and personalized footers.

[Joseph Brennan]

Worth noting, Mailman is a lot faster if you don't send to 1 at a time.

If they're all at the same domain, like our mass mail to staff and 
students, it is really fast:


 smtp to  for 25570 recips, completed in 39.777 seconds
 smtp to  for 41153 recips, completed in 84.754 seconds

Even if they are at many different domains it does well:

 smtp to  for 2016 recips, completed in 158.363 seconds
 smtp to  for 2199 recips, completed in 701.392 seconds


Joseph Brennan
Manager, Email and Systems Applications
Columbia University Information Technology



--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Performance issues with 12K subscribers and 14K subscribers and personalized footers.

[Ralf Hildebrandt]

* Stephen J. Turnbull :
> Bryan Wright writes:
> 
>  > Now I just need to fine-tune sendmail so it speeds up as well, but
>  > that's a question for elsewhere.
> 
> Just a guess, but it may not be sendmail per se.  If there are issues
> with the DNS, that could result in multiple long delays, giving the
> half-hour time you're seeing for not all that many messages.  Of
> course sendmail may be throttled for some reason, or an upstream
> gateway is throttling your host, or something.

But all that should be clearly visible in the log
-- 
Ralf Hildebrandt   Charite Universitätsmedizin Berlin
ralf.hildebra...@charite.deCampus Benjamin Franklin
http://www.charite.de  Hindenburgdamm 30, 12203 Berlin
Geschäftsbereich IT, Abt. Netzwerk fon: +49-30-450.570.155
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Performance issues with 12K subscribers and 14K subscribers and personalized footers.

[Stephen J. Turnbull]

Bryan Wright writes:

 > Now I just need to fine-tune sendmail so it speeds up as well, but
 > that's a question for elsewhere.

Just a guess, but it may not be sendmail per se.  If there are issues
with the DNS, that could result in multiple long delays, giving the
half-hour time you're seeing for not all that many messages.  Of
course sendmail may be throttled for some reason, or an upstream
gateway is throttling your host, or something.
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Performance issues with 12K subscribers and 14K subscribers and personalized footers.

[Bryan Wright]

Mark,

Thanks for the help.  Followed the instructions in the FAQ  to get sendmail 
listening on an alternate port in deferred mode.  That made a huge difference, 
although not as speedy as your environment, 8122 messages made it from mailman 
to sendmail in about 30 minutes.

Sep 30 10:07:18 2013 (22267) 
 smtp to 
catherine.klein for 8122 recips, completed in 1782.344 seconds

Now I just need to fine-tune sendmail so it speeds up as well, but that's a 
question for elsewhere.

Thanks again for the help.

Bryan Wright


-Original Message-
From: Mark Sapiro [mailto:m...@msapiro.net] 
Sent: Saturday, September 28, 2013 3:33 PM
To: Bryan Wright; mailman-users@python.org
Subject: Re: [Mailman-Users] Performance issues with 12K subscribers and 14K 
subscribers and personalized footers.

On 09/27/2013 10:40 PM, Bryan Wright wrote:
> 
> So back to the speed.   If multiple qrunners won't fix the issue, is
> there any way to speed this up? To get it to process multiple outgoing 
> messages at once?


Before trying to send from Mailman in parallel, you need to look into speeding 
up the SMTP interaction between Mailman and sendmail, bevcause that's where the 
real delay exists.


> Could I create multiple lists and then just have the lists subscribed 
> to the one list?  Then each of these lists could use a qrunner?


Yes and no. You could create an umbrella list or use regular_include_lists (see 
the FAQ at <http://wiki.list.org/x/TIA9>) to create multiple outgoing messages 
with fewer recipients, but there is no guarantee that these message's out/ 
queue entries would all wind up in different OutgoingRunner slices. Actually 
ensuring that each message is processed by a different OutgoingRunner would, I 
think, require some kludgey code mods. It would be easier and more productive 
to modify SMTPDirect to use multiple threads. (There is an 'experimental'
MAX_DELIVERY_THREADS setting documented in Defaults.py, but it is not currently 
implemented.)


> Could disk access time be an issue?  Could I place the mailman queue 
> directory in a ram disk?


Putting the queue on a ram disk won't help. The queue entry is only picked up 
at the start of processing and whether it takes microseconds or milliseconds, 
it is not the source of any significant delay.

Putting the server's swap space on a ram disk could help, but there are other 
things to try first.

As I said in my original reply, see the FAQs at <http://wiki.list.org/x/doA9> 
and <http://wiki.list.org/x/q4A9>. A local caching DNS may help and if sendmail 
is doing DNS verification at incoming SMTP time, stopping that will really 
help. Also ensure you haven't changed the default

SMTP_MAX_SESSIONS_PER_CONNECTION = 0

setting.

Note: as a point of reference, my production server is not very high powered, 
runs CentOS 5, and has no specific performance tuning. Messages from Mailman 
use Mailman's VERP feature so each recipient gets an individual message whether 
or not personalization is on. Mailman delivers over 50 messages per second to 
Postfix or 12,000 messages in 4 minutes. This is based on up to 600+ messages 
(my maximum list size is currently 629), and may slow down some with larger 
lists as Postfix gets busier sending, but I'm sure delivery to Postfix of 
12,000 messages would take minutes as opposed to hours.

-- 
Mark Sapiro The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Performance issues with 12K subscribers and 14K subscribers and personalized footers.

[Mark Sapiro]

On 09/27/2013 10:40 PM, Bryan Wright wrote:
> 
> So back to the speed.   If multiple qrunners won't fix the issue, is
> there any way to speed this up? To get it to process multiple outgoing
> messages at once? 


Before trying to send from Mailman in parallel, you need to look into
speeding up the SMTP interaction between Mailman and sendmail, bevcause
that's where the real delay exists.


> Could I create multiple lists and then just have the lists subscribed to
> the one list?  Then each of these lists could use a qrunner?


Yes and no. You could create an umbrella list or use
regular_include_lists (see the FAQ at ) to
create multiple outgoing messages with fewer recipients, but there is no
guarantee that these message's out/ queue entries would all wind up in
different OutgoingRunner slices. Actually ensuring that each message is
processed by a different OutgoingRunner would, I think, require some
kludgey code mods. It would be easier and more productive to modify
SMTPDirect to use multiple threads. (There is an 'experimental'
MAX_DELIVERY_THREADS setting documented in Defaults.py, but it is not
currently implemented.)


> Could disk access time be an issue?  Could I place the mailman queue
> directory in a ram disk? 


Putting the queue on a ram disk won't help. The queue entry is only
picked up at the start of processing and whether it takes microseconds
or milliseconds, it is not the source of any significant delay.

Putting the server's swap space on a ram disk could help, but there are
other things to try first.

As I said in my original reply, see the FAQs at
 and . A local
caching DNS may help and if sendmail is doing DNS verification at
incoming SMTP time, stopping that will really help. Also ensure you
haven't changed the default

SMTP_MAX_SESSIONS_PER_CONNECTION = 0

setting.

Note: as a point of reference, my production server is not very high
powered, runs CentOS 5, and has no specific performance tuning. Messages
from Mailman use Mailman's VERP feature so each recipient gets an
individual message whether or not personalization is on. Mailman
delivers over 50 messages per second to Postfix or 12,000 messages in 4
minutes. This is based on up to 600+ messages (my maximum list size is
currently 629), and may slow down some with larger lists as Postfix gets
busier sending, but I'm sure delivery to Postfix of 12,000 messages
would take minutes as opposed to hours.

-- 
Mark Sapiro The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Performance issues with 12K subscribers and 14K subscribers and personalized footers.

[Bryan Wright]

That makes sense now.   And explains why all my attempts to speed up processing 
have hear no effect.   So the final message should be logged once delivery is 
complete.

So back to the speed.   If multiple qrunners won't fix the issue, is there any 
way to speed this up? To get it to process multiple outgoing messages at once?

Could I create multiple lists and then just have the lists subscribed to the 
one list?  Then each of these lists could use a qrunner?

Could disk access time be an issue?  Could I place the mailman queue directory 
in a ram disk?

Sent from my Verizon Wireless 4G LTE Smartphone



 Original message 
From: Mark Sapiro 
Date: 09/28/2013 12:00 AM (GMT-06:00)
To: mailman-users@python.org
Subject: Re: [Mailman-Users] Performance issues with 12K subscribers and 14K 
subscribers and personalized footers.


On 09/27/2013 09:32 PM, Mark Sapiro wrote:
> Also, you only see them while delivery of the post is
> proceeding because of multiple qrunners, but I'm not sure why locking
> doesn't stop even that.


Actually, it's because OutgoingRunner doesn't lock the list. I was
confusing it with IncomingRunner which does lock the list.

--
Mark Sapiro The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/bryan.wright%40rigaku.com
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Performance issues with 12K subscribers and 14K subscribers and personalized footers.

[Mark Sapiro]

On 09/27/2013 09:32 PM, Mark Sapiro wrote:
> Also, you only see them while delivery of the post is
> proceeding because of multiple qrunners, but I'm not sure why locking
> doesn't stop even that.


Actually, it's because OutgoingRunner doesn't lock the list. I was
confusing it with IncomingRunner which does lock the list.

-- 
Mark Sapiro The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Performance issues with 12K subscribers and 14K subscribers and personalized footers.

[Mark Sapiro]

On 09/27/2013 11:17 AM, Bryan Wright wrote:
> 
> I've configured mailman for 48 outgoing qrunners, and have tried as many as 
> 72 qrunners, with no noticeable difference, it's like it's using only the 1st 
> outgoing qrunner to process the message and the rest stay there idle.  And 
> then it uses that one qrunner process to create each separate message.  So 
> 12k individual messages processed by one qrunner thread each time.  Is there 
> some way to speed this up and/or get it to use multiple outgoing qrunners 
> simultaneously for the same post, since with personalization it has to create 
> individual copies of each message?


In your case this will not help at all. A single OutgoingRunner will be
just as fast (or slow) because all the messages are for one or at most
two lists, and list lock contention will thwart any attempt at
parallelism. Furthermore, parallism only works for multiple messages.
Delivery of a single post is always handled by a single OutgoingRunner.


> The other odd thing is it doesn't log all 12K messages in the smtp log from 
> mailman, but it logs more than 1.  I would have thought it would either log 
> only 1 message, or all 12K messages.
> 
> Here is the SMTP log from mailman:
> 
> Sep 26 14:13:17 2013 (12407) 
>  smtp to newsletter for 1 
> recips, completed in 10.368 seconds
> Sep 26 15:11:57 2013 (12439) 
>  smtp to newsletter for 1 
> recips, completed in 8.107 seconds
> Sep 26 15:42:19 2013 (12381) 
>  smtp to newsletter for 1 
> recips, completed in 0.086 seconds
> Sep 26 16:56:31 2013 (12458) 
>  smtp to newsletter for 1 
> recips, completed in 11.179 seconds
> Sep 26 17:21:12 2013 (12469) 
>  smtp to newsletter for 1 
> recips, completed in 1.012 seconds
> Sep 26 19:10:10 2013 (12429) 
>  smtp to newsletter for 1 
> recips, completed in 10.988 seconds
> Sep 26 20:23:43 2013 (12449)  
> smtp to newsletter for 1 recips, completed in 6.958 seconds
> Sep 27 00:47:25 2013 (12380) 
>  smtp to newsletter for 1 
> recips, completed in 6.096 seconds
> Sep 27 01:17:13 2013 (12448) 
>  smtp to newsletter for 1 
> recips, completed in 6.166 seconds
> Sep 27 02:11:27 2013 (12442) 
>  smtp to newsletter for 1 
> recips, completed in 5.095 seconds
> Sep 27 02:30:36 2013 (12399) 
>  smtp to newsletter for 1 
> recips, completed in 6.013 seconds
> Sep 27 02:40:08 2013 (12472) 
>  smtp to newsletter for 1 
> recips, completed in 5.356 seconds
> Sep 27 05:07:31 2013 (12380) 
>  smtp to newsletter for 1 
> recips, completed in 0.083 seconds
> Sep 27 05:08:20 2013 (12422) 
>  smtp to newsletter for 1 
> recips, completed in 0.077 seconds
> Sep 27 05:08:23 2013 (12474) 
>  smtp to newsletter for 1 
> recips, completed in 0.073 seconds
> Sep 27 05:31:24 2013 (12438) 
>  smtp to newsletter for 1 
> recips, completed in 10.251 seconds
> Sep 27 06:56:34 2013 (12469) 
>  smtp to newsletter for 1 
> recips, completed in 5.967 seconds
> Sep 27 06:56:50 2013 (12396) 
>  smtp to newsletter for 1 
> recips, completed in 5.153 seconds
> Sep 27 07:02:33 2013 (12452) 
>  smtp to newsletter for 1 
> recips, completed in 10.248 seconds
> Sep 27 07:59:36 2013 (12406) 
>  smtp to newsletter for 1 
> recips, completed in 11.085 seconds
> Sep 27 08:05:46 2013 (12417) 
>  smtp to newsletter for 1 
> recips, completed in 10.141 seconds
> Sep 27 08:10:38 2013 (12449) 
>  smtp to newsletter for 1 
> recips, completed in 10.232 seconds
> Sep 27 08:11:40 2013 (12435) 
>  smtp to newsletter for 1 
> recips, completed in 10.093 seconds
> Sep 27 08:28:37 2013 (12473) 
>  smtp to newsletter for 1 
> recips, completed in 5.124 seconds
> Sep 27 08:35:08 2013 (12409) 
>  smtp to newsletter for 1 
> recips, completed in 10.346 seconds
> Sep 27 09:51:39 2013 (12428)  
> smtp to newsletter for 1 recips, completed in 5.045 seconds
> Sep 27 10:34:31 2013 (12393) 
>  smtp to newsletter for 1 
> recips, completed in 0.049 seconds
> Sep 27 11:39:33 2013 (12396) 
>  smtp to newsletter for 1 
> recips, completed in 15.162 seconds
> Sep 27 11:40:15 2013 (12389) 
>  smtp to newsletter for 1 
> recips, completed in 10.073 seconds
> Sep 27 11:53:41 2013 (12438) 
>  smtp to newsletter for 1 
> recips, completed in 5.098 seconds
> Sep 27 13:00:33 2013 (12430) 
>  smtp to newsletter for 1 
> recips, completed in 10.125 seconds


These messages are all Mailman generated notices of some kind. None of
them is delivery of the actual post to the 12 or 14 K subscribers.
Further, these do not indicate a backlogged qrunner since these messages
are many minutes apart and processing times, although very long for a
single recipient, are only a few seconds.

Most of these entries look like a single notice, although there may have
been others that are in the slice being processed for the big delivery,
but these

Sep 27 05:07:31 2013 (12380)
 smtp to newsletter
for 1 recips, completed in 0.083 seconds
Sep 27 05:08:20 2013 (12422)
 smtp to newsletter
for 1 recips, completed in 0.077 seconds
Sep 27 05:08:23 2013 (12474)
 smtp to newsletter
for 1 recips,

[Mailman-Users] Performance issues with 12K subscribers and 14K subscribers and personalized footers.

[Bryan Wright]

Hello,

I'm trying to figure out some performance issues with mailman.  I have it 
configured for SMTP direct, and pointed at localhost, port 25.  The MTA is 
sendmail.  Sendmail is configured to use our exchange server as a smart host.  
We have two lists configured and send out 1 message a month to each list.  It 
usually takes about 12-24 hours to finish sending all the messages.  There are 
never more than about 5 to 10 messages in the sendmail queue at any given time, 
so this leads me to believe that the bottleneck is not on the sendmail side.

I've configured mailman for 48 outgoing qrunners, and have tried as many as 72 
qrunners, with no noticeable difference, it's like it's using only the 1st 
outgoing qrunner to process the message and the rest stay there idle.  And then 
it uses that one qrunner process to create each separate message.  So 12k 
individual messages processed by one qrunner thread each time.  Is there some 
way to speed this up and/or get it to use multiple outgoing qrunners 
simultaneously for the same post, since with personalization it has to create 
individual copies of each message?

The other odd thing is it doesn't log all 12K messages in the smtp log from 
mailman, but it logs more than 1.  I would have thought it would either log 
only 1 message, or all 12K messages.

Here is the SMTP log from mailman:

Sep 26 14:13:17 2013 (12407)  
smtp to newsletter for 1 recips, completed in 10.368 seconds
Sep 26 15:11:57 2013 (12439)  
smtp to newsletter for 1 recips, completed in 8.107 seconds
Sep 26 15:42:19 2013 (12381)  
smtp to newsletter for 1 recips, completed in 0.086 seconds
Sep 26 16:56:31 2013 (12458)  
smtp to newsletter for 1 recips, completed in 11.179 seconds
Sep 26 17:21:12 2013 (12469)  
smtp to newsletter for 1 recips, completed in 1.012 seconds
Sep 26 19:10:10 2013 (12429)  
smtp to newsletter for 1 recips, completed in 10.988 seconds
Sep 26 20:23:43 2013 (12449)  
smtp to newsletter for 1 recips, completed in 6.958 seconds
Sep 27 00:47:25 2013 (12380)  
smtp to newsletter for 1 recips, completed in 6.096 seconds
Sep 27 01:17:13 2013 (12448)  
smtp to newsletter for 1 recips, completed in 6.166 seconds
Sep 27 02:11:27 2013 (12442)  
smtp to newsletter for 1 recips, completed in 5.095 seconds
Sep 27 02:30:36 2013 (12399)  
smtp to newsletter for 1 recips, completed in 6.013 seconds
Sep 27 02:40:08 2013 (12472)  
smtp to newsletter for 1 recips, completed in 5.356 seconds
Sep 27 05:07:31 2013 (12380)  
smtp to newsletter for 1 recips, completed in 0.083 seconds
Sep 27 05:08:20 2013 (12422)  
smtp to newsletter for 1 recips, completed in 0.077 seconds
Sep 27 05:08:23 2013 (12474)  
smtp to newsletter for 1 recips, completed in 0.073 seconds
Sep 27 05:31:24 2013 (12438)  
smtp to newsletter for 1 recips, completed in 10.251 seconds
Sep 27 06:56:34 2013 (12469)  
smtp to newsletter for 1 recips, completed in 5.967 seconds
Sep 27 06:56:50 2013 (12396)  
smtp to newsletter for 1 recips, completed in 5.153 seconds
Sep 27 07:02:33 2013 (12452)  
smtp to newsletter for 1 recips, completed in 10.248 seconds
Sep 27 07:59:36 2013 (12406)  
smtp to newsletter for 1 recips, completed in 11.085 seconds
Sep 27 08:05:46 2013 (12417)  
smtp to newsletter for 1 recips, completed in 10.141 seconds
Sep 27 08:10:38 2013 (12449)  
smtp to newsletter for 1 recips, completed in 10.232 seconds
Sep 27 08:11:40 2013 (12435)  
smtp to newsletter for 1 recips, completed in 10.093 seconds
Sep 27 08:28:37 2013 (12473)  
smtp to newsletter for 1 recips, completed in 5.124 seconds
Sep 27 08:35:08 2013 (12409)  
smtp to newsletter for 1 recips, completed in 10.346 seconds
Sep 27 09:51:39 2013 (12428)  
smtp to newsletter for 1 recips, completed in 5.045 seconds
Sep 27 10:34:31 2013 (12393)  
smtp to newsletter for 1 recips, completed in 0.049 seconds
Sep 27 11:39:33 2013 (12396)  
smtp to newsletter for 1 recips, completed in 15.162 seconds
Sep 27 11:40:15 2013 (12389)  
smtp to newsletter for 1 recips, completed in 10.073 seconds
Sep 27 11:53:41 2013 (12438)  
smtp to newsletter for 1 recips, completed in 5.098 seconds
Sep 27 13:00:33 2013 (12430)  
smtp to newsletter for 1 recips, completed in 10.125 seconds

Thanks in advance for any advice/help you can give.

Bryan
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org