Re: [Mailman-Users] slow "out" queue

[Mark Sapiro]

Anil Jangity wrote:

>Actually, this is a internal SmartHost that doesn't require SMTP auth.
>I think a while back I had configured Mailman with --with-mailhost=localhost, 
>a bunch of pages also showed "localhost" as the email domain for some of the 
>list details. That's not what I wanted.


The --with-mailhost config sets DEFAULT_EMAIL_HOST in Defaults.py. This
has nothing to do with the MTA used for sending.


>I assume there won't be issues if all I do is change that variable to 
>SmartHostA in the mm_cfg.py file. I would like everything else to be in tact, 
>and only want the outgoing mail to be directed to smart host. I will test it 
>out in our UAT.


You want to set SMTPHOST (and SMTPPORT if you want other than 25) in
mm_cfg.py. That's all you need to change.

-- 
Mark Sapiro The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] slow "out" queue

[Anil Jangity]

Actually, this is a internal SmartHost that doesn't require SMTP auth.
I think a while back I had configured Mailman with --with-mailhost=localhost, a 
bunch of pages also showed "localhost" as the email domain for some of the list 
details. That's not what I wanted.

I assume there won't be issues if all I do is change that variable to 
SmartHostA in the mm_cfg.py file. I would like everything else to be in tact, 
and only want the outgoing mail to be directed to smart host. I will test it 
out in our UAT.

Thanks!
Anil


On Mar 8, 2012, at 2:23 PM, Mark Sapiro wrote:

> Anil Jangity wrote:
>> 
>> Mailman receives the mail and then posts to the same local MTA on the host 
>> which sends it out to Smart host.
>> I am not sure we can't just have mailman send directly to the smarthost?
>> 
>> What are the implications (on the web site as well as how Mailman functions) 
>> if I change DEFAULT_EMAIL_HOST to be SmartHostA?
> 
> 
> Most likeley, SmartHostA will refuse to relay the mail because Mailman
> hasn't authenticated to it as an authorized user.
> 
> See  for a patch that
> implements SMTP authentication for Mailman.
> 
> -- 
> Mark Sapiro The highway is for gamblers,
> San Francisco Bay Area, Californiabetter use your sense - B. Dylan
> 

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] slow "out" queue

[Mark Sapiro]

Anil Jangity wrote:
>
>Mailman receives the mail and then posts to the same local MTA on the host 
>which sends it out to Smart host.
>I am not sure we can't just have mailman send directly to the smarthost?
>
>What are the implications (on the web site as well as how Mailman functions) 
>if I change DEFAULT_EMAIL_HOST to be SmartHostA?


Most likeley, SmartHostA will refuse to relay the mail because Mailman
hasn't authenticated to it as an authorized user.

See  for a patch that
implements SMTP authentication for Mailman.

-- 
Mark Sapiro The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] slow "out" queue

[Anil Jangity]

Hi Mark,

Thanks. We're getting better performance now once we added this to our sendmail 
.mc file:

FEATURE(`nocanonify')

I wanted to see if we can do further improvements. 
Right now our email flow is like this for lists:

Internet -> ServerA:25 (/etc/aliases mailman) -> ServerA -> SmartHostA

Mailman receives the mail and then posts to the same local MTA on the host 
which sends it out to Smart host.
I am not sure we can't just have mailman send directly to the smarthost?

What are the implications (on the web site as well as how Mailman functions) if 
I change DEFAULT_EMAIL_HOST to be SmartHostA?

Thanks,
Anil


PS: Now I am getting about 3000 recips in about 5 minutes. Still not the 
greatest but it's better than before.


On Feb 22, 2012, at 6:53 PM, Mark Sapiro wrote:

> Anil Jangity wrote:
>> 
>> This sendmail server is setup to relay the mail to smarthost. When I inject 
>> a test mail on port 25 with RCPT and FROM to be a made up junk domain, the 
>> email is still delivered *immediately* to the smarthost, there is no 
>> resolver issues.
>> 
>> If sendmail was really doing DNS lookups, I would think it would be stuck 
>> locally or throw a bounce/error right away.
>> It just seems like it's running serially, and not dequeueing multiple 
>> messages to the smarthost at a time.
> 
> 
> This issue is not a Mailman issue per se. It is either a Sendmail
> configuration issue or something to do with the interaction of
> Sendmail with the smarthost.
> 
> We have given you all the information we have to help you with this. If
> that isn't sufficient, you'll have to pursue this using Sendmail
> support resources. See
> .
> 
> -- 
> Mark Sapiro The highway is for gamblers,
> San Francisco Bay Area, Californiabetter use your sense - B. Dylan
> 

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] slow "out" queue

[Mark Sapiro]

Anil Jangity wrote:
>
>This sendmail server is setup to relay the mail to smarthost. When I inject a 
>test mail on port 25 with RCPT and FROM to be a made up junk domain, the email 
>is still delivered *immediately* to the smarthost, there is no resolver issues.
>
>If sendmail was really doing DNS lookups, I would think it would be stuck 
>locally or throw a bounce/error right away.
>It just seems like it's running serially, and not dequeueing multiple messages 
>to the smarthost at a time.


This issue is not a Mailman issue per se. It is either a Sendmail
configuration issue or something to do with the interaction of
Sendmail with the smarthost.

We have given you all the information we have to help you with this. If
that isn't sufficient, you'll have to pursue this using Sendmail
support resources. See
.

-- 
Mark Sapiro The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] slow "out" queue

[Anil Jangity]

Mark, thanks for the very detailed info below.
I am still stuck on this.

This sendmail server is setup to relay the mail to smarthost. When I inject a 
test mail on port 25 with RCPT and FROM to be a made up junk domain, the email 
is still delivered *immediately* to the smarthost, there is no resolver issues.

If sendmail was really doing DNS lookups, I would think it would be stuck 
locally or throw a bounce/error right away.
It just seems like it's running serially, and not dequeueing multiple messages 
to the smarthost at a time.

Anil
PS: Putting a local DNS cacheing server is a possibility, but not yet.


On Feb 1, 2012, at 10:56 PM, Mark Sapiro wrote:

> Anil Jangity wrote:
> 
>> Does mailman actually (if it works) send to 15 recipients at a time - per 
>> SMTP session?
> 
> 
> Mailman's use of the term session is not consistent with the RFCs. What
> Mailman calls SMTP_MAX_SESSIONS_PER_CONNECTION is actually in RFC 821,
> 2821, 5321 terms, max transactions per session. I.e. one session is
> one connection from HELO/EHLO to QUIT and one transaction is one MAIL
> FROM and everything that follows up to the next MAIL FROM.
> 
> Unless you have some specific MTA reason for setting
> SMTP_MAX_SESSIONS_PER_CONNECTION, you should leave it at the default
> value of 0 meaning unlimited.
> 
> If a message is personalized or VERPd, SMTP_MAX_RCPTS is ignored
> because you can't send a personalized or VERPd message to more than
> one recipient. Otherwize, SMTP_MAX_RCPTS is the maximum number of RCPT
> TO recipients in one MAIL FROM transaction.
> 
> Note that 6000 users per 35 minutes is less than 3 per second. This is
> extremely slow. I typically see 30 to over 60 recipients per second on
> a rather ordinary production server with full VERP.
> 
> See my reply to your original post in this thread at
> 
> and look at the FAQs linked in that post for hints. In particular, do
> not do DNS verification of RCPT TO domains during SMTP from Mailman.
> 
> -- 
> Mark Sapiro The highway is for gamblers,
> San Francisco Bay Area, Californiabetter use your sense - B. Dylan
> 

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] slow "out" queue

[Mark Sapiro]

Anil Jangity wrote:

>Does mailman actually (if it works) send to 15 recipients at a time - per SMTP 
>session?


Mailman's use of the term session is not consistent with the RFCs. What
Mailman calls SMTP_MAX_SESSIONS_PER_CONNECTION is actually in RFC 821,
2821, 5321 terms, max transactions per session. I.e. one session is
one connection from HELO/EHLO to QUIT and one transaction is one MAIL
FROM and everything that follows up to the next MAIL FROM.

Unless you have some specific MTA reason for setting
SMTP_MAX_SESSIONS_PER_CONNECTION, you should leave it at the default
value of 0 meaning unlimited.

If a message is personalized or VERPd, SMTP_MAX_RCPTS is ignored
because you can't send a personalized or VERPd message to more than
one recipient. Otherwize, SMTP_MAX_RCPTS is the maximum number of RCPT
TO recipients in one MAIL FROM transaction.

Note that 6000 users per 35 minutes is less than 3 per second. This is
extremely slow. I typically see 30 to over 60 recipients per second on
a rather ordinary production server with full VERP.

See my reply to your original post in this thread at

and look at the FAQs linked in that post for hints. In particular, do
not do DNS verification of RCPT TO domains during SMTP from Mailman.

-- 
Mark Sapiro The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] slow "out" queue

[Anil Jangity]

Yes.
What exactly does that option do/mean? 
Does mailman actually (if it works) send to 15 recipients at a time - per SMTP 
session?

On Feb 1, 2012, at 5:02 PM, Carl Zwanzig wrote:

> On 2/1/2012 3:57 PM, Anil Jangity wrote:
> 
>> I have these set in mm_cfg.py:
>> 
>> SMTP_MAX_RCPTS = 15
>> SMTP_MAX_SESSIONS_PER_CONNECTION = 50
>> 
>> It seems Qrunner isn't honoring those?
> 
> Did you restart the qrunners after changing those settings?
> 
> z!
> 

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] slow "out" queue

[Anil Jangity]

Actually, looking at the code... it seems it might be because I have these set 
for some of the lists.

VERP_PERSONALIZED_DELIVERIES = 1
OWNERS_CAN_ENABLE_PERSONALIZATION = 1

Thanks

On Feb 1, 2012, at 9:01 PM, Anil Jangity wrote:

> Yes.
> What exactly does that option do/mean? 
> Does mailman actually (if it works) send to 15 recipients at a time - per 
> SMTP session?
> 
> On Feb 1, 2012, at 5:02 PM, Carl Zwanzig wrote:
> 
>> On 2/1/2012 3:57 PM, Anil Jangity wrote:
>> 
>>> I have these set in mm_cfg.py:
>>> 
>>> SMTP_MAX_RCPTS = 15
>>> SMTP_MAX_SESSIONS_PER_CONNECTION = 50
>>> 
>>> It seems Qrunner isn't honoring those?
>> 
>> Did you restart the qrunners after changing those settings?
>> 
>> z!
>> 
> 

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] slow "out" queue

[Carl Zwanzig]

On 2/1/2012 3:57 PM, Anil Jangity wrote:


I have these set in mm_cfg.py:

SMTP_MAX_RCPTS = 15
SMTP_MAX_SESSIONS_PER_CONNECTION = 50

It seems Qrunner isn't honoring those?


Did you restart the qrunners after changing those settings?

z!

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] slow "out" queue

[Anil Jangity]

It seems it is doing about 6000 users in 35 minutes. 
The problem seems to be that sendmail is doing one email connection per 
recipient.

I have these set in mm_cfg.py:

SMTP_MAX_RCPTS = 15
SMTP_MAX_SESSIONS_PER_CONNECTION = 50

It seems Qrunner isn't honoring those?



> 
> Look at Mailman's 'smtp' log. You undoubtedly have a huge backlog in
> the out queue because you aren't processing outgoing SMTP fast enough
> to keep up with demand. In the smtp log you will see entries like
> 
> Jan 20 09:41:14 2012 (17283)  smtp to listname for 223
> recips, completed in 3.500 seconds
> 
> You will observe that each entry's timestamp is equal to that of the
> previous entry plus this entry's 'completed in' time indicating no
> delay between messages. Each entry represents processing of one out/
> queue entry. You will probably also note that the processing rate is
> much less than the 64 recipients/second in the above message which is
> from a full VERP installation.
> 

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] slow "out" queue

[Mark Sapiro]

Anil Jangity wrote:

>We just moved a very old mailman infrastructure mailman to Solaris and 
>sendmail.
>It seems, when a message is posted, the "out" directory doesn't seem to clear 
>out fast enough. 
>
>   1K   archive
>   1K   bounces
>   1K   commands
>   1K   in
>   1K   news
> 403K   out
>   1K   retry
>   1K   shunt
>   1K   virgin
>
>The size just stays like that forever and slowly decrements. When I look in 
>/var/log/syslog (mail log of sendmail), I see deliveries going through.


Look at Mailman's 'smtp' log. You undoubtedly have a huge backlog in
the out queue because you aren't processing outgoing SMTP fast enough
to keep up with demand. In the smtp log you will see entries like

Jan 20 09:41:14 2012 (17283)  smtp to listname for 223
recips, completed in 3.500 seconds

You will observe that each entry's timestamp is equal to that of the
previous entry plus this entry's 'completed in' time indicating no
delay between messages. Each entry represents processing of one out/
queue entry. You will probably also note that the processing rate is
much less than the 64 recipients/second in the above message which is
from a full VERP installation.


>I do a 'truss' of the OutgoingRunner python process and I see:
>
>% truss -p 11997
>recv(10, 0x00748454, 8192, 0)   (sleeping...)
>...
>% 
>
>
>What is it waiting for?


Most likely an SMTP reply from sendmail. See the FAQ's at
 and . This
is almost certainly a case of sendmail not keeping up with Mailman's
demand and if so, will require sendmail tuning to fix.

-- 
Mark Sapiro The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

[Mailman-Users] slow "out" queue

[Anil Jangity]

We just moved a very old mailman infrastructure mailman to Solaris and sendmail.
It seems, when a message is posted, the "out" directory doesn't seem to clear 
out fast enough. 

   1K   archive
   1K   bounces
   1K   commands
   1K   in
   1K   news
 403K   out
   1K   retry
   1K   shunt
   1K   virgin

The size just stays like that forever and slowly decrements. When I look in 
/var/log/syslog (mail log of sendmail), I see deliveries going through.
I do a 'truss' of the OutgoingRunner python process and I see:

% truss -p 11997
recv(10, 0x00748454, 8192, 0)   (sleeping...)
...
% 


What is it waiting for?
--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org