Begin forwarded message:
From: "Andrew Falgout" <[EMAIL PROTECTED]>
Date: Mon Aug 25, 2003 8:22:05 pm Europe/London
To: "Richard Barrett" <[EMAIL PROTECTED]>
Subject: Re: [Mailman-Users] Mailman / Sendmail horribly slowperformance..suggestions please?!?
Reply-To: "Andrew Falgout" <[EMAIL PROTECTED]>
Problem solved. It was archiving. That's all I needed was to figure out
what it was pausing on. I turned off archival and informed my manager that
the overhead associated with it would be too much for the system to bear.
It is FLYING through the queued mails now and delivering them now.
Been a lifesaver, and THANKS!!!
----- Original Message ----- From: "Richard Barrett" <[EMAIL PROTECTED]> To: "Andrew Falgout" <[EMAIL PROTECTED]> Sent: Monday, August 25, 2003 12:11 PM Subject: Re: [Mailman-Users] Mailman / Sendmail horribly slowperformance..suggestions please?!?
On Monday, August 25, 2003, at 04:22 pm, Andrew Falgout wrote:
I have not modified the default values in the SMTPDirect module orMailman's qrunner process each message through a pipeline.
Defaults.py. If I could get qrunner to tell me anything about what
message(s) it is working I could tell you how many recipients were on
that
list. Each list in the current system should be no more than 2-5
people.
We moved over a small subset of mailing lists from the older majordomo
server. However, there are some lists with several hundres recipients
on it
that will be moving in the future when these issues get worked out.
Orphaned locks? (ocean) root $ ls -lt (~mailman/locks) total 16 -rw-rw-r-- 2 mailman mailman 55 Aug 25 2003 qrunner.lock -rw-rw-r-- 2 mailman mailman 55 Aug 25 2003 qrunner.lock.ocean.otr.usm.edu.16302 -rw-rw-r-- 2 mailman mailman 62 Aug 25 2003 gifted.studies.lock -rw-rw-r-- 2 mailman mailman 62 Aug 25 2003 gifted.studies.lock.ocean.otr.usm.edu.16302
smtp log Aug 25 10:16:47 2003 (16302) smtp for 1 recips, completed in 1.400 seconds Aug 25 10:16:47 2003 (16302) smtp for 1 recips, completed in 3.799 seconds Aug 25 10:16:49 2003 (16302) smtp for 1 recips, completed in 1.968 seconds Aug 25 10:16:49 2003 (16302) smtp for 1 recips, completed in 1.990 seconds
I'm guessing this would mean that there are not any orphaned locks. When I restart qrunner by hand, I kill the proces and remove all the locks from the locks directory.
I thought the 3.799 seconds in the smtp log was qrunner telling you how long it took to send the mail through the MTA.
Attached is a modified version of the file $prefix/Mailman/Handlers/HandlerAPI.py
This logs extra information about the progress of each message through the pipeline into the smtp log. This may give some insight as to where the qrunner is spending its time.
The changes in the revised file are as follows:
/Users/richard/development/mailman-htdig/mailman-2.0.13/Mailman/ Handlers
bash-2.05a$ diff -u HandlerAPI.py.orig HandlerAPI.py
--- HandlerAPI.py.orig Mon Aug 25 17:58:17 2003
+++ HandlerAPI.py Mon Aug 25 18:06:01 2003
@@ -73,10 +73,12 @@
def do_pipeline(mlist, msg, msgdata, pipeline):
+ t0x = t1x = t2x = time.time()
while pipeline:
modname = pipeline.pop(0)
mod = __import__('Mailman.Handlers.' + modname)
func = getattr(getattr(getattr(mod, 'Handlers'), modname),
'process')
+ t1x = time.time()
try:
pid = os.getpid()
func(mlist, msg, msgdata)
@@ -138,6 +140,10 @@
traceback.print_exc(file=s)
syslog('error', s.getvalue())
break
+ t2x = time.time()
+ syslog('smtp', 'do_pipeline: list %s, modname %s, startat
%.3f, durn %.3f' %
+ (mlist.internal_name(), modname, (t1x-t0x), (t2x-t1x)))
+ t1x = t2x
return pipeline
The revised files is:
----------------------------------------------------------------------- -----
----
About the best I can do to help.----- Original Message ----- From: "Richard Barrett" <[EMAIL PROTECTED]> To: "Andrew Falgout" <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Monday, August 25, 2003 9:52 AM Subject: Re: [Mailman-Users] Mailman / Sendmail horribly slowperformance..suggestions please?!?
On Monday, August 25, 2003, at 02:28 pm, Andrew Falgout wrote:
You will be using SMTP to the local MTA (which you say is Sendmail) ifThe error log is at 0 bytes. I have not overridden the Defaults.py which shows Defaults.py:MTA_ALIASES_STYLE = 'sendmail'. So I assume that means that sendmail is the MTA.
the following MM config variables are set as follows in either
mm_cfg.py or the Defaults.py file. The following are th standard MM
2.0.13 default values:
DELIVERY_MODULE = 'SMTPDirect' SMTPHOST = 'localhost'
Also are the following default values still in place:
SMTP_MAX_RCPTS = 500 MAX_DELIVERY_THREADS = 0
Aug 25 08:08:31 2003 (27798) smtp for 2 recips, completed in 1.045 seconds Aug 25 08:08:31 2003 (27798) smtp for 2 recips, completed in 1.149 seconds Aug 25 08:12:47 2003 (27798) smtp for 1 recips, completed in 0.208 seconds Aug 25 08:12:47 2003 (27798) smtp for 1 recips, completed in 0.241 seconds Aug 25 08:14:25 2003 (31184) smtp for 1 recips, completed in 0.701 seconds Aug 25 08:14:25 2003 (31184) smtp for 1 recips, completed in 0.716 seconds Aug 25 08:17:21 2003 (31184) smtp for 1 recips, completed in 0.167 seconds Aug 25 08:17:21 2003 (31184) smtp for 1 recips, completed in 0.179 seconds Aug 25 08:19:34 2003 (31184) smtp for 1 recips, completed in 3.962 seconds Aug 25 08:19:34 2003 (31184) smtp for 1 recips, completed in 4.234 seconds
This is just a few of the lines in my /var/log/mailman/smtp file.
There are
a few bounces from time to time, but not on great frequency. I did
send an
email to a test list, and attempted to time when I would see my email
address in the post log, but after 10 minutes I have not seen it.
This
really confuses me, because the mail system on this server delivers
directly to a mailbox in seconds. Why the delay?
I am going to assume that you had not made the edit to $prefix/Mailman/Handlers/SMTPDirect.py I suggested in one of my response and that these log entries are the regular ones?
Bear in mind that the number in brackets in these log entries - (27798) for instance - is the process id of the qurnner handling the message and the same qrunner is handling multiple messages.
The smtp log entries are interesting if you have a large backlog of messages in the qfiles directory because:
1. only a few recipients are being delivered for each log entry. Are your mailing list that sparely populated for subscribers? Do these smtp log entries tie in with post entries for lists with few members and /var/log/mail entries for outgoing mail through the MTA?
2. quite long periods - 2, 3, 4 minutes in the sample log entries you
give - elapse between log entries. But the log entries are for the
same
process so it has been continuing to run during these pregnant pauses.
But what is it doing?
Some thoughts. Mailman's qrunner can get slow if it is:
a. archiving and
b. the messages for a list are large messages and/or
c. the number of messages in the current archive of a list is large.
On the other hand, the pregnant pauses could be because MM is encountering long delays when opening SMTP connections to the MTA.
What sort of shape are the traffic/lists that MM is handling.
If you can catch the qrunner at it then maybe strace will give you some idea of what is doing: waiting on a socket open to the MTA maybe.
Another possibility is if by some means a bunch of lists have got orphaned locks over them so that the qrunner is spinning its wheels trying to find a list it can operate on. Does anything indicative of this show up in the $prefix/locks directory.
----- Original Message ----- From: "Richard Barrett" <[EMAIL PROTECTED]> To: "Andrew Falgout" <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Sunday, August 24, 2003 3:36 PM Subject: Re: [Mailman-Users] Mailman / Sendmail horribly slow performance..suggestions please?!?
On Sunday, August 24, 2003, at 12:51 am, Andrew Falgout wrote:
I'm currently running mailman-2.0.13-1 eith sendmail-8.11.6-25.72.
Posts
are processed very fast, but delivery is more than slow. It takes
1-2
minutes per post to delivery mail. Evaluating what is going on in
the
queue is a little more than cumbersome. Over the course of 3-4
days
mails
have been piling up in the qfiles directory to the tune of 4000
mails
and
waiting. There are mails 3 days old waiting to go out. I'm
looking
for
ways to evaluate what is in the queue, why it's in the queue, and
how
to
tell qrunner to report more information on what it's thinking.
Like
perhaps the msgid of the email it's trying to process at this time.
Are you using the local sendmail instance as the outgoing SMTP server?
Regardless, what do the Mailman post, smtp and smtp-failure logs show about mail being handled by the system. How does this tie in with the MTA's mail log.
If your outgoing mail is being handled by the local sendmail MTA
then
I
would normally expect Mailman to handoff to the MTA without much
delay.
If MM is having problems handing off to sendmail then you might find
some evidence of this in the smtp-failure log.-- Thank You, Andrew
-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- +--
+-
+-
+-+-+-
- Andrew Wilson Falgout (RHCE) Linux, E Pluribus
UNIX! -
- Systems Admin II, TIU Office: (601) 266-5603
-
- University of Southern Mississippi
-
- Fax: (601) 266-6788
-
-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
+-+-
+-+-+-+
----------------------------------------------------------------------- Richard Barrett http://www.openinfo.co.uk
------------------------------------------------------ Mailman-Users mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
This message was sent to: [EMAIL PROTECTED] Unsubscribe or change your options at http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
