Re: [Mailman-Users] BIG discard problem

[Tokio Kikuchi]

Hi,
Did you solve this problem ?
You can prevent so big discard problem again by applying this patch:
http://sourceforge.net/tracker/index.php?func=detail&aid=790494&group_id=103&atid=300103
Cheers,
Robert Echlin wrote:
Hi,
I have a problem discarding pending messages for the admin.
One of our mail lists has accumulated over 10,000 (ten thousand) spam
messages.
--
Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp
http://weather.is.kochi-u.ac.jp/
--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

Re: [Mailman-Users] BIG discard problem

[Stephen J. Turnbull]

> "Robert" == Robert Echlin <[EMAIL PROTECTED]> writes:

Robert> That looks like an interesting solution. I'll make a note
Robert> of it for next time.

Well, last I heard your server was still trying to create the page and
choking.  This will solve that, because there is an index of pending
requests as well as the individual messages.  Removing the messages
causes the server to create a page containing 1 (or whatever it
was) "I can't find this file" items, and it will then remove the
missing message from the index, too.  But it's slo-o-o-ow and takes a
huge amount of RAM (I don't know why---I'll hazard a guess that it's
holding all those messages in memory, probably as Unicode strings to
add insult to injury).

This solution also clears the index, and your server will unjam.
Because it's not futzing with trying to create a web page at the same
time, it should be a matter of a few seconds (it's always been
instantaneous for me, up to about 500 held messages).

Robert> The docs describe how to start "withlist", but I think I
Robert> would have to learn the internals of the Python code
Robert> pretty thoroughly to be able to reproduce this from
Robert> scratch.

True.  However, this particular code gets posted every few weeks.
Consider it a magic spell.

Robert> What's going on in the ...? And what will it prompt me
Robert> with that I need to hit ?

"..." _is_ the prompt.  Python uses indentation to create blocks, not
a begin_token ... end_token sequence.  Changing the prompt tells the
users "you're in a block, be careful to match preceding indentation."
However, here the block is implicit in the statement after the colon,
so you can end it immediately.

Robert> I also think this answers one of my previous questions -
Robert> it would have been useful to ask the developers list.

Not really.  You just happened to have the misfortune that all of the
six or so (now seven ) mailman-users folks who have posted this
recipe (or URLs to it) in the past are either on vacation or misread
your original post to indicate that you tried this and it didn't work
for you.  Most of the developers who would bother to respond to such
an OT question are already active on mailman-users.  (Why is it OT?
Because it's a solved problem, with the solution well-known on the
user list.)

Cheers!

-- 
Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp
University of TsukubaTennodai 1-1-1 Tsukuba 305-8573 JAPAN
   Ask not how you can "do" free software business;
  ask what your business can "do for" free software.
--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

Re: [Mailman-Users] BIG discard problem

[Brad Knowles]

At 10:11 AM -0400 2004-08-12, David Relson wrote:
 I'm running mailman using the Mandrake 10.0 rpms.  I apologize for not
 mentioning that fact.  I assumed that the division of files between
 /usr/local/mailman and /var/lib/mailman was standard.  Evidently I was
 mistaken.
	Anything done by any other group that takes the Mailman software 
and packages it up in some fashion, will almost certainly make fairly 
extensive modifications to the Mailman installation process so that 
it conforms to whatever standards they've chosen regarding the 
separation of source versus executable code, code versus data/queue 
directories, etc

	Unless you (or someone else) installed the software directly from 
the source tarballs provided by the Mailman project, and you didn't 
over-ride any of the default installation procedures, you cannot 
assume that anything on your system is anything remotely like 
anything on anyone else's system.

	This is why I went into pretty extensive detail regarding the 
amount of information asked for in the FAQ entry at 
.

 AFAICT, recursive grep isn't bothered by the number of files in a
 directory.
	Hopefully, it will "Do The Right Thing" with regards to large 
numbers of files to be processed, just like xargs.  But this is 
something that would have to be implemented internally to the code, 
and your version of grep may work one way, while someone else's works 
a different way.

	Contrariwise, xargs works pretty much the same everywhere.  Well, 
everywhere that it is available.

 Anyhow, details aside on grepping source code, I find mailman to be a
 useful and valuable tool.  It's been running bogofilter's mailing lists
 for several months without problem.
 Thanks for the great work.
Thanks!
--
--
Brad Knowles, <[EMAIL PROTECTED]>
"Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755
  SAGE member since 1995.  See  for more info.
--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

Re: [Mailman-Users] BIG discard problem

[David Relson]

On Thu, 12 Aug 2004 10:40:30 +0200
Brad Knowles wrote:

> At 9:44 PM -0400 2004-08-11, David Relson wrote:
> 
> >  No!  Recursive grep commands include a directory specification,
> >  hence don't need to be executed from a specific subdirectory.  My
> >  "egrep -r... /usr/lib/mailman" command searches mailman's source
> >  tree.
> 
>   I have no idea what OS you're using and where the "source" code 
> is placed relative to the data files, etc... on your system, but on 
> Mailman default installations, /usr/local/mailman is the home 
> directory for *everything* related to Mailman, including 
> source/executable code, log files, data files, queue directories, 
> etc
> 
>   If you run a recursive grep of the sort you recommend on the 
> entire /usr/local/mailman directory hierarchy, you are likely to run 
> into the 20,000 file problem previously mentioned.
> 
>   If you want to limit yourself to just the source code, you need 
> to search /usr/local/mailman/Mailman, /usr/local/mailman/bin, etc... 
> and make sure you don't include any of those log directories, queue 
> directories, etc

Brad,

I'm running mailman using the Mandrake 10.0 rpms.  I apologize for not
mentioning that fact.  I assumed that the division of files between
/usr/local/mailman and /var/lib/mailman was standard.  Evidently I was
mistaken.

AFAICT, recursive grep isn't bothered by the number of files in a
directory.  As bogofilter's lead developer, I have a spam archive with
large monthly directories.  I frequently use recursive grep on that
archive.  As a test, I just ran it without incident for a directory with
37,619 files.

Anyhow, details aside on grepping source code, I find mailman to be a
useful and valuable tool.  It's been running bogofilter's mailing lists
for several months without problem.  

Thanks for the great work.

David
--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

Re: [Mailman-Users] BIG discard problem

[Robert Echlin]

- Original Message - 
From: "Stephen J. Turnbull" <[EMAIL PROTECTED]>
To: "Robert Echlin" <[EMAIL PROTECTED]>
Cc: <[EMAIL PROTECTED]>
Sent: Thursday, August 12, 2004 3:37 AM
Subject: Re: [Mailman-Users] BIG discard problem

Thanks, Stephen

That looks like an interesting solution. I'll make a note of it for next
time.

The docs describe how to start "withlist", but I think I would have to learn
the internals of the Python code pretty thoroughly to be able to reproduce
this from scratch.

Is "m" a standard variable automagically defined by withlist to be the mail
list mentioned on the python command line?
Is mm_cfg.DISCARD a function that HandleRequest is expecting to call to do
the real work, or a constant that tells HandleRequest which of several
predefined tasks it should do?
What's going on in the ...? And what will it prompt me with that I need to
hit ?

I also think this answers one of my previous questions - it would have been
useful to ask the developers list. As you suggest, writing a one line loop
of Python code would be the standard solution there.

Is there any other place where this "standard" solution would be found?

Robert

> Didn't you find the standard solution?  To wit:
>
> As user mailman (I think actually you just need write permissions
> in the relevant places, so group mailman Works For Me YMMV) execute:
>
> python -i bin/withlist listname
>
> Then type:
>
> >>> m.Lock()
> >>> from Mailman import mm_cfg
> >>> h = m.GetHeldMessageIds()
> >>> for i in h: m.HandleRequest(i,mm_cfg.DISCARD)
> ... 
> >>> m.Save()
> >>> 
>
> The triple punctuation are Python prompts; you don't type those, you
> wait for them to appear and then type.  All the punctuation (including
> empty parens) must be typed as shown, capitalization must be exactly
> as shown.  The stuff in  are names of single keys
> (Enter and Control + d), not to be typed literally.
>
> On a Red Hat Linux system you probably will need to use python2,
> python2.2 or python2.3 instead of just python, because python
> corresponds to the Golden Oldie 1.5.2 version.


--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

Re: [Mailman-Users] BIG discard problem

[Robert Echlin]

- Original Message - 
From: "Robert Echlin" <[EMAIL PROTECTED]>
To: "David Relson" <[EMAIL PROTECTED]>
Cc: <[EMAIL PROTECTED]>
Sent: Wednesday, August 11, 2004 1:38 PM
Subject: Re: [Mailman-Users] BIG discard problem
> For my next try, I have set the timeout to 1500 seconds, instead of 300,
in
> httpd.conf and restarted apache. Looking at top, I expect that I read it
> incorrectly (see above) when I thought it had run for 14.6 minutes,
because
> it is hovering around 16% of memory, and the timeout in apache would have
> limited it to 300 seconds (4.5 minutes of CPU time at 90%).

Results: it timed out again.
But the memory had kept fluctuating, so I figured it was still working when
it died.
So I upped the timeout again, restarted apache, and tried to load the page
again...

And it came up almost immediately!!!
With only one (1) pending request!!!
So it must have been deleting items from it's records when it found the
heldmsg files were missing, and hadn't got back to the web browser before
the time out.

Anyway, the secret sequence for getting rid of 10914 or more messages in
your pending requests queue seems to be:
- delete mailman/data/heldmsg-yourlistname-*.pck
  - thanks Eric Schmitz
  - you may not be able to delete them all at once
- I used
  rm -f heldmsg-listname-01*.pck
  rm -f heldmsg-listname-02*.pck
  etc up to 15*
- there are other ways including using xargs
- then you set your timeout real high in Apache and open the "tend to
pending moderator requests" link in your web browser
- and wait 25 minutes (1GHZ Celeron CPU under Linux 2.4 kernel).
- and try again

If I were doing this again, I would look at the size and dates of other
files belonging to my list after each iteration of trying the link, as it
must have a list of requests some place. $MAILMAN_HOME/lists//request.pck was over 5M when I started this process. Now it is 20
bytes. That's indicative of something. Also, I looked at its contents at one
time and I think they included the subjects of the messages. But I don't
know what would happen if I blew that file away, at the same time as blowing
away the heldmsg files. That's a useful item to test some time.

Thanks guys,

Robert

--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

Re: [Mailman-Users] BIG discard problem

[Brad Knowles]

At 9:44 PM -0400 2004-08-11, David Relson wrote:
 No!  Recursive grep commands include a directory specification, hence
 don't need to be executed from a specific subdirectory.  My "egrep -r
 ... /usr/lib/mailman" command searches mailman's source tree.
	I have no idea what OS you're using and where the "source" code 
is placed relative to the data files, etc... on your system, but on 
Mailman default installations, /usr/local/mailman is the home 
directory for *everything* related to Mailman, including 
source/executable code, log files, data files, queue directories, 
etc

	If you run a recursive grep of the sort you recommend on the 
entire /usr/local/mailman directory hierarchy, you are likely to run 
into the 20,000 file problem previously mentioned.

	If you want to limit yourself to just the source code, you need 
to search /usr/local/mailman/Mailman, /usr/local/mailman/bin, etc... 
and make sure you don't include any of those log directories, queue 
directories, etc

--
Brad Knowles, <[EMAIL PROTECTED]>
"Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755
  SAGE member since 1995.  See  for more info.
--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

Re: [Mailman-Users] BIG discard problem

[Stephen J. Turnbull]

> "Robert" == Robert Echlin <[EMAIL PROTECTED]> writes:

Robert> Stuff I looked at:
Robert> - 2004 mailman list messages.
Robert> - google for "discard mailman script"
Robert> - mailman faq (search for discard)
Robert> - docs on www.list.org
Robert> - bin/discard.py seems to require some sort of a file as input,
Robert> but there are no docs for it on http://www.list.org/site.html,
Robert> where other command line programs seem to be documented.
Robert> If it's like Skip's mmdiscard.py, 
http://manatee.mojam.com/~skip/python/
Robert> then it needs a file created by looking at the web interface.
Robert> - David Wolfskill's dispose.py seems to require a file input like discard.
Robert> - bulk-vette requires an include file called mailman,
Robert>   which doesn't seem to exist in my version (suggestions??)

Didn't you find the standard solution?  To wit:

As user mailman (I think actually you just need write permissions
in the relevant places, so group mailman Works For Me YMMV) execute:

python -i bin/withlist listname

Then type:

>>> m.Lock()
>>> from Mailman import mm_cfg
>>> h = m.GetHeldMessageIds()
>>> for i in h: m.HandleRequest(i,mm_cfg.DISCARD)
... 
>>> m.Save()
>>> 

The triple punctuation are Python prompts; you don't type those, you
wait for them to appear and then type.  All the punctuation (including
empty parens) must be typed as shown, capitalization must be exactly
as shown.  The stuff in  are names of single keys
(Enter and Control + d), not to be typed literally.

On a Red Hat Linux system you probably will need to use python2,
python2.2 or python2.3 instead of just python, because python
corresponds to the Golden Oldie 1.5.2 version.

Caveat: I haven't actually used this since we installed 2.1.5 (the
previous version was 2.0.13), but a test (on a list with no held
messages---it's the admin's playpen list, I'm not going to do this on
a live list unless I know it's all spam) showed no syntax errors or
anything like that, should still work.

-- 
Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp
University of TsukubaTennodai 1-1-1 Tsukuba 305-8573 JAPAN
   Ask not how you can "do" free software business;
  ask what your business can "do for" free software.
--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

Re: [Mailman-Users] BIG discard problem

[David Relson]

On Thu, 12 Aug 2004 02:03:21 +0200
Brad Knowles wrote:

> At 6:47 PM -0400 2004-08-11, David Relson wrote:
> 
> >  In this case, you're mistaken.  The double quotes work fine for
> >  this purpose (though the '-w' should be removed).
> 
>   If it's in double quotes, the shell will still try to interpret 
> the wildcard.  If you want to protect it from that, you need to put 
> it in single quotes, or put a backslash in front of the asterisk -- 
> even though it's within the double quotes.

Hi Brad,

OK, the quoting behavior is shell dependent.  I use bash and it does
what's needed.  Here's some further examples/evidence:

[EMAIL PROTECTED] relson]$ ls sy.*k
ls: sy.*k: No such file or directory

### 1. show files matching sy*k ###

[EMAIL PROTECTED] relson]$ ls sy*k
sylpheed.log.bak  sylpheedrc.bak

### 2. use same pattern in echo command with double quotes ###

[EMAIL PROTECTED] relson]$ echo "sy*k"
sy*k

--- shell displays pattern, not filenames ---
 
>   The exact behaviour will differ depending on which shell you're 
> using, but all shells I know of, from Bourne shell under BSD Unix 2.9 
> running on a PDP 11/70 in 1984 up through tcsh, bash, ksh, and other 
> more advanced shells running on more modern machines, will all try to 
> interpret what they think is a wildcard unless it is properly escaped 
> -- either through quoting or application of preceding backslash 
> characters.
> 
> >  The egrep command is intended for scanning mailman's source code
> >  with the goal of finding the relevant html (or html generator) in
> >  order to determine the cgi parameters for discarding deferred
> >  items.
> 
>   Then this needs to be done in a specific subdirectory tree which
>   
> would include Python source code and not other general data files, 
> such as $MAILMAN_HOME/Mailman or $MAILMAN_HOME/bin, because otherwise 
> you still run into the possibility of trying to throw 20,000+ files 
> at it.

No!  Recursive grep commands include a directory specification, hence
don't need to be executed from a specific subdirectory.  My "egrep -r
... /usr/lib/mailman" command searches mailman's source tree.

Question:  What shell are you using?  Have you tried executing my
command?

Regards,

David
--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

Re: [Mailman-Users] BIG discard problem

[Brad Knowles]

At 6:47 PM -0400 2004-08-11, David Relson wrote:
 In this case, you're mistaken.  The double quotes work fine for this
 purpose (though the '-w' should be removed).
	If it's in double quotes, the shell will still try to interpret 
the wildcard.  If you want to protect it from that, you need to put 
it in single quotes, or put a backslash in front of the asterisk -- 
even though it's within the double quotes.

	The exact behaviour will differ depending on which shell you're 
using, but all shells I know of, from Bourne shell under BSD Unix 2.9 
running on a PDP 11/70 in 1984 up through tcsh, bash, ksh, and other 
more advanced shells running on more modern machines, will all try to 
interpret what they think is a wildcard unless it is properly escaped 
-- either through quoting or application of preceding backslash 
characters.

 The egrep command is intended for scanning mailman's source code with
 the goal of finding the relevant html (or html generator) in order to
 determine the cgi parameters for discarding deferred items.
	Then this needs to be done in a specific subdirectory tree which 
would include Python source code and not other general data files, 
such as $MAILMAN_HOME/Mailman or $MAILMAN_HOME/bin, because otherwise 
you still run into the possibility of trying to throw 20,000+ files 
at it.

--
Brad Knowles, <[EMAIL PROTECTED]>
"Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755
  SAGE member since 1995.  See  for more info.
--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

Re: [Mailman-Users] BIG discard problem

[David Relson]

On Thu, 12 Aug 2004 00:32:40 +0200
Brad Knowles wrote:

> At 3:21 PM -0400 2004-08-11, David Relson wrote:
> 
> >  Remember "deferred" is the default.  I bet that selecting that
> >  checkbox justs adds a simple key/value pair like "all=yes" to the
> >  cgi request. The origin of the "discard all deferred items" (or
> >  whatever the wording is) can likely be found with:
> >
> > egrep -i -r -w "all.*defer" /usr/lib/mailman/
> 
>   Keep in mind that this will bomb out with 20,000 files in a
>   given 
> directory, just like all other uses of wildcards passed to the shell. 
> You need to escape the wildcard and pass things through a "find ... | 
> xargs" command line, as I demonstrated earlier.
> 
>   If you meant to quote the wildcard and protect it from
>   evaluation 
> by the shell, you need to use single quotes and not double quotes.

Brad,

In this case, you're mistaken.  The double quotes work fine for this
purpose (though the '-w' should be removed). 

The egrep command is intended for scanning mailman's source code with
the goal of finding the relevant html (or html generator) in order to
determine the cgi parameters for discarding deferred items.

David
--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

Re: [Mailman-Users] BIG discard problem

[Brad Knowles]

At 3:21 PM -0400 2004-08-11, David Relson wrote:
 Remember "deferred" is the default.  I bet that selecting that checkbox
 justs adds a simple key/value pair like "all=yes" to the cgi request.
 The origin of the "discard all deferred items" (or whatever the wording
 is) can likely be found with:
egrep -i -r -w "all.*defer" /usr/lib/mailman/
	Keep in mind that this will bomb out with 20,000 files in a given 
directory, just like all other uses of wildcards passed to the shell. 
You need to escape the wildcard and pass things through a "find ... | 
xargs" command line, as I demonstrated earlier.

	If you meant to quote the wildcard and protect it from evaluation 
by the shell, you need to use single quotes and not double quotes.

--
Brad Knowles, <[EMAIL PROTECTED]>
"Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755
  SAGE member since 1995.  See  for more info.
--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

Re: [Mailman-Users] BIG discard problem

[David Relson]

On Wed, 11 Aug 2004 13:38:52 -0400
Robert Echlin wrote:

...[snip]...


> #2 sounds useful, but it may not be. It almost certainly requires that
> you pass a lot of parameters to provide the identities of all the
> requests (some multiple of 11000 or so parameters) that need to be
> discarded. Remember that it says "discard all deferred items". So it
> has to know which items you have left marked as "deferred".

Remember "deferred" is the default.  I bet that selecting that checkbox
justs adds a simple key/value pair like "all=yes" to the cgi request. 
The origin of the "discard all deferred items" (or whatever the wording
is) can likely be found with:

   egrep -i -r -w "all.*defer" /usr/lib/mailman/

or something similar.
--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

Re: [Mailman-Users] BIG discard problem

[Robert Echlin]

- Original Message - 
From: "David Relson" <[EMAIL PROTECTED]>
Cc: <[EMAIL PROTECTED]>
Sent: Wednesday, August 11, 2004 12:10 PM
Subject: Re: [Mailman-Users] BIG discard problem
> Hi Robert,
>
> We're way beyond my knowledge of mailman at this point !  However, I've
> got some alternate ideas from browser usage and web site development.
> Note:  liberally apply grains of salt to these ideas - they're untested.
>
> The 15 minute timeout is likely your browser deciding the web server is
> b0rked.  Very possibly the web server just has too much to do.  The
> browser is running one of mailman's python scripts so it can display the
> generated html.

The python process (owned by Apache) that was taking 200M of RAM was also
taking 95% to 99% of CPU time for 14.6 minutes, according to top, just
before it stopped when the

>
> Idea 1:  Separate the time to run the script from browser display, i.e.
> manually run the python script with appropriate arguments and save the
> output to a file.  Put the file on your web server and use the browser
> to display it.  Click on the appropriate buttons and you're done.

I may be able to use this one.
- tried - failed - file is accessed as
http://server/mailman/admindb/listname
 and the file is /var/mailman/admindb - try to run it as
/var/mailman/admindb/listname and you get an error from bash that admindb is
not a directory.
Apache probably calls it directly, after first setting some stuff up in the
environment variables so it knows what path it was called with. I could try
that later I suppose.

For my next try, I have set the timeout to 1500 seconds, instead of 300, in
httpd.conf and restarted apache. Looking at top, I expect that I read it
incorrectly (see above) when I thought it had run for 14.6 minutes, because
it is hovering around 16% of memory, and the timeout in apache would have
limited it to 300 seconds (4.5 minutes of CPU time at 90%).


> Idea 2: Find the html source code for the "pending requests" page and
> locate the "discard all deferred items" checkbox.  Determine which
> script gets run and the appropriate arguments.  Then run the script from
> the command line.

#2 sounds useful, but it may not be. It almost certainly requires that you
pass a lot of parameters to provide the identities of all the requests (some
multiple of 11000 or so parameters) that need to be discarded. Remember that
it says "discard all deferred items". So it has to know which items you have
left marked as "deferred".

Robert

--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

Re: [Mailman-Users] BIG discard problem

[Brad Knowles]

At 11:50 AM -0400 2004-08-11, Robert Echlin wrote:
 So I deleted the mailman/data/heldmsg-yourlistname-*.pck files - and I had
 to delete them in stages, as the infamous  "Argument list too long" message
 bit me.
In these kinds of cases, what you can usually do is something like this:
# cd $MAILMAN_HOME/data
# find . -name heldmsg-yourlistname\*.pck -print | xargs rm
	And that will do the job for you.  Of course, the "#" prompt 
means to do this as the root user, and you want to make sure you 
include the backslash before the asterisk so that the shell doesn't 
try to interpret the wildcard, and instead passes that to find.  Read 
the man page on xargs to see how it's used.

 And now I tried again and it timed out again after another 15 minutes.
 What next?
As for that, I wish I could be of assistance.
--
Brad Knowles, <[EMAIL PROTECTED]>
"Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755
  SAGE member since 1995.  See  for more info.
--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

Re: [Mailman-Users] BIG discard problem

[David Relson]

On Wed, 11 Aug 2004 11:50:06 -0400
Robert Echlin wrote:

...[snip]...

> The page timed after about 15 minutes, and the python process
> disappeared.
> 
> So I deleted the mailman/data/heldmsg-yourlistname-*.pck files - and I
> had to delete them in stages, as the infamous  "Argument list too
> long" message bit me.
> 
> And now I tried again and it timed out again after another 15 minutes.
> 
> What next?
> 
> Robert

Hi Robert,

We're way beyond my knowledge of mailman at this point !  However, I've
got some alternate ideas from browser usage and web site development. 
Note:  liberally apply grains of salt to these ideas - they're untested.

The 15 minute timeout is likely your browser deciding the web server is
b0rked.  Very possibly the web server just has too much to do.  The
browser is running one of mailman's python scripts so it can display the
generated html.  

Idea 1:  Separate the time to run the script from browser display, i.e.
manually run the python script with appropriate arguments and save the
output to a file.  Put the file on your web server and use the browser
to display it.  Click on the appropriate buttons and you're done.

Idea 2: Find the html source code for the "pending requests" page and
locate the "discard all deferred items" checkbox.  Determine which
script gets run and the appropriate arguments.  Then run the script from
the command line.

Good luck.

David
--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

Re: [Mailman-Users] BIG discard problem

[Eric Schmitz]

Robert Echlin wrote:
The page timed after about 15 minutes, and the python process disappeared.
So I deleted the mailman/data/heldmsg-yourlistname-*.pck files - and I had
to delete them in stages, as the infamous  "Argument list too long" message
bit me.
And now I tried again and it timed out again after another 15 minutes.
What next?
   Beats me. Sounds like the problem is elsewhere, although I still 
would bet that having those 20,000 files out of the way will help with 
that page loading once you do find the actual problem. But I don't know 
the "guts" of Mailman, so no idea what to suggest next. Sorry! I'll be 
very interested to know if someone comes up with a fix.

-Eric Schmitz
--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

Re: [Mailman-Users] BIG discard problem

[Robert Echlin]

- Original Message - 
From: "Robert Echlin" <[EMAIL PROTECTED]>
To: "David Relson" <[EMAIL PROTECTED]>
Cc: <[EMAIL PROTECTED]>
Sent: Wednesday, August 11, 2004 11:01 AM
Subject: Re: [Mailman-Users] BIG discard problem


> - Original Message - 
> From: "David Relson" <[EMAIL PROTECTED]>
> Cc: <[EMAIL PROTECTED]>
> Sent: Tuesday, August 10, 2004 7:38 PM
> Subject: Re: [Mailman-Users] BIG discard problem
>
>
> > On Tue, 10 Aug 2004 18:22:06 -0500
> > Eric Schmitz wrote:
> >
> > > Go to the mailman/data directory, and delete all
> > > heldmsg-yourlistname-*.pck files. Then go to your admindb page. You'll
> > >
> > > probably still see the listing of all the thousands of held messages.
> > > Click "submit all data", and they should disappear.
> > >
> > > *Use at your own risk*! This feels like something of a hack to me,
> > >
> > > but you seem a little desperate. ;-)
> > >
> > > -Eric Schmitz
> >
> > With 2.1.5 there's a "Discard all messages marked Defer" checkbox. I use
> > that, followed by the "Submit All Data" button.  Seems that's all that's
> > needed.
>
> Thanks, David, Eric, and Brad.
>
> My problem is that this page *never* appears. It doesn't seem to time out
> either, at over 20 minutes. The Python process keeps getting bigger until
it
> is using about 200M of RAM, then it fluctuates by 5 or 10M. And while it
is
> running you can't open another window on any admin activity on that list.
> And some people thought I was overspecifiying by getting 1GB or RAM in
that
> server! 
>
> I'm going to start it up again, and see if it finishes after 2 or three
> hours, or times out.


The page timed after about 15 minutes, and the python process disappeared.

So I deleted the mailman/data/heldmsg-yourlistname-*.pck files - and I had
to delete them in stages, as the infamous  "Argument list too long" message
bit me.

And now I tried again and it timed out again after another 15 minutes.

What next?

Robert

--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

Re: [Mailman-Users] BIG discard problem

[Robert Echlin]

- Original Message - 
From: "David Relson" <[EMAIL PROTECTED]>
Cc: <[EMAIL PROTECTED]>
Sent: Tuesday, August 10, 2004 7:38 PM
Subject: Re: [Mailman-Users] BIG discard problem


> On Tue, 10 Aug 2004 18:22:06 -0500
> Eric Schmitz wrote:
>
> > Go to the mailman/data directory, and delete all
> > heldmsg-yourlistname-*.pck files. Then go to your admindb page. You'll
> >
> > probably still see the listing of all the thousands of held messages.
> > Click "submit all data", and they should disappear.
> >
> > *Use at your own risk*! This feels like something of a hack to me,
> >
> > but you seem a little desperate. ;-)
> >
> > -Eric Schmitz
>
> With 2.1.5 there's a "Discard all messages marked Defer" checkbox. I use
> that, followed by the "Submit All Data" button.  Seems that's all that's
> needed.

Thanks, David, Eric, and Brad.

My problem is that this page *never* appears. It doesn't seem to time out
either, at over 20 minutes. The Python process keeps getting bigger until it
is using about 200M of RAM, then it fluctuates by 5 or 10M. And while it is
running you can't open another window on any admin activity on that list.
And some people thought I was overspecifiying by getting 1GB or RAM in that
server! 

I'm going to start it up again, and see if it finishes after 2 or three
hours, or times out.

By the way, this is not a big list, it has 8 subscribers, and a few real
messages a month. The problem with this list is that we have about 2000
pages of documentation on our web site, each with a link to this list. In
some of the spam storms in the last few months the number of messages from
non-subscribers got a bit out of control, so we turned off allowing messages
from non-subscribers. The next version of the docs will have a link to a
form.

Robert

> David
> --
> Mailman-Users mailing list
> [EMAIL PROTECTED]
> http://mail.python.org/mailman/listinfo/mailman-users
> Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
> Searchable Archives:
http://www.mail-archive.com/mailman-users%40python.org/

--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

Re: [Mailman-Users] BIG discard problem

[David Relson]

On Tue, 10 Aug 2004 18:22:06 -0500
Eric Schmitz wrote:

> Robert,
> 
> This is probably a really dirty way to go about it, but I've done 
> this before and it worked, and without hosing the system. This worked 
> with version 2.1.2. I haven't tried it with 2.1.5, because there's a 
> "discard all marked defer" option, and I haven't let the queue stack
> up. But I've killed over 4000 held messages at a shot by doing this
> with 2.1.2.
> 
> Go to the mailman/data directory, and delete all 
> heldmsg-yourlistname-*.pck files. Then go to your admindb page. You'll
> 
> probably still see the listing of all the thousands of held messages. 
> Click "submit all data", and they should disappear.
> 
> *Use at your own risk*! This feels like something of a hack to me,
> 
> but you seem a little desperate. ;-)
> 
> -Eric Schmitz

With 2.1.5 there's a "Discard all messages marked Defer" checkbox. I use
that, followed by the "Submit All Data" button.  Seems that's all that's
needed.

David
--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

Re: [Mailman-Users] BIG discard problem

[Eric Schmitz]

Robert,
   This is probably a really dirty way to go about it, but I've done 
this before and it worked, and without hosing the system. This worked 
with version 2.1.2. I haven't tried it with 2.1.5, because there's a 
"discard all marked defer" option, and I haven't let the queue stack up. 
But I've killed over 4000 held messages at a shot by doing this with 2.1.2.

   Go to the mailman/data directory, and delete all 
heldmsg-yourlistname-*.pck files. Then go to your admindb page. You'll 
probably still see the listing of all the thousands of held messages. 
Click "submit all data", and they should disappear.

   *Use at your own risk*! This feels like something of a hack to me, 
but you seem a little desperate. ;-)

-Eric Schmitz
Robert Echlin wrote:
- Original Message - 
From: "Robert Echlin" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Monday, August 09, 2004 3:31 PM
Subject: [Mailman-Users] BIG discard problem


Hi,
I have a problem discarding pending messages for the admin.
One of our mail lists has accumulated over 10,000 (ten thousand) spam
messages.
The web page to show pending messages, after several minutes, maybe an
hour,
still seems to be churning away, so fix methods based on looking at the
web
page won't work.

... snipped the verbose stuff the FAQs say to include when posting

What I think I want to do:
DELETE the (appropriate, offending) file!!!
Actually, shut down /etc/init.d/mailman, and apache.
I should shut down sendmail, too, I expect.
Delete the file (request.pck???).
touch the filename.
chmod to match what it was.
restart stopped daemons.
Will this work?
Any better ideas?

No suggestions in over 24 hours.
Should I be asking this on a different list, like maybe the dev list?
Robert
--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
--
"We need more than just a victory. We need to win in such a way
that everything changes, that we're not threatened again. Because
I don't want to go through this one more time, or have others go
through it after us." --Starhawk, The Fifth Sacred Thing
)0(  =.
--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

Re: [Mailman-Users] BIG discard problem

[Brad Knowles]

At 6:48 PM -0400 2004-08-10, Robert Echlin wrote:
 No suggestions in over 24 hours.
	Problem is, it can take a while to get a response from someone 
who knows what they're doing in this kind of situation.  While 
Mailman is used at a large number of sites, and some of those are 
pretty big, not all those experienced Mailman admins are on this 
list, and those that are may not monitor it on a regular basis.

 Should I be asking this on a different list, like maybe the dev list?
	That might be useful, or you might end up waiting a few days for 
a useful response from someone, no matter what.  I'd say that you 
could give it a try if you like, but don't be too surprised if it 
takes a while to get an answer.

	I hadn't responded until now because you had obviously done your 
homework, and I didn't think I had anything more that I could say.

--
Brad Knowles, <[EMAIL PROTECTED]>
"Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755
  SAGE member since 1995.  See  for more info.
--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

Re: [Mailman-Users] BIG discard problem

[Robert Echlin]

- Original Message - 
From: "Robert Echlin" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Monday, August 09, 2004 3:31 PM
Subject: [Mailman-Users] BIG discard problem


> Hi,
> I have a problem discarding pending messages for the admin.
>
> One of our mail lists has accumulated over 10,000 (ten thousand) spam
> messages.
> The web page to show pending messages, after several minutes, maybe an
hour,
> still seems to be churning away, so fix methods based on looking at the
web
> page won't work.

... snipped the verbose stuff the FAQs say to include when posting

> What I think I want to do:
> DELETE the (appropriate, offending) file!!!
>
> Actually, shut down /etc/init.d/mailman, and apache.
> I should shut down sendmail, too, I expect.
> Delete the file (request.pck???).
> touch the filename.
> chmod to match what it was.
> restart stopped daemons.
>
> Will this work?
> Any better ideas?

No suggestions in over 24 hours.
Should I be asking this on a different list, like maybe the dev list?

Robert

--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

[Mailman-Users] BIG discard problem

[Robert Echlin]

Hi,
I have a problem discarding pending messages for the admin.

One of our mail lists has accumulated over 10,000 (ten thousand) spam
messages.
The web page to show pending messages, after several minutes, maybe an hour,
still seems to be churning away, so fix methods based on looking at the web
page won't work.
There is a python process owned by Apache churning away using up 200M of
memory. Stopping /etc/init.d/mailman and closing my browser eventually kills
it.

Info:
  lists//request.pck is over 5M.
  lists//pending.pck is only 1263 bytes.

Stuff I looked at:
- 2004 mailman list messages.
- google for "discard mailman script"
- mailman faq (search for discard)
- docs on www.list.org
- bin/discard.py seems to require some sort of a file as input,
but there are no docs for it on http://www.list.org/site.html,
where other command line programs seem to be documented.
If it's like Skip's mmdiscard.py, http://manatee.mojam.com/~skip/python/
then it needs a file created by looking at the web interface.
- David Wolfskill's dispose.py seems to require a file input like discard.
- bulk-vette requires an include file called mailman,
  which doesn't seem to exist in my version (suggestions??)

Config:
Mailman 2.1.5 on RH Linux 9

What I think I wanna do:
DELETE the (appropriate, offending) file!!!

Actually, shut down /etc/init.d/mailman, and apache.
I should shut down sendmail, too, I expect.
Delete the file (request.pck???).
touch the filename.
chmod to match what it was.
restart stopped daemons.

Will this work?
Any better ideas?

Robert

--
Robert Echlin
Contractor - Infrastructure Specialist
[EMAIL PROTECTED]
Voice: (613) 745-4242 extension 232
Fax: (613) 745-5560

Stilo Corporation
1900 City Park Drive, Suite 504
Ottawa, Ontario, CanadaK1J 1A3
http://www.stilo.com

This message, including any attachments, is for the sole use of the
intended recipient(s) and may contain confidential and privileged
information. Any unauthorized review, use, disclosure, copying, or
distribution is strictly prohibited. If you are not the intended
recipient(s) please contact the sender by reply email and destroy
all copies of the original message and any attachments.

--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/