Re: [Mailman-Users] Python process size grows 30x in 8 hours (memory
I did a test - I disabled the SpamAssassin integration and watched the heap grow steadily - I do not believe its SA related: [EMAIL PROTECTED]:mailman-2.1.9 10:51pm 68 # pmap 22804 | egrep heap 08175000 14060K rwx--[ heap ] [EMAIL PROTECTED]:mailman-2.1.9 10:51pm 69 # pmap 22804 | egrep heap 08175000 16620K rwx--[ heap ] [EMAIL PROTECTED]:mailman-2.1.9 10:52pm 70 # pmap 22804 | egrep heap 08175000 16620K rwx--[ heap ] [EMAIL PROTECTED]:mailman-2.1.9 10:53pm 75 # pmap 22804 | egrep heap 08175000 18924K rwx--[ heap ] [EMAIL PROTECTED]:mailman-2.1.9 10:54pm 81 # pmap 22804 | egrep heap 08175000 19692K rwx--[ heap ] [EMAIL PROTECTED]:mailman-2.1.9 10:55pm 82 # pmap 22804 | egrep heap 08175000 19692K rwx--[ heap ] Trying to find a way to look at the contents of the heap or at least limit its growth. Or is there not a way expire restart mailman processes analogous to the apache httpd process expiration (designed to mitigate this kind of resource growth over time)? thanks On 7/1/08 9:58 PM, Brad Knowles [EMAIL PROTECTED] wrote: On 7/1/08, Mark Sapiro wrote: In this snapshot PID USERNAME LWP PRI NICE SIZE RES STATETIMECPU COMMAND 10123 mailman1 590 314M 311M sleep1:57 0.02% python 10131 mailman1 590 310M 307M sleep1:35 0.01% python 10124 mailman1 590 309M 78M sleep0:45 0.10% python 10134 mailman1 590 307M 81M sleep1:27 0.01% python 10125 mailman1 590 307M 79M sleep0:42 0.01% python 10133 mailman1 590 44M 41M sleep0:14 0.01% python 10122 mailman1 590 34M 30M sleep0:43 0.39% python 10127 mailman1 590 31M 27M sleep0:40 0.26% python 10130 mailman1 590 30M 26M sleep0:15 0.03% python 10129 mailman1 590 28M 24M sleep0:19 0.10% python 10126 mailman1 590 28M 25M sleep1:07 0.59% python 10132 mailman1 590 27M 24M sleep1:00 0.46% python 10128 mailman1 590 27M 24M sleep0:16 0.01% python 10151 mailman1 590 9516K 3852K sleep0:05 0.01% python 10150 mailman1 590 9500K 3764K sleep0:00 0.00% python Which processes correspond to which runners. And why are the two processes that have apparently done the least the ones that have grown the most. In contrast, the mail server for python.org shows the following: top - 06:54:48 up 29 days, 9:09, 4 users, load average: 1.05, 1.08, 0.95 Tasks: 151 total, 1 running, 149 sleeping, 0 stopped, 1 zombie Cpu(s): 0.2% user, 1.1% system, 0.0% nice, 98.7% idle PID USER PR VIRT NI RES SHR S %CPUTIME+ %MEM COMMAND 1040 mailman9 42960 0 41m 12m S0 693:59.44 2.1 ArchRunner:0:1 -s 1041 mailman9 22876 0 20m 7488 S0 478:18.62 1.0 BounceRunner:0:1 1045 mailman9 20412 0 19m 10m S0 3031:12 0.9 OutgoingRunner:0: 1043 mailman9 20476 0 18m 4968 S0 127:02.62 0.9 IncomingRunner:0: 1042 mailman9 18564 0 17m 7316 S0 11:34.14 0.9 CommandRunner:0:1 1046 mailman 11 17276 0 15m 10m S1 66:32.16 0.8 VirginRunner:0:1 1044 mailman9 11568 0 9964 5184 S0 12:34.04 0.5 NewsRunner:0:1 -s And those are the only Python-related processes that show up in the first twenty lines. -- Fletcher Cocquyt Senior Systems Administrator Information Resources and Technology (IRT) Stanford University School of Medicine Email: [EMAIL PROTECTED] Phone: (650) 724-7485 -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Python process size grows 30x in 8 hours (memory leak?)
Back at the beginning of this thread, Fletcher Cocquyt wrote: Config: Solaris 10 x86 Python 2.5.2 Mailman 2.1.9 (8 Incoming queue runners - the leak rate increases with this) SpamAssassin 3.2.5 At this point I am looking for ways to isolate the suspected memory leak - I am looking at using dtrace: http://blogs.sun.com/sanjeevb/date/200506 Any other tips appreciated! With Solaris 10, you can interpose the libumem library when starting those python processes. This gives you different malloc()/free() allocators including extra instrumentation that is low enough in overhead to run in a production environment, and (when combined with mdb) a powerful set of debugging tools. Set LD_PRELOAD, UMEM_DEBUG, UMEM_LOGGING environment variables in the parent process before starting python so they will inherit the settings. If you have to, you could replace 'python' with a script that sets what you want in the environment and then runs the python executable. I know this will be looking at the lower, native layers of the problem, and you may not see the upper (python) part of the stack very well, but libumem has been a big help to me so I thought I would mention it. Here are two references... there are many more if you start searching: Identifying Memory Management Bugs Within Applications Using the libumem Library http://access1.sun.com/techarticles/libumem.html Solaris Modular Debugger Guide http://docs.sun.com/db/doc/806-6545 Hope this helps - this is too long, so I'll stop now. Tim -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Unsubscribing
On Die, 2008-07-01 at 16:39 +1000, Doug Laidlaw wrote: I am unsubscribing. Thanks very much for helping with my inquiry, Mark. The other queries don't apply to me, and I can offer no help with them. On my inquiry, one would think that an OS that wants to rule the world would be compatible with the most popular mailing list software. Oh, well... You have to complain about this (Mailman should run on Windows) to MSFT. But I don't think that is what you meant (or need). Bernd -- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
[Mailman-Users] Mailman Auto Install
Hi, I bought a dedicated server to host my list with .. but for my surprise, mailman did not exist in its control panel which is webmin .. in cpanel u can just click a button and it will do everything but not in webmin .. I tried to follow the installation instructions from the site (list.org) but it's like reading chineese!! I couldn't understand a thing .. is there like a script to automatically install mailman like in forums ? u just run a single command and it automatically extracts the mailman files (which I didn't know how to do that in the first place) and asks you a few questions to configure mailman??? please help .. Thanks.. _ Need to know now? Get instant answers with Windows Live Messenger. http://www.windowslive.com/messenger/connect_your_way.html?ocid=TXT_TAGLM_WL_messenger_072008 -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] How can I check status of or abort a mailing inprogress?
John Hicks wrote: Heady from this success, I re-reconfigured the list to enable full personalization. My client posted his first message using this configuration Sunday night and it is still being processed (Tuesday night). The smtp log is filling with entries indicating either one or three recipients were processed. [I understand the reason for the singletons but not the triplets.] These entries occur in batches of a hundred or so and then pause for an hour or two. I have no idea how far the system has gotten in processing the entire list. Is there any way to find out? (The message already appears in the archives but there is no entry in the Mailman post log.) My client wants me to cancel the post, but I can't find any indication of how one would go about doing that. Nothing in the documentation, wiki, FAQ, and only one post on the subject on this list (but without an answer to the question). The only difference between personalize = Yes, and personalize = Full is the contents of the To: header in the message. Normally, this would have no effect whatsoever on the SMTP interaction between Mailman and the MTA. Also, with respect to logging, I don't understand the singletons or triplets without more information. If there are no failures (smtp-failure log), even with Full personalization, the SMTPDirect handler writes one smtp log entry and the post log entry at the end after everything is delivered to the MTA. Brad posted how to cancel if you can - stop Mailman. Look at the entries in the 'out' and 'retry' queues with bin/dumpdb or bin/show_qfiles and remove the ones for this message. There will be at most one in 'out'. I'm not sure about retry. For anything that left Mailman , you have to look to the MTA. What are the message-ids in the smtp log entries? What's in the smtp-failure log.? -- Mark Sapiro [EMAIL PROTECTED]The highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Python process size grows 30x in 8 hours (memory
Fletcher Cocquyt wrote: I did a test - I disabled the SpamAssassin integration and watched the heap grow steadily - I do not believe its SA related: OK. Does your MTA limit the size of incoming messages? Can it? At some point in the next day or so, I'm going to make a modified scripts/post script which will queue incoming messages in qfiles/bad and then move them to qfiles/in only if they are under a certain size. I'm really curious to see if that will help. Trying to find a way to look at the contents of the heap or at least limit its growth. Or is there not a way expire restart mailman processes analogous to the apache httpd process expiration (designed to mitigate this kind of resource growth over time)? bin/mailmanctl could be modified to do this automatically, but currently only does it on command (restart) or signal (SIGINT), but I gather you're already running a cron that does a periodic restart. -- Mark Sapiro [EMAIL PROTECTED]The highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Mailman Auto Install
Khalil Abbas wrote: Hi, I bought a dedicated server to host my list with .. but for my surprise, mailman did not exist in its control panel which is webmin .. in cpanel u can just click a button and it will do everything but not in webmin .. I tried to follow the installation instructions from the site (list.org) but it's like reading chineese!! I couldn't understand a thing .. is there like a script to automatically install mailman like in forums ? u just run a single command and it automatically extracts the mailman files (which I didn't know how to do that in the first place) and asks you a few questions to configure mailman??? please help .. End original message. - Installing mailman from source assumes that you have some understanding of how to do a fair number of things from the command line interface. Having said that, I think the documentation is not too complicated and if you take your time you should be able to follow it as long as the other pieces you need are installed (like Apache web server) and you have root access to do certain things to configure the system. Some OS distributions have pre-packaged versions of mailman that can be installed with a package manager like yum but the problem I see with these is that they are often reorganized to place components of the system in different places that the OS vendor thinks are appropriate. This can make troubleshooting problems a bit more difficult. In my opinion, I'd say that in the long run you will be better off learning how to administer the system from the command line so you don't have to rely on control panels. Knowing this makes it far easier to get the system working the way you want and lets you more effectively fix problems. Dragon ~~~ Venimus, Saltavimus, Bibimus (et naribus canium capti sumus) ~~~ -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
[Mailman-Users] Mailman group on LinkedIn
Folks: On a lark, I created a Mailman group on LinkedIn. If you would like to join the group, visit http://www.linkedin.com/e/gis/134598/181E8C14C43E. david -- IBM i on Power Systems - For when you can't afford to be out of business -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Python process size grows 30x in 8 hours (memory
On 7/2/08 8:15 AM, Mark Sapiro [EMAIL PROTECTED] wrote: Fletcher Cocquyt wrote: I did a test - I disabled the SpamAssassin integration and watched the heap grow steadily - I do not believe its SA related: OK. Does your MTA limit the size of incoming messages? Can it? No, Yes # maximum message size #O MaxMessageSize=0 At some point in the next day or so, I'm going to make a modified scripts/post script which will queue incoming messages in qfiles/bad and then move them to qfiles/in only if they are under a certain size. I'm really curious to see if that will help. Yes, having a global incoming maxmessagesize limit and handler (what will the sender receive back?) for mailman would be useful. Trying to find a way to look at the contents of the heap or at least limit its growth. Or is there not a way expire restart mailman processes analogous to the apache httpd process expiration (designed to mitigate this kind of resource growth over time)? bin/mailmanctl could be modified to do this automatically, but currently only does it on command (restart) or signal (SIGINT), but I gather you're already running a cron that does a periodic restart. -- Fletcher Cocquyt Senior Systems Administrator Information Resources and Technology (IRT) Stanford University School of Medicine Email: [EMAIL PROTECTED] Phone: (650) 724-7485 -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Mailman group on LinkedIn
David Gibbs said the following on 02/07/2008 17.21: If you would like to join the group, visit http://www.linkedin.com/e/gis/134598/181E8C14C43E. Good idea! I just joined the group. Ciao, luigi -- / +--[Luigi Rosa]-- \ Hackers do it with all sorts of characters. -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Python process size grows 30x in 8 hours (memory
Last night I added ulimit -v 5 To the /etc/init.d/mailman script and restarted And I am seeing the processes stop growing at 49M - and so far no adverse affects. I view this as a workaround until the underlying cause is determined... But it also allows me to bump my incoming runners from 8 to 16 and manage/enforce the overall memory footprint - I like it (so far) last pid: 8371; load averages: 0.07, 0.11, 0.16 08:57:41 91 processes: 90 sleeping, 1 on cpu CPU states: 98.2% idle, 0.5% user, 1.3% kernel, 0.0% iowait, 0.0% swap Memory: 1640M real, 792M free, 697M swap in use, 2602M swap free PID USERNAME LWP PRI NICE SIZE RES STATETIMECPU COMMAND 7565 mailman1 590 49M 46M sleep0:07 0.01% python 7571 mailman1 590 35M 31M sleep0:04 0.01% python 7551 mailman1 590 35M 32M sleep0:18 0.02% python 7554 mailman1 590 31M 27M sleep0:22 0.02% python 7568 mailman1 590 31M 28M sleep0:03 0.01% python 7574 mailman1 590 30M 28M sleep0:03 0.01% python 7570 mailman1 590 30M 27M sleep0:14 0.01% python 7560 mailman1 590 30M 27M sleep0:12 0.15% python 7562 mailman1 590 27M 25M sleep0:03 0.02% python 7566 mailman1 590 27M 25M sleep0:03 0.01% python 7569 mailman1 590 27M 25M sleep0:04 0.01% python 7561 mailman1 590 27M 24M sleep0:03 0.01% python 7558 mailman1 590 26M 24M sleep0:19 0.02% python 7572 mailman1 590 26M 23M sleep0:03 0.01% python 7567 mailman1 590 26M 23M sleep0:03 0.01% python On 7/1/08 11:43 PM, Fletcher Cocquyt [EMAIL PROTECTED] wrote: Hey thanks, I really appreciated the responsiveness, and helpful analysis by the folks on this list BTW I am experimenting with ulimit -v to limit the growth of mailman procs (added it to the init script) Will see what the growth looks like overnight and report back - thanks On 7/1/08 10:21 PM, Brad Knowles [EMAIL PROTECTED] wrote: On 7/1/08, Fletcher Cocquyt wrote: Fletcher Cocquyt Senior Systems Administrator Information Resources and Technology (IRT) Stanford University School of Medicine BTW, in case it hasn't come through yet -- I am very sensitive to your issues. In my real life, I am currently employed as a Sr. System Administrator at the University of Texas at Austin, with about ~50,000 students and ~20,000 faculty and staff, and one of my jobs is helping out with both the mail system administration and the mailing list system administration. So, just because I post messages quoting the current statistics we're seeing on python.org, that doesn't mean I'm not sensitive to the problems you're seeing. All I'm saying is that we're not currently seeing them on python.org, so it may be a bit more difficult for us to directly answer your questions, although we'll certainly do everything we can do help. -- Fletcher Cocquyt Senior Systems Administrator Information Resources and Technology (IRT) Stanford University School of Medicine Email: [EMAIL PROTECTED] Phone: (650) 724-7485 -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Python process size grows 30x in 8 hours (memory
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Jul 2, 2008, at 11:15 AM, Mark Sapiro wrote: At some point in the next day or so, I'm going to make a modified scripts/post script which will queue incoming messages in qfiles/bad and then move them to qfiles/in only if they are under a certain size. I'm really curious to see if that will help. This should be moved to mailman-developers, but in general it's an interesting idea. In MM3 I've split the incoming queue into two separate queues. The incoming queue now solely determines the disposition of the message, i.e. held, rejected, discarded or accepted. If accepted, the message is moved to a pipeline queue where it's munged for delivery (i.e. headers and footers added, etc.). MM3 also has an LMTP queue runner, which I'd like to make the default delivery mechanism for 3.0 and possibly 2.2 (yes, I still have a todo to back port MM3's new process architecture to 2.2). Although it's not there right now, it would be trivial to add a check on the raw size of the message before it's parsed. If it's too large then it can be rejected before the email package attempts to parse it, and that would give the upstream LTMP client (i.e. your MTA) a better diagnostic. It still makes sense to put a size limit in your MTA so it never hits the LMTP server because the string will still be in the Python process's memory. But at least you won't pay the penalty for parsing such a huge message just to reject it later. Trying to find a way to look at the contents of the heap or at least limit its growth. Or is there not a way expire restart mailman processes analogous to the apache httpd process expiration (designed to mitigate this kind of resource growth over time)? bin/mailmanctl could be modified to do this automatically, but currently only does it on command (restart) or signal (SIGINT), but I gather you're already running a cron that does a periodic restart. This is a good idea. It might be better to do this in Runner._doperiodic(). - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iEYEARECAAYFAkhrpt4ACgkQ2YZpQepbvXHAtgCgj0E1IJjf4kkv4TNKkzcB+RFF VxAAn1k01dLfPeKPcOgMxDneSyEB/5Ro =qiO5 -END PGP SIGNATURE- -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Python process size grows 30x in 8 hours (memory
Fletcher Cocquyt wrote: Or is there not a way expire restart mailman processes analogous to the apache httpd process expiration (designed to mitigate this kind of resource growth over time)? You can do mailmanctl restart, but that's not really a proper solution to this problem. -- Brad Knowles [EMAIL PROTECTED] LinkedIn Profile: http://tinyurl.com/y8kpxu -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Apache and pipermail 403 issues
Currently when ever I try to get to the mailman archives for any of the mailing lists I revcive a 403 error. When I check the apache error log i see the following line: [Wed Jul 02 10:28:52 2008] [error] [client some ip] client denied by server configuration: var-install-location/archives/public/ The Private folder is owned by what apache runs under. Also the apache config file has Alias /pipermail/ var-install-location/archives/public/ Directory var-install-location AllowOverride All Options +FollowSymLinks AddDefaultCharset Off /Directory Directory install-prefix AllowOverride All Options +FollowSymLinks AddDefaultCharset Off /Directory ScriptAlias /mailman/ prefix/cgi-bin/ Directory prefix/cgi-bin AllowOverride All Order allow,deny Allow from all /Directory Apache version: 2.2.6 Mailman version: 1.1.11rc1 If there are any ideas on how to fix this please let me know. Thanks -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Python process size grows 30x in 8 hours (memory
I am hopeful our esteemed code maintainers are thinking the built in restart idea is a good one: BW wrote: Or is there not a way expire restart mailman processes analogous to the apache httpd process expiration (designed to mitigate this kind of resource growth over time)? bin/mailmanctl could be modified to do this automatically, but currently only does it on command (restart) or signal (SIGINT), but I gather you're already running a cron that does a periodic restart. This is a good idea. It might be better to do this in Runner._doperiodic(). On 7/2/08 9:22 AM, Brad Knowles [EMAIL PROTECTED] wrote: Fletcher Cocquyt wrote: Or is there not a way expire restart mailman processes analogous to the apache httpd process expiration (designed to mitigate this kind of resource growth over time)? You can do mailmanctl restart, but that's not really a proper solution to this problem. -- Fletcher Cocquyt Senior Systems Administrator Information Resources and Technology (IRT) Stanford University School of Medicine Email: [EMAIL PROTECTED] Phone: (650) 724-7485 -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Python process size grows 30x in 8 hours (memory
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Jul 2, 2008, at 1:12 PM, Fletcher Cocquyt wrote: I am hopeful our esteemed code maintainers are thinking the built in restart idea is a good one: Optionally, yes. By default, I'm not so sure. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iEYEARECAAYFAkhrt4UACgkQ2YZpQepbvXE9kACeLg04R4n22C4X3VInoJaaCqyI MdkAoJjgj0qwONIKM425QHh/Glxpo4gm =yOaG -END PGP SIGNATURE- -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Python process size grows 30x in 8 hours (memory
Interesting, the growth per python is limited to 50M by ulimit -v 5, but I'm seeing each one gradually take up that limit - then what ? (stay tuned! - mailman fails to malloc?) load averages: 0.14, 0.11, 0.11 10:14:43 120 processes: 119 sleeping, 1 on cpu CPU states: 93.9% idle, 3.7% user, 2.4% kernel, 0.0% iowait, 0.0% swap Memory: 1640M real, 646M free, 858M swap in use, 2436M swap free PID USERNAME LWP PRI NICE SIZE RES STATETIMECPU COMMAND 7566 mailman1 590 49M 46M sleep0:08 0.01% python 7565 mailman1 590 49M 46M sleep0:09 0.02% python 7557 mailman1 590 48M 46M sleep0:08 0.01% python 7569 mailman1 590 48M 45M sleep0:09 0.01% python 7552 mailman1 590 47M 45M sleep0:08 0.01% python 7561 mailman1 590 47M 45M sleep0:08 0.06% python 7570 mailman1 590 47M 44M sleep0:20 0.01% python 7571 mailman1 590 35M 31M sleep0:05 0.01% python 7551 mailman1 590 35M 32M sleep0:31 0.16% python 7554 mailman1 590 31M 27M sleep0:37 0.35% python 7568 mailman1 590 31M 28M sleep0:05 0.01% python 7574 mailman1 590 30M 28M sleep0:05 0.37% python 7560 mailman1 590 30M 27M sleep0:20 0.02% python 7562 mailman1 590 27M 25M sleep0:06 0.03% python 7558 mailman1 590 26M 24M sleep0:33 0.35% python On 7/2/08 9:01 AM, Fletcher Cocquyt [EMAIL PROTECTED] wrote: Last night I added ulimit -v 5 To the /etc/init.d/mailman script and restarted And I am seeing the processes stop growing at 49M - and so far no adverse affects. I view this as a workaround until the underlying cause is determined... But it also allows me to bump my incoming runners from 8 to 16 and manage/enforce the overall memory footprint - I like it (so far) last pid: 8371; load averages: 0.07, 0.11, 0.16 08:57:41 91 processes: 90 sleeping, 1 on cpu CPU states: 98.2% idle, 0.5% user, 1.3% kernel, 0.0% iowait, 0.0% swap Memory: 1640M real, 792M free, 697M swap in use, 2602M swap free PID USERNAME LWP PRI NICE SIZE RES STATETIMECPU COMMAND 7565 mailman1 590 49M 46M sleep0:07 0.01% python 7571 mailman1 590 35M 31M sleep0:04 0.01% python 7551 mailman1 590 35M 32M sleep0:18 0.02% python 7554 mailman1 590 31M 27M sleep0:22 0.02% python 7568 mailman1 590 31M 28M sleep0:03 0.01% python 7574 mailman1 590 30M 28M sleep0:03 0.01% python 7570 mailman1 590 30M 27M sleep0:14 0.01% python 7560 mailman1 590 30M 27M sleep0:12 0.15% python 7562 mailman1 590 27M 25M sleep0:03 0.02% python 7566 mailman1 590 27M 25M sleep0:03 0.01% python 7569 mailman1 590 27M 25M sleep0:04 0.01% python 7561 mailman1 590 27M 24M sleep0:03 0.01% python 7558 mailman1 590 26M 24M sleep0:19 0.02% python 7572 mailman1 590 26M 23M sleep0:03 0.01% python 7567 mailman1 590 26M 23M sleep0:03 0.01% python On 7/1/08 11:43 PM, Fletcher Cocquyt [EMAIL PROTECTED] wrote: Hey thanks, I really appreciated the responsiveness, and helpful analysis by the folks on this list BTW I am experimenting with ulimit -v to limit the growth of mailman procs (added it to the init script) Will see what the growth looks like overnight and report back - thanks On 7/1/08 10:21 PM, Brad Knowles [EMAIL PROTECTED] wrote: On 7/1/08, Fletcher Cocquyt wrote: Fletcher Cocquyt Senior Systems Administrator Information Resources and Technology (IRT) Stanford University School of Medicine BTW, in case it hasn't come through yet -- I am very sensitive to your issues. In my real life, I am currently employed as a Sr. System Administrator at the University of Texas at Austin, with about ~50,000 students and ~20,000 faculty and staff, and one of my jobs is helping out with both the mail system administration and the mailing list system administration. So, just because I post messages quoting the current statistics we're seeing on python.org, that doesn't mean I'm not sensitive to the problems you're seeing. All I'm saying is that we're not currently seeing them on python.org, so it may be a bit more difficult for us to directly answer your questions, although we'll certainly do everything we can do help. -- Fletcher Cocquyt Senior Systems Administrator Information Resources and Technology (IRT) Stanford University School of Medicine Email: [EMAIL PROTECTED] Phone: (650) 724-7485 -- Mailman-Users mailing list Mailman-Users@python.org
Re: [Mailman-Users] Python process size grows 30x in 8 hours - dtrace stack
I had a parallel thread on the dtrace list to get memleak.d running http://blogs.sun.com/sanjeevb/date/200506 - I just got this stack trace from a 10 second sample of the most actively growing python mailman process - the output is explained by Sanjeev on his blog, but I'm hoping the stack trace will point the analysis towards a cause for why my mailman processes are growing abnormally I will see if the findleaks.pl analysis of this output returns anything Thanks! 0 42246 realloc:return Ptr=0x824c268 Oldptr=0x0 Size=16 libc.so.1`realloc+0x33a python`addcleanup+0x45 python`convertsimple+0x145d python`vgetargs1+0x259 python`_PyArg_ParseTuple_SizeT+0x1d python`posix_listdir+0x55 python`PyEval_EvalFrameEx+0x59ff python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalFrameEx+0x49ff python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalCode+0x22 python`PyRun_FileExFlags+0xaf python`PyRun_SimpleFileExFlags+0x156 python`Py_Main+0xa6b python`main+0x17 python`_start+0x80 0 42249 free:entry Ptr=0x824c268 0 42244 lmalloc:return Ptr=0xcf890300 Size=16 libc.so.1`lmalloc+0x143 libc.so.1`opendir+0x3e python`posix_listdir+0x6d python`PyEval_EvalFrameEx+0x59ff python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalFrameEx+0x49ff python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalCode+0x22 python`PyRun_FileExFlags+0xaf python`PyRun_SimpleFileExFlags+0x156 python`Py_Main+0xa6b python`main+0x17 python`_start+0x80 0 42244 lmalloc:return Ptr=0xcf894000 Size=8192 libc.so.1`lmalloc+0x143 libc.so.1`opendir+0x3e python`posix_listdir+0x6d python`PyEval_EvalFrameEx+0x59ff python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalFrameEx+0x49ff python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalCode+0x22 python`PyRun_FileExFlags+0xaf python`PyRun_SimpleFileExFlags+0x156 python`Py_Main+0xa6b python`main+0x17 python`_start+0x80 0 42249 free:entry Ptr=0x86d78f0 ^C 0 42246 realloc:return Ptr=0x824c268 Oldptr=0x0 Size=16 libc.so.1`realloc+0x33a python`addcleanup+0x45 python`convertsimple+0x145d python`vgetargs1+0x259 python`_PyArg_ParseTuple_SizeT+0x1d python`posix_listdir+0x55 python`PyEval_EvalFrameEx+0x59ff python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalFrameEx+0x49ff python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalCode+0x22 python`PyRun_FileExFlags+0xaf python`PyRun_SimpleFileExFlags+0x156 python`Py_Main+0xa6b python`main+0x17 python`_start+0x80 0 42249 free:entry Ptr=0x824c268 0 42244 lmalloc:return Ptr=0xcf890300 Size=16 libc.so.1`lmalloc+0x143 libc.so.1`opendir+0x3e python`posix_listdir+0x6d python`PyEval_EvalFrameEx+0x59ff python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalFrameEx+0x49ff python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalCode+0x22 python`PyRun_FileExFlags+0xaf python`PyRun_SimpleFileExFlags+0x156 python`Py_Main+0xa6b python`main+0x17 python`_start+0x80 0 42244 lmalloc:return Ptr=0xcf894000 Size=8192 libc.so.1`lmalloc+0x143 libc.so.1`opendir+0x3e python`posix_listdir+0x6d python`PyEval_EvalFrameEx+0x59ff python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalFrameEx+0x49ff
Re: [Mailman-Users] Python process size grows 30x in 8 hours - dtrace stack
Below is the findleaks output from a ~5minute sample of a python runner - I will take a larger sample to see if this is representative or not: (again the reference is http://blogs.sun.com/sanjeevb/date/200506 ) Thanks [EMAIL PROTECTED]:~ 2:10pm 67 # ./findleaks.pl ./ml.out Ptr=0xcf890340 Size=16 libc.so.1`lmalloc+0x143 libc.so.1`opendir+0x3e python`posix_listdir+0x6d python`PyEval_EvalFrameEx+0x59ff python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalFrameEx+0x49ff python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalCode+0x22 python`PyRun_FileExFlags+0xaf python`PyRun_SimpleFileExFlags+0x156 python`Py_Main+0xa6b python`main+0x17 python`_start+0x80 - Ptr=0xcf894000 Size=8192 libc.so.1`lmalloc+0x143 libc.so.1`opendir+0x3e python`posix_listdir+0x6d python`PyEval_EvalFrameEx+0x59ff python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalFrameEx+0x49ff python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalCode+0x22 python`PyRun_FileExFlags+0xaf python`PyRun_SimpleFileExFlags+0x156 python`Py_Main+0xa6b python`main+0x17 python`_start+0x80 - On 7/2/08 1:54 PM, Fletcher Cocquyt [EMAIL PROTECTED] wrote: I had a parallel thread on the dtrace list to get memleak.d running http://blogs.sun.com/sanjeevb/date/200506 - I just got this stack trace from a 10 second sample of the most actively growing python mailman process - the output is explained by Sanjeev on his blog, but I'm hoping the stack trace will point the analysis towards a cause for why my mailman processes are growing abnormally I will see if the findleaks.pl analysis of this output returns anything Thanks! 0 42246 realloc:return Ptr=0x824c268 Oldptr=0x0 Size=16 libc.so.1`realloc+0x33a python`addcleanup+0x45 python`convertsimple+0x145d python`vgetargs1+0x259 python`_PyArg_ParseTuple_SizeT+0x1d python`posix_listdir+0x55 python`PyEval_EvalFrameEx+0x59ff python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalFrameEx+0x49ff python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalCode+0x22 python`PyRun_FileExFlags+0xaf python`PyRun_SimpleFileExFlags+0x156 python`Py_Main+0xa6b python`main+0x17 python`_start+0x80 0 42249 free:entry Ptr=0x824c268 0 42244 lmalloc:return Ptr=0xcf890300 Size=16 libc.so.1`lmalloc+0x143 libc.so.1`opendir+0x3e python`posix_listdir+0x6d python`PyEval_EvalFrameEx+0x59ff python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalFrameEx+0x49ff python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalCode+0x22 python`PyRun_FileExFlags+0xaf python`PyRun_SimpleFileExFlags+0x156 python`Py_Main+0xa6b python`main+0x17 python`_start+0x80 0 42244 lmalloc:return Ptr=0xcf894000 Size=8192 libc.so.1`lmalloc+0x143 libc.so.1`opendir+0x3e python`posix_listdir+0x6d python`PyEval_EvalFrameEx+0x59ff python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalFrameEx+0x49ff python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalCode+0x22 python`PyRun_FileExFlags+0xaf python`PyRun_SimpleFileExFlags+0x156 python`Py_Main+0xa6b python`main+0x17 python`_start+0x80 0 42249 free:entry Ptr=0x86d78f0 ^C 0 42246 realloc:return Ptr=0x824c268 Oldptr=0x0 Size=16 libc.so.1`realloc+0x33a python`addcleanup+0x45
Re: [Mailman-Users] Python process size grows 30x in 8 hours (memory
Fletcher Cocquyt wrote: Interesting, the growth per python is limited to 50M by ulimit -v 5, but I'm seeing each one gradually take up that limit - then what ? (stay tuned! - mailman fails to malloc?) Look in Mailman's error and qrunner logs. -- Mark Sapiro [EMAIL PROTECTED]The highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Apache and pipermail 403 issues
Robert Campbell wrote: Currently when ever I try to get to the mailman archives for any of the mailing lists I revcive a 403 error. When I check the apache error log i see the following line: [Wed Jul 02 10:28:52 2008] [error] [client some ip] client denied by server configuration: var-install-location/archives/public/ The Private folder is owned by what apache runs under. Also the apache config file has Alias /pipermail/ var-install-location/archives/public/ Directory var-install-location AllowOverride All Options +FollowSymLinks AddDefaultCharset Off You could try adding here Order allow,deny Allow from all Often, Apache configs are set up to deny access to everything and then selectively allow the various DocumentRoot directories. Such a configuration would not allow access to the var-install-location directory. /Directory -- Mark Sapiro [EMAIL PROTECTED]The highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Mailman group on LinkedIn
On Wed, Jul 2, 2008 at 11:48 AM, Luigi Rosa [EMAIL PROTECTED] wrote: David Gibbs said the following on 02/07/2008 17.21: If you would like to join the group, visit http://www.linkedin.com/e/gis/134598/181E8C14C43E. Good idea! I just joined the group. And for those on Facebook: http://www.facebook.com/group.php?gid=17128034665 -Jim P. -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Python process size grows 30x in 8 hours (memory
Fletcher Cocquyt wrote: On 7/2/08 8:15 AM, Mark Sapiro [EMAIL PROTECTED] wrote: At some point in the next day or so, I'm going to make a modified scripts/post script which will queue incoming messages in qfiles/bad and then move them to qfiles/in only if they are under a certain size. I'm really curious to see if that will help. Yes, having a global incoming maxmessagesize limit and handler (what will the sender receive back?) for mailman would be useful. The attached 'post' file is a modified version of scripts/post. It does the following compared to the normal script. The normal script reads the message from the pipe from the MTA and queues it in the 'in' queue for processing by an IncomingRunner. This script receives the message and instead queues it in the 'bad' queue. It then looks at the size of the 'bad' queue entry (a Python pickle that will be just slightly larger than the message text). If the size is less than MAXSIZE bytes (a parameter near the beginning of the script, currently set to 100, but which you can change as you desire), it moves the queue entry from the 'bad' queue to the 'in' queue for processing. The end result is queue entries smaller than MAXSIZE will be processed normally, and entries = MAXSIZE will be left in the 'bad' queue for manual examination (with bin/dumpdb or bin/show_qfiles) and either manual deletion or manual moving to the 'in' queue for processing. The delivery is accepted by the MTA in either case so the poster sees nothing unusual. This is not intended to be used in a normal production environment. It is only intended as a debug aid to see if IncomingRunners will not grow so large if incoming message size is limited. -- Mark Sapiro [EMAIL PROTECTED]The highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Python process size grows 30x in 8 hours (memory
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Jul 2, 2008, at 11:01 PM, Mark Sapiro wrote: The attached 'post' file is a modified version of scripts/post. Hi Mark, there was no attachment. It does the following compared to the normal script. The normal script reads the message from the pipe from the MTA and queues it in the 'in' queue for processing by an IncomingRunner. This script receives the message and instead queues it in the 'bad' queue. It then looks at the size of the 'bad' queue entry (a Python pickle that will be just slightly larger than the message text). If the size is less than MAXSIZE bytes (a parameter near the beginning of the script, currently set to 100, but which you can change as you desire), it moves the queue entry from the 'bad' queue to the 'in' queue for processing. I'm not sure 'bad' should be used. Perhaps a separate queue called 'raw'? It is nice that files MAXSIZE need only be left in 'bad'. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iEYEARECAAYFAkhsQekACgkQ2YZpQepbvXEBPQCfUUH1ZxUkzXUVfkPF0iZ5c2sK JPMAoJiJNehIX+E24fyYeAQMbKkwI2Kv =MF40 -END PGP SIGNATURE- -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Python process size grows 30x in 8 hours (memory
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Barry Warsaw wrote:
| On Jul 2, 2008, at 11:01 PM, Mark Sapiro wrote:
|
| The attached 'post' file is a modified version of scripts/post.
|
| Hi Mark, there was no attachment.
Yes, I know. I was just about to resend. It is attached here. The MUA I
used to send the previous message gives any attachment without an
extension Content-Type: application/octet-stream, so the list's content
filtering removed it.
| It does the following compared to the normal script.
|
| The normal script reads the message from the pipe from the MTA and
| queues it in the 'in' queue for processing by an IncomingRunner. This
| script receives the message and instead queues it in the 'bad' queue.
| It then looks at the size of the 'bad' queue entry (a Python pickle
| that will be just slightly larger than the message text). If the size
| is less than MAXSIZE bytes (a parameter near the beginning of the
| script, currently set to 100, but which you can change as you
| desire), it moves the queue entry from the 'bad' queue to the 'in'
| queue for processing.
|
| I'm not sure 'bad' should be used. Perhaps a separate queue called
| 'raw'? It is nice that files MAXSIZE need only be left in 'bad'.
If we're going to do something like this going forward, we can certainly
change the queue. For this 'debug' effort, I wanted to keep it simple
and use an existing mm_cfg queue name.
- --
Mark Sapiro [EMAIL PROTECTED]The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.7 (MingW32)
iD8DBQFIbERTVVuXXpU7hpMRAtD3AJ4wak9befESKQlF3t2ZKos9W2WuTQCfbOCB
Yh9VIStJMHWfiLVlYjM5uoo=
=bU1+
-END PGP SIGNATURE-
# -*- python -*-
#
# Copyright (C) 1998,1999,2000,2001,2002 by the Free Software Foundation, Inc.
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
Accept posts to a list and handle them properly.
The main advertised address for a list should be filtered to this program,
through the mail wrapper. E.g. for list [EMAIL PROTECTED]', the `test'
alias would deliver to this script.
Stdin is the mail message, and argv[1] is the name of the target mailing list.
import os
import sys
import paths
from Mailman import mm_cfg
from Mailman import Utils
from Mailman.i18n import _
from Mailman.Queue.sbcache import get_switchboard
from Mailman.Logging.Utils import LogStdErr
LogStdErr(error, post)
MAXSIZE = 100
def main():
# TBD: If you've configured your list or aliases so poorly as to get
# either of these first two errors, there's little that can be done to
# save your messages. They will be lost. Minimal testing of new lists
# should avoid either of these problems.
try:
listname = sys.argv[1]
except IndexError:
print sys.stderr, _('post script got no listname.')
sys.exit(1)
# Make sure the list exists
if not Utils.list_exists(listname):
print sys.stderr, _('post script, list not found: %(listname)s')
sys.exit(1)
# Immediately queue the message for the incoming qrunner to process. The
# advantage to this approach is that messages should never get lost --
# some MTAs have a hard limit to the time a filter prog can run. Postfix
# is a good example; if the limit is hit, the proc is SIGKILL'd giving us
# no chance to save the message.
bdq = get_switchboard(mm_cfg.BADQUEUE_DIR)
filebase = bdq.enqueue(sys.stdin.read(),
listname=listname,
tolist=1, _plaintext=1)
frompath= os.path.join(mm_cfg.BADQUEUE_DIR, filebase + '.pck')
topath= os.path.join(mm_cfg.INQUEUE_DIR, filebase + '.pck')
if os.stat(frompath).st_size MAXSIZE:
os.rename(frompath,topath)
if __name__ == '__main__':
main()
--
Mailman-Users mailing list
Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe:
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Python process size grows 30x in 8 hours (memory
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Jul 2, 2008, at 11:15 PM, Mark Sapiro wrote:
Yes, I know. I was just about to resend. It is attached here. The
MUA I
used to send the previous message gives any attachment without an
extension Content-Type: application/octet-stream, so the list's
content
filtering removed it.
Ah, np.
| It does the following compared to the normal script.
|
| The normal script reads the message from the pipe from the MTA and
| queues it in the 'in' queue for processing by an IncomingRunner.
This
| script receives the message and instead queues it in the 'bad'
queue.
| It then looks at the size of the 'bad' queue entry (a Python pickle
| that will be just slightly larger than the message text). If the
size
| is less than MAXSIZE bytes (a parameter near the beginning of the
| script, currently set to 100, but which you can change as you
| desire), it moves the queue entry from the 'bad' queue to the 'in'
| queue for processing.
|
| I'm not sure 'bad' should be used. Perhaps a separate queue called
| 'raw'? It is nice that files MAXSIZE need only be left in 'bad'.
If we're going to do something like this going forward, we can
certainly
change the queue. For this 'debug' effort, I wanted to keep it simple
and use an existing mm_cfg queue name.
Excellent point. A couple of very minor comments on the file, but
other than that, it looks great. (I know you copied this from the
original file, but still I can't resist. ;)
# Copyright (C) 1998,1999,2000,2001,2002 by the Free Software
Foundation, Inc.
1998-2008
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
02110-1301, USA.
Accept posts to a list and handle them properly.
The main advertised address for a list should be filtered to this
program,
through the mail wrapper. E.g. for list [EMAIL PROTECTED]', the
`test'
alias would deliver to this script.
Stdin is the mail message, and argv[1] is the name of the target
mailing list.
import os
import sys
import paths
from Mailman import mm_cfg
from Mailman import Utils
from Mailman.i18n import _
from Mailman.Queue.sbcache import get_switchboard
from Mailman.Logging.Utils import LogStdErr
LogStdErr(error, post)
MAXSIZE = 100
def main():
# TBD: If you've configured your list or aliases so poorly as to
get
# either of these first two errors, there's little that can be
done to
# save your messages. They will be lost. Minimal testing of new
lists
# should avoid either of these problems.
try:
listname = sys.argv[1]
except IndexError:
print sys.stderr, _('post script got no listname.')
sys.exit(1)
# Make sure the list exists
if not Utils.list_exists(listname):
print sys.stderr, _('post script, list not found: %
(listname)s')
sys.exit(1)
# Immediately queue the message for the incoming qrunner to
process. The
# advantage to this approach is that messages should never get
lost --
# some MTAs have a hard limit to the time a filter prog can run.
Postfix
# is a good example; if the limit is hit, the proc is SIGKILL'd
giving us
# no chance to save the message.
bdq = get_switchboard(mm_cfg.BADQUEUE_DIR)
filebase = bdq.enqueue(sys.stdin.read(),
listname=listname,
tolist=1, _plaintext=1)
Should probably use True there instead of 1.
frompath= os.path.join(mm_cfg.BADQUEUE_DIR, filebase + '.pck')
topath= os.path.join(mm_cfg.INQUEUE_DIR, filebase + '.pck')
Space in front of the =
if os.stat(frompath).st_size MAXSIZE:
os.rename(frompath,topath)
Space after the comma.
if __name__ == '__main__':
main()
- -Barry
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (Darwin)
iEYEARECAAYFAkhsTbQACgkQ2YZpQepbvXGGigCfe+w4Ynz/qvFEp6VmurbySv42
b6cAoJa9wuSaql8dLUo8/VXT/Sxiu9pW
=Ywsy
-END PGP SIGNATURE-
--
Mailman-Users mailing list
Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe:
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Security Policy: http://wiki.list.org/x/QIA9
[Mailman-Users] Bounce Message, ReturnPath/ErrorsTo issue
Configuration - CentOS 5 64 bit - Mailman 2.1.9 - Qmail with Plesk 8.3 mods I have an interesting bounce issue occurring. - Email is sent to a Mailman List - List sends to each address - One of the addresses bounces - The bounce message ends with the right FROM and TO headers, but the wrong RETURN-PATH and ERRORS-TO headers. Is this a Mailman or Qmail issue? Here is the bounce message -- Return-Path: [EMAIL PROTECTED] Received: (qmail 28263 invoked from network); 2 Jul 2008 14:46:36 -0400 Received: from localhost (HELO [MAIL.RIGHTDOMAIN.COM]) (127.0.0.1) by localhost with SMTP; 2 Jul 2008 14:46:36 -0400 Return-Path: Delivered-To: [EMAIL PROTECTED] Received: (qmail 28252 invoked for bounce); 2 Jul 2008 14:46:34 -0400 Date: 2 Jul 2008 14:46:34 -0400 From: [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: failure notice Errors-To: [EMAIL PROTECTED] Hi. This is the qmail-send program at [MAIL.RIGHTDOMAIN.COM]. I'm afraid I wasn't able to deliver your message to the following addresses. This is a permanent error; I've given up. Sorry it didn't work out. [BADEMAIL ADDRESS]: [REMOTEIP] does not like recipient. Remote host said: 550-5.1.1 The email account that you tried to reach does not exist. Please 550-5.1.1 try double-checking the recipient's email address for typos 550-5.1.1 or unnecessary spaces. Learn more at 550 5.1.1 http://mail.google.com/support/bin/answer.py?answer=6596 23si5229629hsd.10 Giving up on [REMOTEIP]. --- Below this line is a copy of the message. Return-Path: [EMAIL PROTECTED] Received: (qmail 28243 invoked from network); 2 Jul 2008 14:46:34 -0400 Received: from localhost (HELO [MAIL.RIGHTDOMAIN.COM]) (127.0.0.1) by localhost with SMTP; 2 Jul 2008 14:46:34 -0400 Return-Path: [SENDEREMAIL] Delivered-To: [EMAIL PROTECTED] Received: (qmail 28072 invoked from network); 2 Jul 2008 14:46:32 -0400 Received: [OMMITTED] Received: [OMMITTED] Message-ID: [OMMITTED] Date: Wed, 02 Jul 2008 14:46:27 -0400 From: [OMMITTED] User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: [EMAIL PROTECTED] Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: [members] whiter water trip update X-BeenThere: [EMAIL PROTECTED] X-Mailman-Version: 2.1.9 Precedence: list Errors-To: [EMAIL PROTECTED] [ORIGINAL MESSAGE] -- In Service, Michael Kabot President - SOAR Scouting Online Affordable Reliable [EMAIL PROTECTED] 585-388-0211 www.soarol.com -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Bounce Message, ReturnPath/ErrorsTo issue
Michael Kabotwrote: Configuration - CentOS 5 64 bit - Mailman 2.1.9 - Qmail with Plesk 8.3 mods I have an interesting bounce issue occurring. - Email is sent to a Mailman List - List sends to each address - One of the addresses bounces - The bounce message ends with the right FROM and TO headers, but the wrong RETURN-PATH and ERRORS-TO headers. Is this a Mailman or Qmail issue? Here is the bounce message -- Return-Path: [EMAIL PROTECTED] Received: (qmail 28263 invoked from network); 2 Jul 2008 14:46:36 -0400 Received: from localhost (HELO [MAIL.RIGHTDOMAIN.COM]) (127.0.0.1) by localhost with SMTP; 2 Jul 2008 14:46:36 -0400 It looks like below here up to my next comment are the actual bounce notice headers with Return-Path: indicating a null envelope from which should be the case for a normal bounce. The headers above here look like Mailman received this bounce and resent it to a list owner. When sending to a list owner, Mailman sets the envelope from the site list (mailman) -bounces address. Is [WRONGDOMAIN.COM] the domain of the site list? Return-Path: Delivered-To: [EMAIL PROTECTED] Received: (qmail 28252 invoked for bounce); 2 Jul 2008 14:46:34 -0400 Date: 2 Jul 2008 14:46:34 -0400 From: [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: failure notice And the header below may have also been added by Mailman in resending. Errors-To: [EMAIL PROTECTED] Hi. This is the qmail-send program at [MAIL.RIGHTDOMAIN.COM]. I'm afraid I wasn't able to deliver your message to the following addresses. This is a permanent error; I've given up. Sorry it didn't work out. [BADEMAIL ADDRESS]: [REMOTEIP] does not like recipient. Remote host said: 550-5.1.1 The email account that you tried to reach does not exist. Please 550-5.1.1 try double-checking the recipient's email address for typos 550-5.1.1 or unnecessary spaces. Learn more at 550 5.1.1 http://mail.google.com/support/bin/answer.py?answer=6596 23si5229629hsd.10 Giving up on [REMOTEIP]. --- Below this line is a copy of the message. Return-Path: [EMAIL PROTECTED] Received: (qmail 28243 invoked from network); 2 Jul 2008 14:46:34 -0400 Received: from localhost (HELO [MAIL.RIGHTDOMAIN.COM]) (127.0.0.1) by localhost with SMTP; 2 Jul 2008 14:46:34 -0400 Return-Path: [SENDEREMAIL] Delivered-To: [EMAIL PROTECTED] Received: (qmail 28072 invoked from network); 2 Jul 2008 14:46:32 -0400 Received: [OMMITTED] Received: [OMMITTED] Message-ID: [OMMITTED] Date: Wed, 02 Jul 2008 14:46:27 -0400 From: [OMMITTED] User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: [EMAIL PROTECTED] Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: [members] whiter water trip update X-BeenThere: [EMAIL PROTECTED] X-Mailman-Version: 2.1.9 Precedence: list Errors-To: [EMAIL PROTECTED] [ORIGINAL MESSAGE] -- The thing that's strange in all the above is that Mailman (at least standard GNU Mailman) doesn't redirect bounces like this. It sends a totally new message to the list owner with the original bounce attached. Also, assuming [BADEMAIL ADDRESS]: in the notice was enclosed in as in [EMAIL PROTECTED]:, it would not have been an unrecognized bounce in 2.1.9. Is this Mailman 2.1.9 installed from our distribution, or is it a Plesk or other package? -- Mark Sapiro [EMAIL PROTECTED]The highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Mailman 2.1.11 final has been released.
Barry Finkel writes: I have not yet installed my 2.1.11 package on my test machine, but my records show that I had the identical messages when I built my 2.1.9 package, which I use in production. I am not an expert in the C language and what type conversions are done. But I would not worry about these messages. I do not know if the C code can be changed to eliminate these messages. It is very difficult to safely eliminate these messages. It's probably not worth trying. -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
[Mailman-Users] Generating list archive from a mbox file
Dear All, I have a list with of 500 user and 20-25 post per day. My list administrator disable list archiving by mistake from last three month and now he want them back. As we know mailman save mails in mbox formate and from that file it generate archive regularly. So i take all the mails that we want to upload on the list and save them in mbox format and generate archive manually with the help of 'bin/arch'. It dose work it generate archive of those three mount on the list there are some problem in the archive mail there attachment are not working and some mails lost there body and some text are unreadable. Can some help me out if there is another way to figure it out . Thanks Nitin -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
