Re: [Mailman-Users] A quick question about slicing outgoing qrunner

[Chris Nulk]

Hi Mark,

Thanks for the information.  I plan on looking at the MTA. 
Unfortunately, when we migrated Mailman to a new server, I don't have 
the same abilities to effect changes as I did on the old server.  The 
server is now managed by a different group and I just run an 
"application" on their server.  Makes identifying and fixing problems a 
major pain in ...


No problem setting the number of queues to a power of two.  I used 3 as 
an example.  Thanks for you input, Barry, for clarifying it.


Jim, the settings you are using look interesting.  I will have to think 
on it.


The biggest issue I have right now is modifications to the MTA.  We are 
using sendmail (it was used on the old server). When we migrated 
Mailman to the new server, I had to help them setup sendmail since I set 
it up on the old server.  The admins don't know sendmail or postfix 
except on a very minimal level.  It is not part of their normal 
responsibilities.


Anyone willing to take this off-list and help me figure out how to 
switch from sendmail to postfix?  Every time I have tried postfix, it 
has ended very miserably. :)


Thanks,
Chris

On 9/7/2016 4:23 PM, Mark Sapiro wrote:

On 09/07/2016 02:54 PM, Chris Nulk wrote:

I have read the archives regarding slicing the qrunner queues. Mailman
here runs on a single system (virtual) along with the local MTA and web
server.  Our outgoing queue recently got bogged down.  I would like to
increase the number of queues for the outgoing qrunner to three instead
of one.


The first thing you need to understand is why the outgoing queue got
"bogged down". Depending on the actual cause, increasing the number of
OutgoingRunner slices may not help. Usually, doing things in the MTA is
more effective. The most effective thing often is setting up a separate
submission port for Mailman that bypasses  lot of checks.



My question is will I encounter any problems with changing (in
mm_cfg.py) the QRUNNERS entry to:

QRUNNERS = [
 ('ArchRunner', 1), # messages for the archiver
 ('BounceRunner',   1), # for processing the qfile/bounces directory
 ('CommandRunner',  1), # commands and bounces from the outside world
 ('IncomingRunner', 1), # posts from the outside world
 ('NewsRunner', 1), # outgoing messages to the nntpd
 ('OutgoingRunner', 3), # outgoing messages to the smtpd (change to
three '3' queues)
 ('VirginRunner',   1), # internally crafted (virgin birth) messages
 ('RetryRunner',1), # retry temporarily failed deliveries
 ]


That should be fine assuming you haven't set

USE_MAILDIR = No

but see caveat. Another way which doesn't depend on USE_MAILDIR or a
future release not adding or deleting a new queue is

QRUNNERS.remove(('OutgoingRunner', 1))
QRUNNERS.append(('OutgoingRunner', 3))

The QRUNNERS list is not order sensitive so you don't need to insert the
new entry where the old one was removed.

Caveat: Documentation says the number of slices must be a power of two.
This is based on the theory that slice membership is based on the last
bits of the queue entry name (hash), but it hasn't been that way for
ages if ever. I'm certain the code works with 3 slices, but that's
relatively untested.



--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

[Mailman-Users] A quick question about slicing outgoing qrunner

[Chris Nulk]

Hello all,

I have read the archives regarding slicing the qrunner queues. Mailman 
here runs on a single system (virtual) along with the local MTA and web 
server.  Our outgoing queue recently got bogged down.  I would like to 
increase the number of queues for the outgoing qrunner to three instead 
of one.


My question is will I encounter any problems with changing (in 
mm_cfg.py) the QRUNNERS entry to:


QRUNNERS = [
('ArchRunner', 1), # messages for the archiver
('BounceRunner',   1), # for processing the qfile/bounces directory
('CommandRunner',  1), # commands and bounces from the outside world
('IncomingRunner', 1), # posts from the outside world
('NewsRunner', 1), # outgoing messages to the nntpd
('OutgoingRunner', 3), # outgoing messages to the smtpd (change to three 
'3' queues)
('VirginRunner',   1), # internally crafted (virgin birth) messages
('RetryRunner',1), # retry temporarily failed deliveries
]

I also plan on performance tuning of the MTA.

Thanks,
Chris
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Mailman and Thunderbird

[Chris Nulk]

Hello,

It could be the user needs to use the manage identities feature in 
Thunderbird.  If you have an email address A that also has an alias 
address B, when you set up the email address A account in Thunderbird, 
it will use the A email address when you reply to messages even if the 
message was sent to the B email address.  Using the Manage Identities 
feature (on the Account Settings page near the bottom), you can add the 
alias email address B.  Once done, when you reply to a message sent to 
the A address, Thunderbird will use the A address for the From: field.  
When you reply to a message sent to the B address, Thunderbird will use 
the B address for the From: field.  If you have multiple address that 
all consolidate to one address, this is a convenient way to receive mail 
and reply using the correct address for the reply.


Just a thought,
Chris

On 4/20/2016 1:14 AM, Christian F Buser via Mailman-Users wrote:

Hi all

I have no idea where this question belongs to... but there might be some 
knowing person out here, I hope!

We switched our lists to a new provider and now are using Mailman 2.1.20 from 
Cpanel.

A user complained that his Thunderbird mail application (on MacOS X, if this matters) does no longer treat 
incoming list mails properly - means, the message is treated by Thunderbird as if it was sent to his 
"default" mail address (we call it "A" for now) instead of the address he used to 
subscibe to the list (address "B").

This would not be a big problem - but when he replies to a list message, the reply is sent from 
"A" instead of "B", resulting in an error message from Mailman...

He has analysed the headers and compared with the old list setup headers. As far as he could see, 
the only difference was that Mailman list includes an "Envelope-To"-header with his 
subscription address, while the precious list version included a "Delivered-To"-header.

I am not a "mail header specialist", but I would guess that these two headers are added 
by the receiving mail server and not by the list software. Is this assumption correct? If not, can 
Mailman be configured to send the "Delivered-To"-header?

And if there are by chance any Thunderbird-specialists among us: what can he do 
to correct the situation?

Thank you, Christian



--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Renaming a list

[Chris Nulk]

On 3/24/2016 11:49 AM, Mark Sapiro wrote:

On 03/24/2016 09:17 AM, Chris Nulk wrote:

I did a little searching around our lists.  It seems the attachments
directory is always created and populated whether or not scrub_nondigest
was ever set to Yes.  So I can't really use the existence of the
attachments directory or the lack of it being populated as a check for
if scrub_nondigest was ever set to Yes.


No, but here's what you can look for.

If scrub_nondigest is True, the message will be scrubbed during
processing before any delivery and before the message is added to the
cumulative mbox. Thus there will be one copy of each scrubbed attachment
in the attachments directory and a link in the message body and the
scrubbed attachment will not be in the cumulative mbox.



[SNIPPED a big chunk]


The loss of data problem is if scrub_nondigest is No and was Yes in the
past, the attachments that were scrubbed when it was Yes will be lost.

Thanks for the information Mark.  It is a messy situation.  I decided to 
simplify the changes I made to clone_list.


1. The -b/--rebuild_archives option now requires a Yes/No/? answer 
(don't know if I did it right in the parser)

2. If -b/--rebuild_archives is:
a. specified on the command line but without Yes/No/?, '?' is assumed,
b. '?', program prints warning message and exits,
c. 'No', program proceeds with list cloning and does NOT rebuild 
the archives,
d. 'Yes', program proceeds with list cloning and DOES rebuild the 
archives (no warnings).


Hopefully, the changes will catch casual mistakes.

Below is the diff from Mark's original clone_list.  Please let me know 
if there are any improvements to be made.


Thanks,
Chris

--- diff file 
--- clone_list2016-03-18 10:28:14.0 -0700
+++ clone_list_scu2016-03-29 11:23:22.0 -0700
@@ -109,6 +109,17 @@
   dest='archives', action='store_true',
   help="""\
 Clone the archives of the old list. The default is an empty archive.""")
+parser.add_argument('-b', '--rebuild_archives',
+  dest='rebuild_archives', nargs='?',
+  const='?', default='No',
+  help="""\
+Rebuild the archives of the new list. Requires -a/--archives.
+  * WARNING *
+If scrub_nondigest was 'Yes' at any time in a list's past,
+rebuilding the archives will lose the scrubbed attachments from the
+'Yes' period.  Please review the settings and archives for the list
+prior to rebuilding the archives.  At a minimum, make backups of the
+list and its archives.""")
 parser.add_argument('-e', '--extra_files',
   dest='extra', action='store_true',
   help="""\
@@ -186,6 +197,25 @@
 abort("%s doesn't appear to be a valid email address" % 
ns.owner)

 if ns.extra and not ns.clone_members:
 abort('-e/--extra_files requires -m/--members.')
+
+if ns.rebuild_archives.lower() not in set(['no', 'n', 'yes', 'y', 
'?']):
+abort('%s is not valid for -b/--rebuild_archives.  Please 
answer Yes or No.' % ns.rebuild_archives)

+rebuild_archives = ns.rebuild_archives[0].lower()
+if rebuild_archives == '?':
+abort("""
+Specifying rebuild_archives requires an affirmative indication for
+%s's archives to be rebuilt.
+
+   * WARNING *
+If scrub_nondigest was 'Yes' at any time in %s's past,
+rebuilding the archives will lose the scrubbed attachments from the
+'Yes' period.  Please review the settings and archives for %s
+prior to rebuilding the archives.  At a minimum, make backups of the
+list and its archives.""" % (old_list, old_list, old_list))
+
+if rebuild_archives == 'y' and not ns.archives:
+abort('-b/--rebuild_archives requires -a/--archives.')
+
 if ns.verbose:
 print 'Getting %s list...' % ns.old_list
 ol = MailList(old_list, lock=False)
@@ -268,5 +298,26 @@
(ns.old_list, ns.new_list))
 copy_archives(old_list, new_list)

+if rebuild_archives == 'y':
+if ns.verbose:
+print 'Rebuilding %s archives...' % ns.new_list
+archcmd = os.path.join(os.path.dirname(sys.argv[0]), 'arch')
+if not os.path.isfile(archcmd):
+abort("""%s doesn't exist.
+Am I installed in Mailman's bin/ directory?""" % archcmd)
+rbld

Re: [Mailman-Users] Renaming a list

[Chris Nulk]

On 3/23/2016 6:42 PM, Mark Sapiro wrote:

On 03/23/2016 11:15 AM, Chris Nulk wrote:

I have done limited testing (4 member list with a few archived messages)
and it appears to be working.  The archives were moved and rebuilt.
When I changed the scrub_nondigest setting to Yes, the list is renamed,
archives moved, and an error message that the archives were not rebuilt.


Actually, is scrub_nondigest is Yes, bin/arch --wipe will work but there
are lots of caveats. In this case, bin/arch --wipe will just not remove
the 'attachments' directory. The issue is if scrub_nondigest was always
Yes for this list, that's the right thing to do, but it it was No at
some time in the past and changed to Yes, there are already two copies
of each attachment from the No period (one scrubbed from the archive and
one from the plain digest) and running bin/arch --wip will leave those
and create  third copies from the attachments in the mbox.

The more serious issue is if scrub_nondigest was Yes at some time in the
past and is now No, bin/arch --wipe will lose the attachments from the
Yes period.

In any case, for attachments scrubbed by scrub_nondigest, there are
links to them in the message bodies in the mbox and the list names in
those links will now be wrong, and bin/arch won't (can't) fix them.

This is why I chose in my script to kick the issue back to the user.


I figured it was a bit more complicated.  So, there is still a 
possibility of losing data if scrub_nondigest is No since the list 
doesn't maintain a history of previous settings.






Based on my testing, I noticed the error message for not rebuilding the
archives is a bit terse.  For my target audience it needs to be a little
more helpful.  So, I changed the lines:

if ol.scrub_nondigest:
 abort('Scrub_nondigest is YES for %s.  WILL NOT REBUILD
ARCHIVES.' % ns.old_list)
else:

with

 if ol.scrub_nondigest:
 abort("""ARCHIVES NOT REBUILT.

Scrub_nondigest for %s is set to Yes.  To prevent losing any scrubbed
attachments, you will need to rebuild and fix the listname links in
the archives using another method than using bin/arch.
""" % ns.old_list)
 else:


That much seems good, but as I note above it's a bit more complicated if
the scrub_nondigest setting has changed over time.


I did a little searching around our lists.  It seems the attachments 
directory is always created and populated whether or not scrub_nondigest 
was ever set to Yes.  So I can't really use the existence of the 
attachments directory or the lack of it being populated as a check for 
if scrub_nondigest was ever set to Yes.


Is the anything that can be checked for that would indicate 
scrub_nondigest was Yes at any point in the past?  Obviously, if it is 
currently Yes it doesn't matter.


Lacking the ability to check,  I suppose a warning should be generated 
prior to running the bin/arch.  In addition, the '-b/--rebuild_archives' 
option should default to false unless there is a positive indication the 
user wants to run it ('-b y / --rebuild_archives=Yes').  The default for 
the rebuild_archives option would be No.


If there is any way to tell if scrub_nondigest has been set to Yes in 
the list's past, please let me know.  I am going to add the warning and 
try making the changes to the rebuild_archives options. I will post the 
diff from the Mark's original clone_list.


Thanks Mark,
Chris
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Renaming a list

[Chris Nulk]

On 3/23/2016 10:39 AM, Adam McGreggor wrote:

On Tue, Mar 22, 2016 at 11:02:10AM -0700, Chris Nulk wrote:

I have a copy of Mark's clone_list command and it will allow us to
rename lists.  Which is great and not a problem for me.  However, I
am not the person normally involved with list creation, etc. Another
group does it.  Normally through the web interface which works well
for them.  Unless a mistake is made then they have to use the
command line tools.  Again, not a problem for me but that group has
very limited knowledge of linux/unix and Mailman.

If they can be assumed to use old list and new list as parameters
(although I could do a check for each…)

 https://gist.github.com/adamamyl/6909815#file-rename-list

may be useful

invoked as `rename-list old-list new-list`

Thanks for the info Adam.  I was planning on creating a script to do the 
renaming like your script.   While looking at the Mailman FAQ's on 
renaming a list, I saw the link to Mark's clone_list.  To make it easier 
for the group that would be using the script, I added in the bits to 
also copy the archives and rebuild them.  I am just happy I don't have 
to write the bash script I was planning.  The process has eighteen steps 
including notifying the list owner (important), doing the work, 
verifying everything, and administrative work.  The new process using 
clone_list has seven steps most of which is administrative.


Thanks again,
Chris
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Renaming a list

[Chris Nulk]

Thanks for the info Bill.

I agree getting off on the right foot for a new list is important. We 
have a basic form users fill out which includes our taxonomy for placing 
the lists.  The message the group gets has all the information.  The 
problem is sometimes bits of the key taxonomy are left off.


Thanks,
Chris

On 3/23/2016 7:56 AM, The Mailing List System Admin wrote:

CN> What I have done is modify the clone_list command to combine the
CN> renaming/cloning of a list and include an option to rebuild the
CN> archives.  However, the archives will not be rebuilt if the
CN> scrub_nondigest setting is Yes.

Sounds like a worthwhile enhancement.

MS> I will try adding it to my script and doing a bit of testing, but I
MS> think it will be fine.

Does this mean it will be included in the next release of MM v2?

As for getting off on the right foot for a new list, as part of our
new list application process...

http://lists.unh.edu/MM/NewList/

we have the new list owner pick an initial flavor for their list --
announce, discussion, moderated.  The backend uses one of three MM
list definition templates to apply the appropriate defaults for the
given list type.  I used this system for about 15 years with ListProc
and it has always worked well.  (I'd offer the share the code, but it
is in Perl.  It has extensive commenting, but would still probably
take a fair amount of grokking and hacking to get it to work at a new
site.  It was never designed to distributed.)

BTW -- if anyone goes through the application process above to take a
look, please use the list name 'bogus.list' so I can ignore/delete it.



--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Renaming a list

[Chris Nulk]

On 3/22/2016 5:03 PM, Mark Sapiro wrote:

On 03/22/2016 11:02 AM, Chris Nulk wrote:

Below my message is a diff of my changes to clone_list.  While I try
testing my changes on a test system, I would appreciate it if someone
could take a look at my changes to ensure I am not off-base with my
process.


I have looked at it and it looks fine to me.

I will try adding it to my script and doing a bit of testing, but I
think it will be fine.


Thanks for looking at it Mark.

I have done limited testing (4 member list with a few archived messages) 
and it appears to be working.  The archives were moved and rebuilt.  
When I changed the scrub_nondigest setting to Yes, the list is renamed, 
archives moved, and an error message that the archives were not rebuilt.


Based on my testing, I noticed the error message for not rebuilding the 
archives is a bit terse.  For my target audience it needs to be a little 
more helpful.  So, I changed the lines:


   if ol.scrub_nondigest:
abort('Scrub_nondigest is YES for %s.  WILL NOT REBUILD 
ARCHIVES.' % ns.old_list)

   else:

with

if ol.scrub_nondigest:
abort("""ARCHIVES NOT REBUILT.

Scrub_nondigest for %s is set to Yes.  To prevent losing any scrubbed
attachments, you will need to rebuild and fix the listname links in
the archives using another method than using bin/arch.
""" % ns.old_list)
else:

I will do a little more testing on some larger lists but it looks promising.

Thank you again Mark,
Chris

--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

[Mailman-Users] Renaming a list

[Chris Nulk]

Hello all,

We have come up a few times where a list was created incorrectly. 
Normally, we catch the mistake, delete the list, and recreate it 
properly.  However, there are times when the mistake is caught after 
having been in use.  So, the delete and recreate is not available to use.


I have a copy of Mark's clone_list command and it will allow us to 
rename lists.  Which is great and not a problem for me.  However, I am 
not the person normally involved with list creation, etc. Another group 
does it.  Normally through the web interface which works well for them.  
Unless a mistake is made then they have to use the command line tools.  
Again, not a problem for me but that group has very limited knowledge of 
linux/unix and Mailman.


I can easily train them to use the Mailman tools, however, ensuring they 
do the appropriate checks to avoid other problems is more difficult.  
Specifically, they can run the clone_list command but rebuilding the 
archives requires checking if the scrub_nondigest settings is No (this 
is explained in the clone_list help).


What I have done is modify the clone_list command to combine the 
renaming/cloning of a list and include an option to rebuild the 
archives.  However, the archives will not be rebuilt if the 
scrub_nondigest setting is Yes.


Below my message is a diff of my changes to clone_list.  While I try 
testing my changes on a test system, I would appreciate it if someone 
could take a look at my changes to ensure I am not off-base with my process.


Thank you,
Chris

--- My diff of clone_list ---
--- clone_list2016-03-18 10:28:14.0 -0700
+++ clone_list_scu2016-03-22 10:19:43.0 -0700
@@ -109,6 +109,10 @@
   dest='archives', action='store_true',
   help="""\
 Clone the archives of the old list. The default is an empty archive.""")
+parser.add_argument('-b', '--rebuild_archives',
+  dest='rebuild_archives', action='store_true',
+  help="""\
+Rebuild the archives of the new list. Requires -a/--archives.""")
 parser.add_argument('-e', '--extra_files',
   dest='extra', action='store_true',
   help="""\
@@ -186,6 +190,8 @@
 abort("%s doesn't appear to be a valid email address" % 
ns.owner)

 if ns.extra and not ns.clone_members:
 abort('-e/--extra_files requires -m/--members.')
+if ns.rebuild_archives and not ns.archives:
+abort('-b/--rebuild_archives requires -a/--archives.')
 if ns.verbose:
 print 'Getting %s list...' % ns.old_list
 ol = MailList(old_list, lock=False)
@@ -268,5 +274,29 @@
(ns.old_list, ns.new_list))
 copy_archives(old_list, new_list)

+if ns.rebuild_archives:
+if ns.verbose:
+print 'Rebuilding %s archives...' % ns.new_list
+if ol.scrub_nondigest:
+abort('Scrub_nondigest is YES for %s.  WILL NOT REBUILD 
ARCHIVES.' % ns.old_list)

+else:
+archcmd = os.path.join(os.path.dirname(sys.argv[0]), 'arch')
+if not os.path.isfile(archcmd):
+abort("""%s doesn't exist.
+Am I installed in Mailman's bin/ directory?""" % archcmd)
+rbld = subprocess.Popen([archcmd,
+ '--wipe',
+ new_list
+],
+stdout=subprocess.PIPE,
+stderr=subprocess.PIPE
+   )
+so, se = rbld.communicate()
+if rbld.returncode:
+abort('unable to rebuild archives for %s\n%s' % 
(ns.new_list, se))

+# If there was stdout output, print it. It is probably aliases.
+if so:
+print so
+
 if __name__ == '__main__':
 main()

--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] automating adding and removing subscribers

[Chris Nulk]

Hello,

It depends on how much work you want to do and how timely the changes 
need to be made.  Here is what we have and do...


We have two lists for each group of users.  Both lists are used. One 
list is dynamically built using LDAP.  That list is considered the 
master list.  Members are not allowed to remove themselves nor change 
any options.  The second list is a regular list built using data from 
the master list.  Users can removed themselves and change their options.


To automate the adding and removing of subscribers on the second list, we:
1. if .today exists, rename .today to 
.yesterday

2. dump the list membership of the masters list to .today
3. diff -i --context=0 .today .yesterday -> 
.diff  so that each line of .diff has a '+' before 
the address to be added and a '-' before the address to be removed.
4. Split .diff into .add and .remove where 
.add has all the '+' addresses (without the '+') and 
.remove has the equivalent for address to remove.
5. Use the regular Mailman tools, add_members and remove_members, to add 
and remove subscribers to the second list.


We run this process once a day.  When a new user shows up in the master 
list, it takes a day to make it to the second list. Acceptable for us.  
Run more often if you need faster updates to the second list.


The master list doesn't have to be a list.  It can be a list of address 
from a database, LDAP, or anything else.  All the processing to the 
master list information can be done on another system or the same 
system.  Only send the .add and .remove files to the 
Mailman system for final processing.


The process works for us.  Diff occasionally glitches on closely matched 
names but a follow on process run weekly fixes the problem.


Good Luck,
Chris

On 10/20/2015 2:42 PM, Steven Jones wrote:

Hi,

Is there a way to automate the adding and removing subscribers from another 
system? such that the list is kept concurrent?



regards

Steven
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/cnulk%40scu.edu


--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Additional questions about ban_list

[Chris Nulk]

On 10/15/2015 9:33 PM, Mark Sapiro wrote:

Mark Sapiro wrote:

On 10/15/2015 02:37 PM, Chris Nulk wrote:

to the following ( I have added ** at the beginning of the lines I added
to indicate the changes - in practice the ** would be spaces)

 def GetBannedPattern(self, email):
 """Returns matched entry in ban_list if email matches.
 Otherwise returns None.
 """
 ban = False
 for pattern in self.ban_list:
 if pattern.startswith('^'):
 # This is a regular expression match
 try:
 if re.search(pattern, email, re.IGNORECASE):
 ban = True
 break
 except re.error:
 # BAW: we should probably remove this pattern
 pass
**  elif pattern.startswith('@'):
**  listname = self.internal_name()  # is this correct?
**  try:
**  mname = pattern[1:].lower().strip()
**  if mname == listname:
**  # don't reference your own list
**  syslog('error',
**  'Ban_list listfor %s references own list',
**  listname)
**  else:
**  mother = MailList(mname, lock=0)
**  if mother.isMember(email):
**  ban = True
**  break
**  except Errors.MMUnknownListError:
**  syslog('error',
**  'Ban_list for list %s references non-existent
list %s',
**  listname, mname)
 else:
 # Do the comparison case insensitively
 if pattern.lower() == email.lower():
 ban = True
 break
 if ban:
 return pattern
 else:
 return None

Am I on the correct path?


Yes.


One more thing however. While the above code looks good for processing
@listname entries in ban_list, you won't be able to add them via the
list admin GUI unless you also modify Mailman/Gui/GUIBase.py. You need
to find the lines in that module that in 2.1.18-1 are

 elif (wtype == mm_cfg.EmailListEx and addr.startswith('@')
 and property.endswith('_these_nonmembers')):

only indented more than above and change them to

 elif (wtype == mm_cfg.EmailListEx and addr.startswith('@')
 and (property.endswith('_these_nonmembers') or
  property == 'ban_list')):

If you don't make that change, you will still be able to add @listname
entries to a list's ban_list with withlist or other scripts, but the web
admin GUI will report them as bad email addresses.

Thank you for letting me know.  I am going to try to make the changes 
soon.  We try not to make changes on Fridays unless we really want to 
come in on Saturday to fix things.


Thanks,
Chris
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Additional questions about ban_list

[Chris Nulk]

On 10/15/2015 4:01 PM, Mark Sapiro wrote:

On 10/15/2015 03:25 PM, Chris Nulk wrote:


On 10/15/2015 3:07 PM, Mark Sapiro wrote:


This was done for subscribe_auto_approval, and
while it's kind of kludgy, the error log messages that you have above
for 'Ban_list listfor %s references own list' and 'Ban_list for list %s
references non-existent list %s' refer to subscribe_auto_approval rather
than ban_list or something variable.

Thus if you had 2.1.19 or later, the code change is much simpler if you
don't mind the error log messages saying subscribe_auto_approval when
they might mean ban_list.

Okay.  That makes sense.


I have decided to refactor this code to allow the attribute name to be
passed as the argument to GetPattern that says to allow @listname
syntax. This will make the error messages log the correct attribute name
and will simplify extending @listname recognition to other attributes in
the future. Along with this, I'm replacing the matches_p() function in
Moderate.py with calls to the GetPattern() method so all this is in one
place.

I haven't committed this change yet as it still needs testing, but I
expect it to be in 2.1.21.


Excellent.  Thank you for the hard work.

Chris
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Additional questions about ban_list

[Chris Nulk]

On 10/15/2015 3:07 PM, Mark Sapiro wrote:

On 10/15/2015 02:37 PM, Chris Nulk wrote:

The question now is if I change the following in Mailman/MailList.py

 def GetBannedPattern(self, email):
 """Returns matched entry in ban_list if email matches.
 Otherwise returns None.

Am I on the correct path?


Yes.

Great.  Thank you for the help.

Mark, if you have the time and/or inclination, could you explain your
comments about

 Allowing @list_name in ban_list is a simple code modification if you
 don't care if various 'error' log messages such as list references
 itself or references non-existent list refer to
 'subscribe_auto_approval' even if the error is in ban_list.


In Mailman 2.1.19 much of the code in the GetBannedPattern() method was
moved to a new GetPattern() method which optionally supports the
@listname convention.
Okay.  I have made a number of code changes to 2.1.9 and several of the 
dropped out/were not needed when we went to 2.1.18-1.  There still are a 
number of changes that keep us from switching without taking a close 
look at needs to be done.  I have recorded all the changes I have made, 
the problem is finding the time to work on it.


Thank you for the information though.


This was done for subscribe_auto_approval, and
while it's kind of kludgy, the error log messages that you have above
for 'Ban_list listfor %s references own list' and 'Ban_list for list %s
references non-existent list %s' refer to subscribe_auto_approval rather
than ban_list or something variable.

Thus if you had 2.1.19 or later, the code change is much simpler if you
don't mind the error log messages saying subscribe_auto_approval when
they might mean ban_list.

Okay.  That makes sense.

Thanks again for the help.

Chris
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Additional questions about ban_list

[Chris Nulk]

Hello all,

I would like to thank Aditya Jain and Mark Sapiro for the help with 
adding my negative regexp to the ban_list.  I apologize for being late 
in my response.  I was OBE'd.


Next, in my original message, I forgot to mention I am using Mailman 
v2.1.18-1 so the kind code assistance Mark provided doesn't quite 
match.  My fault for not putting in the version of Mailman. However, 
looking at the code, it seems to be remarkably similar to code I had to 
modify when using an earlier version of Mailman.


The question now is if I change the following in Mailman/MailList.py

def GetBannedPattern(self, email):
"""Returns matched entry in ban_list if email matches.
Otherwise returns None.
"""
ban = False
for pattern in self.ban_list:
if pattern.startswith('^'):
# This is a regular expression match
try:
if re.search(pattern, email, re.IGNORECASE):
ban = True
break
except re.error:
# BAW: we should probably remove this pattern
pass
else:
# Do the comparison case insensitively
if pattern.lower() == email.lower():
ban = True
break
if ban:
return pattern
else:
return None

to the following ( I have added ** at the beginning of the lines I added 
to indicate the changes - in practice the ** would be spaces)


def GetBannedPattern(self, email):
"""Returns matched entry in ban_list if email matches.
Otherwise returns None.
"""
ban = False
for pattern in self.ban_list:
if pattern.startswith('^'):
# This is a regular expression match
try:
if re.search(pattern, email, re.IGNORECASE):
ban = True
break
except re.error:
# BAW: we should probably remove this pattern
pass
**  elif pattern.startswith('@'):
**  listname = self.internal_name()  # is this correct?
**  try:
**  mname = pattern[1:].lower().strip()
**  if mname == listname:
**  # don't reference your own list
**  syslog('error',
**  'Ban_list listfor %s references own list',
**  listname)
**  else:
**  mother = MailList(mname, lock=0)
**  if mother.isMember(email):
**  ban = True
**  break
**  except Errors.MMUnknownListError:
**  syslog('error',
**  'Ban_list for list %s references non-existent 
list %s',

**  listname, mname)
else:
# Do the comparison case insensitively
if pattern.lower() == email.lower():
ban = True
break
if ban:
return pattern
else:
return None

Am I on the correct path?

Mark, if you have the time and/or inclination, could you explain your 
comments about


Allowing @list_name in ban_list is a simple code modification if you
don't care if various 'error' log messages such as list references
itself or references non-existent list refer to
'subscribe_auto_approval' even if the error is in ban_list.

Thanks to everyone for the help,
Chris



On 10/6/2015 5:43 PM, Mark Sapiro wrote:

On 10/06/2015 08:07 AM, Chris Nulk wrote:

1. The ban_list attribute is to help prevent unwanted people from
subscribing to a list, however, I want to restrict who can subscribe to
the list and ban anyone else.  I have the regex for who I want to allow
to subscribe but there isn't an allow_list attribute.  Is there an easy
way of allowing a regex to control who is able to subscribe?  Or, is
there a way to easily invert the regex logic and use it in ban_list?  As
an example (not the real regex), say I want to only allow @gmail.com to
subscribe to the list but no one else.


As Aditya Jain replied, you can use Python RE negative lookahead
assertions to create regexps with "doesn't match" conditions. See
<https://docs.python.org/2/library/re.html#regular-expression-syntax>.


Although the posted regexp modified for gmail

^[^@]+@(?!(.*\.)?gmail\.com$)

will not match and hence allow, addresses like u...@subdomain.gmail.com.
To allow only '@gmail.com' addresses, e.g. to ban all non-'@gmail.com'
addresses, use

^[^@]+@(?!gmail\.com$)

Often, i

[Mailman-Users] Additional questions about ban_list

[Chris Nulk]

Hello all,

I have been loosely following the discussion regarding regex in the 
ban_list attribute.  I now find myself being asked to create a list 
which will most likely use the ban_list attribute.  So, I have a few 
questions.


1. The ban_list attribute is to help prevent unwanted people from 
subscribing to a list, however, I want to restrict who can subscribe to 
the list and ban anyone else.  I have the regex for who I want to allow 
to subscribe but there isn't an allow_list attribute.  Is there an easy 
way of allowing a regex to control who is able to subscribe?  Or, is 
there a way to easily invert the regex logic and use it in ban_list?  As 
an example (not the real regex), say I want to only allow @gmail.com to 
subscribe to the list but no one else.


2. An additional requirement is to restrict a subgroup of the addresses 
from subscribing.  In short, I want to allow all @gmail.com addresses to 
subscribe except for a known subgroup.  Now the known subgroup is in a 
Mailman list.  So, can I use a Mailman list in the ban_list attribute 
similar to using a list in *_these_members attributes?  Or, would I have 
to modify the code to allow using a Mailman list in the ban_list attribute?


Thank you for your consideration,
Chris
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Regexp for blocking addresses

[Chris Nulk]

On 9/25/2015 9:16 PM, Mark Sapiro wrote:

On 09/25/2015 09:01 PM, Mark Sapiro wrote:

Here's another idea.

Find the following in /path/to/mailman/Mailman/MailList.py

 def GetBannedPattern(self, email):
 """Returns matched entry in ban_list if email matches.
 Otherwise returns None.
 """
 return self.GetPattern(email, self.ban_list)

and change it to

bad_users = ['joeblow@gmailcom',
  'johndoe@gmailcom',
  'jackblack@gmailcom',
  ...   (the rest of the addrs to ban)
 ]

...


Ooops. In the above addition, the bad_users line needs to be indented 4
spaces to line up with the following 'def ...'.



Would it make sense to define the "bad users" in mm_cfg.py?  That way it 
would be a bit easier to add/remove/change addresses when needed.  Also, 
wouldn't updating Mailman replace the modified 
/path/to/mailman/Mailman/MailList.py?


Just a thought,
Chris
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] ListProc to Mailman migration -- any tools or guides?

[Chris Nulk]

On 3/2/2015 12:44 PM, bill.co...@unh.edu wrote:

 Chris Nulk  recently wrote, in part...


Remember that communications with the list owners is extremely
important.


Indeed!


[snip]

We did change how we populated several of our lists though
which was a bit of a challange (we went from populating via a
static mechanism once per year to dynamic LDAP populating).


I have a similar issue -- I maintain student announcement lists
by college and class year, or even down to major code, with a
weekly update from Registrar's records.  I'll have to build that
all over again for Mailman.  Doesn't look like it should be too
bad using the command line tools.


If you can build LDAP queries to derive your list membership by college, 
class year, etc., you may want to look into using the LDAPMembership 
adapter.  The only issues are list members cannot remove themselves from 
the list or change any user options.  We use it to build four of our 
important "essential" communications lists for our community here 
(members are not allowed to remove themselves from the "essential" lists).




Thanks for the observations and advice.

...BC


No problem.  Happy to help.

Chris
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Jump from Python 2.4 to 2.7...

[Chris Nulk]

On 3/2/2015 9:53 AM, Mark Sapiro wrote:

On 03/02/2015 08:22 AM, bill.co...@unh.edu wrote:

My question is, are there any quirks or gotchas that I should be
aware of in going from the old RHEL/Python platform to the new
one?  Is there anything I can do on the old system now that will
make migration to the new platform easier?  For example, I could
use Pythonbrew to install a private v2.7 Python on the old box
for Mailman to use, rather than relying on the system's older
Python.


Are you installing Mailman 2.1.19 on the Red Hat system or using
whatever Red Hat rpm package that's available, and if the latter, what
is that? And, what Mailman will you be using on the RHEL 7 system?

I ask because the FAQ at  may be relevant.

Assuming you will be installing Mailman 2.1.19 from source, I would be
more concerned over potential difficulties due to the older Python than
with any gotchas from moving to a newer Python although there are some
compatibility issues with older Mailman versions and newer Python. See
 for more on both kinds.



In general is it a good practice to give Mailman it's own Python
installation rather than relying on and dinking with the system's
Python.


My production server is Centos 5 with Python 2.4.3, but I also have
Python 2.7.9 installed and configure Mailman
--with-python=/usr/bin/python2.7. I wouldn't necessarily call it "good
practice", but it's certainly a viable approach.



Interesting.  We moved our Mailman instance from CentOS 5.x running 
Mailman 2.1.9 to SLES version ? running Mailman 2.1.18-1 and python 2.7.


When we switched, the OS was some concern but the main decision we made 
was to use Mailman from the source at list.org and python 2.7. We have 
made some custom modifications here and getting Mailman from the source 
would allow us to upgrade Mailman a little easier than from the OS 
maintainer.   By divorcing our Mailman instance from the version 
provided by the OS maintainer, we can pick up our instance and switch to 
a newer host easier.


Chris


--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] ListProc to Mailman migration -- any tools or guides?

[Chris Nulk]

A few years ago, we switched from ListProc to Mailman.  We didn't have 
as extensive a set of lists as do you however.


Remember that communications with the list owners is extremely important.

What we did was:
0. Have a firm timeframe to complete the change over.
1. Try to weed out as many lists as you can that may not be used anymore.
2. Set up Mailman in parallel with ListProc.
3. Create new lists in Mailman to replace the lists in ListProc (we 
changed our naming convention so there wasn't a 1 to 1 move).  If your 
lists are going to be named the same, simple create a list in Mailman 
for each list to be moved from ListProc.

4. List owners of the Mailman lists should be the same as ListProc lists.
5. Communicate with the list owners of the ListProc lists and let them 
know how to access and configure the list on Mailman.
6. Give list owners time to make their changes (e.g. add moderators, set 
moderation, etc.) and test the list.
7. After list owner testing, populate the list (or have the list owner 
do it) membership.
8. Communicate with the list owners for a cutover time (either 
individually or all at once) to do any file conversions like archives, etc.

9. Monitor the process carefully.
10. Remember that firm timeframe, well some list owners are still going 
to miss it.  Be prepared.


Overall, the process went very smoothly for us.  We did not convert the 
archives.  We did change how we populated several of our lists though 
which was a bit of a challange (we went from populating via a static 
mechanism once per year to dynamic LDAP populating).


Chris


On 3/2/2015 7:24 AM, bill.co...@unh.edu wrote:

I'm going to be migrating about 400 lists with 25K subscribers
from ListProc to Mailman.  I'm looking for whatever tools I can
find to aid in the conversion process.  Of particular concern is
the creation of the replacement list in Mailman with a reasonable
mapping of the ListProc settings into equivalent Mailman
features, and of course the conversion of the subscribers.
Archive conversion would be nice too, but is a much lower
priority.

In my searching so far I've found a few bits and pieces, but
nothing comprehensive.  So before I get out the hammer and saw, I
just want to make sure I'm not missing a good resource for doing
this.

...BC



--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Small issue (hopefully) with upgrading/migrating from v2.1.9 to v2.1.18-1

[Chris Nulk]

On Wed, Jan 14, 2015 at 9:43 PM, Mark Sapiro  wrote:

> On 01/14/2015 09:38 PM, Chris Nulk wrote:
> > On Wed, Jan 14, 2015 at 6:33 PM, Mark Sapiro  wrote:
> >>
> >> DRY
> >>
> >
> > ​DRY?
>
> <http://en.wikipedia.org/wiki/Don%27t_repeat_yourself>
>
> ​ Well, I try not to but sometimes it doesn't work out.  Though I do try
to iterate dups out as I have time to revisit.

Chris

​P.S. I am more used to acronyms like FUBAR, DILLIGAF, and others.​  :)
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Upgrading/migrating Mailman v2.1.9 -> v2.1.18-1

[Chris Nulk]

On Wed, Jan 14, 2015 at 7:03 PM, Mark Sapiro  wrote:

> On 01/14/2015 09:31 AM, Chris Nulk wrote:
> >
>
> ​..​


>
> LDAPMemberships.py is not part of GNU Mailman. I think the latest
> version is yours, but in any case you can continue to use whatever
> you're using.
>

​Yeah, I know.  That list is an extract of the lists I have modified.  I
just copied and pasted it into the message.​



>
> For the others, if you want to look at changes in advance, go to
> <
> http://bazaar.launchpad.net/~mailman-coders/mailman/2.1/revision/944?remember=1484&compare_revid=1484
> >
> to compare rev 944 (2.1.9) with rev 1484 (2.1.18-1), scroll down to the
> list where file names are on the left preceded with > and click the > on
> the ones you're interested in to see the diff
>
>
>
> > I will also most likely do the same thing I did when I did the original
> > mods to Version.py and versions.py.  I will bump the DATA_FILE_VERSION
> > by one and make only the minimal change needed.
>
>
> in 2.1.18-1, DATA_FILE_VERSION is already 104. The only reason to
> increment DATA_FILE_VERSION is so versions.Update() will run when an
> older list in first instantiated. Since your changes were made at
> DATA_FILE_VERSION = 97, there's no point in incrementing
> DATA_FILE_VERSION beyond 104.
>

​Hmmm.  I have already changed it to 105, restarted Mailman, and created a
couple of lists to test.  Is there any harm with deleting the lists,
backing out the change, and restarting Mailman?  If I can do that, I can
take the file off the list of changed files since the only change is the
DATA_FILE_VERSION number.  I haven't migrated the real lists over yet so
there is no damage there.​



>
>
> > Actually, here are the
> > two patch files I plan on using for those two files.  So I will make
> > copies of the original files (for all files to be modified) and for
>
> ​​


>
> This was the wrong place to add that code. This section of versions.py
> is intended to transform old attributes from say Mailman 2.0 so that
> they have the equivalent semantics in the current version.
>
> The proper way to add a new attribute is in the NewVars(l) function at
> the end around line 430 add
>
> add_only_if_missing('accept_special_posters', [])
>
> ​Cool.  I will redo the change to do it the proper way.  Thanks for
information.

Mark, you have been awesome with your assistance not only to me but all the
other users of Mailman.

Thank you for your excellent support.

Chris
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Small issue (hopefully) with upgrading/migrating from v2.1.9 to v2.1.18-1

[Chris Nulk]

On Wed, Jan 14, 2015 at 6:33 PM, Mark Sapiro  wrote:

> On 01/14/2015 04:11 PM, Chris Nulk wrote:
> >
> > I have run into a small problem.  One of the customizations I did was to
>
> ​.​

>
> >
> > I think that addtional check is preventing my additional
> > attribute/property from accepting lists in the field.
>
>
> Correct.
>

​Thanks for input.  From the last time I did mods to the source​, you
helped me with a few changes.  Specifically, v2.1.9 didn't allow lists in
those fields.  You sent me a patch that backported that change from
v2.1.10.  When I did my checking, I saw that I no longer needed to do the
backport so the GUIBase.py file was dropped from needing the modification.
I am trying to make the minimal changes needed.  Looks like I will have to
add the file back.  Not a problem.


>
> > Before I go off and possibly muck my installation up, can I simply
> > duplicate the 'elif' section with the property.endswith code and replace
> > the '_these_nonmembers' with my attribute/property?
> >
> > Or is a better solution to do something list:
> >  elif (wtype == mm_cfg.EmailListEx and
> > addr.startswith('@')
> > and (property.endswith('_these_nonmembers')
> > or (property.endswith('my_attribute'))):
> > Would that work?
> >
> > Or even more optimially, I am open to the correct solution to my issue.
>
>
> I would replace the elif clause with
>
>  elif (wtype == mm_cfg.EmailListEx and addr.startswith('@') and
>  (property.endswith('_these_nonmembers') or
>   property == 'my_attribute')
>
> indented to the proper level.
>

​Cool.  I will make the change like you recommend.


>
> DRY
>

​DRY?

Thanks Mark.  You are an awesome help.

Chris

P.S. Using GMail to reply just really * (better not say it).   Who
thought up this P o* P** interface?
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

[Mailman-Users] Small issue (hopefully) with upgrading/migrating from v2.1.9 to v2.1.18-1

[Chris Nulk]

Hello,

In a previous message, I mentioned my plans on upgrading/migrating a 
customized v2.1.9 installation to a customized v2.1.18-1 installation.  
I have analyzed the changes needed and made to the v2.1.18-1 installation.


I have run into a small problem.  One of the customizations I did was to 
added an additional attribute to the sender filters similar to the 
*_these_nonmembers fields.  On the v2.1.9 installation, my customization 
will accept individual email addresses, regex's, and lists.  On the 
v2.1.18-1 installation, it only accepts the individual email addresses.  
I took a closer look at the code (GUIBase.py in the Gui directory) and I 
believe the solution lies there.


The code on the v2.1.18-1 installation has some additional checks. 
Specifically, the following:

# See if this is a context that accepts regular
# expressions, and that the re is legal
if wtype == mm_cfg.EmailListEx and 
addr.startswith('^'):

try:
re.compile(addr)
except re.error:
bad_addrs.append(addr)
elif (wtype == mm_cfg.EmailListEx and 
addr.startswith('@')

and property.endswith('_these_nonmembers')):
# XXX Needs to be reviewed for list@domain names.
# don't reference your own list
if addr[1:] == mlist.internal_name():
bad_addrs.append(addr)
# check for existence of list?  For now allow
# reference to list before creating it.
else:
bad_addrs.append(addr)

On v2.1.9, the code did not check for the 'and 
property.endswith('_these_nonmembers')):'.


I think that addtional check is preventing my additional 
attribute/property from accepting lists in the field.


Before I go off and possibly muck my installation up, can I simply 
duplicate the 'elif' section with the property.endswith code and replace 
the '_these_nonmembers' with my attribute/property?


Or is a better solution to do something list:
 elif (wtype == mm_cfg.EmailListEx and 
addr.startswith('@')
and (property.endswith('_these_nonmembers') 
or (property.endswith('my_attribute'))):

Would that work?

Or even more optimially, I am open to the correct solution to my issue.

Thanks for any assistance,
Chris


--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Upgrading/migrating Mailman v2.1.9 -> v2.1.18-1

[Chris Nulk]

On 1/14/2015 7:57 AM, Mark Sapiro wrote:

Thanks for your help Mark.


On 01/13/2015 09:04 PM, Chris Nulk wrote:

Are there any issues I should watch for using the following process?

...

Does the process seem reasonable and workable?


It looks OK to me.

Great.

The question about customizations is how different is the code between
v2.1.9 and 2.1.18-1?


Many modules are unchanged. Some have significant changes. Some have
minor changes.


Below is a list of the files I have modified (base directory is 
/usr/lib/mailman).  Should I pay special attention due to changes to any 
particular files?


bin/add_members
bin/non_members
Mailman/LDAPMemberships.py
Mailman/ListAdmin.py
Mailman/MailList.py
Mailman/Version.py
Mailman/versions.py
Mailman/Cgi/listinfo.py
Mailman/Cgi/options.py
Mailman/Gui/GUIBase.py
Mailman/Gui/Privacy.py
Mailman/Handlers/Decorate.py
Mailman/Handlers/Moderate.py
Mailman/Queue/IncomingRunner.py






If I create patches with the difference between the original v2.1.9 code
and my customizations using diff -Nu, will I be able to apply those patches
to the v2.1.18-1 code?


That's the way to go. See the recent post at
<https://mail.python.org/pipermail/mailman-users/2015-January/078261.html>
for more on this.

Yep.  Read the message.  Thanks for the reminder.



Naturally, I will make copies of any files that will be changed before
applying the patches.  Also, since I do add additional attributes to the
list configs, I will be very careful with changes to versions.py and
Version.py.


Your main concern is with DATA_FILE_VERSION in Versions.py. As long as
you apply the changes before moving the lists You just need to be sure
that DATA_FILE_VERSION is > than that of the old installation. Then
versions.py will be run to update the lists when they are moved.

As far as additional attributes are concerned, you could ignore the
whole issue. I wouldn't if for no other reasons than documentation and
consistency, but your lists will presumably have all the additional
attributes already so the code in versions.py to add them (if missing)
won't do any harm, but won't actually be needed. As I said though, I
would ensure the code was there just for documentation and consistency
and in case at some point you migrate a list that doesn't have the
attributes.

I will definitely make the mods prior to moving the lists since I will 
want to make sure Mailman is still functional.


I will also most likely do the same thing I did when I did the original 
mods to Version.py and versions.py.  I will bump the DATA_FILE_VERSION 
by one and make only the minimal change needed. Actually, here are the 
two patch files I plan on using for those two files.  So I will make 
copies of the original files (for all files to be modified) and for 
Version.py I will check for the current DATA_FILE_VERSION number and 
adjust appropriately.


Patch for Version.py
---
--- Version.py.original2008-05-24 13:44:12.0 -0700
+++ Version.py.Modified2009-09-25 10:47:02.0 -0700
@@ -37,7 +37,9 @@
(REL_LEVEL << 4)  | (REL_SERIAL << 0))

 # config.pck schema version number
-DATA_FILE_VERSION = 96
+# Updated the DATA_FILE_VERSION due to adding a new
+#variable/attribute (previous value = 96) --Modified 24-Sep-2009
+DATA_FILE_VERSION = 97

 # qfile/*.db schema version number
 QFILE_SCHEMA_VERSION = 3

Patch file for versions.py
-
--- versions.py.original2008-05-24 13:44:12.0 -0700
+++ versions.py.Modified2009-09-24 14:15:11.0 -0700
@@ -113,6 +113,11 @@
 l.forward_auto_discards = mm_cfg.DEFAULT_FORWARD_AUTO_DISCARDS
 if not hasattr(l, 'generic_nonmember_action'):
 l.generic_nonmember_action = 
mm_cfg.DEFAULT_GENERIC_NONMEMBER_ACTION

+
+# Add an additional list attribute if not present --Modified 
24-Sep-2009

+if not hasattr(l, 'accept_special_posters'):
+l.accept_special_posters = []
+
 # Now convert what we can...  Note that the interaction between the
 # MM2.0.x attributes `moderated', `member_posting_only', and 
`posters' is
 # so confusing, it makes my brain really ache.  Which is why they 
go away



Thanks again for the assistance Mark,
Chris
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

[Mailman-Users] Upgrading/migrating Mailman v2.1.9 -> v2.1.18-1

[Chris Nulk]

Hello all,

I will be upgrading/migrating from a v2.1.9 Mailman installation to a
v2.1.18-1 Mailman installation.  In addition, I will be upgrading the
hardware and will be keeping the name of the system the same.  I have a few
questions if anyone can help.

I will be keeping the current installation (v2.1.9) running while I build
the new installation (v2.1.18-1).

Are there any issues I should watch for using the following process?

Assuming the current system is named 'lists.example.com' and the new system
is named 'newlists.example.com' (temporarily)

- Build the new installation (v2.1.18-1), configure and test it
- Migrate all the list configs and list archives from the current system to
the new system
- run check_perms -f (just to be safe)
- run fix_url on the new system (probably not necessary but I want to make
sure the lists are working on the new system).
- test several of the key lists (receiving messages, sending messages to
members, ability to make config changes)
- change the current system from 'lists.example.com' to '
oldlists.example.com', change/update system name/DNS/firewall entries, run
check_perms and fix_url, and restart Mailman
- change the new system from 'newlists.example.com' to 'lists.example.com',
change/update system name/DNS/firewall entries, run check_perms and
fix_url, and restart Mailman
- test several of the key lists for proper operation

At a first pass, I think I have covered most (hopefully, all) potential
issues.  The process should allow me make sure the new installation is
working before the lists are moved, after the lists are moved but before
the system name is changed, and after the system named is change.  The
result should be a new upgraded and migrated Mailman installation.

Does the process seem reasonable and workable?

The next issue that I have to mix in are customizations to Mailman.  I plan
on adding the customizations back in between the first two steps above with
additional testing of course.

The question about customizations is how different is the code between
v2.1.9 and 2.1.18-1?
If I create patches with the difference between the original v2.1.9 code
and my customizations using diff -Nu, will I be able to apply those patches
to the v2.1.18-1 code?

Naturally, I will make copies of any files that will be changed before
applying the patches.  Also, since I do add additional attributes to the
list configs, I will be very careful with changes to versions.py and
Version.py.


Any help is appreciated.

Thanks,
Chris
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Writing a custom handler

[Chris Nulk]

On 7/3/2013 9:44 PM, Mark Sapiro wrote:

On 07/03/2013 07:40 AM, Chris Nulk wrote:

Okay.  Makes sense.  Here is the modified do_discard_globalban code:

def do_discard_globalban(mlist, msg, sender):
 # forward discarded message to site administrator(s) if defined
 #in mm_cfg.GLOBALBANLIST_NOTIFY
 notifylist = []
 if mm_cfg.GLOBALBANLIST_NOTIFY:
 notifylist.append(mm_cfg.GLOBALBANLIST_NOTIFY)


You might consider making mm_cfg.GLOBALBANLIST_NOTIFY a list of 1 or
more addresses in which case the above would be

 if mm_cfg.GLOBALBANLIST_NOTIFY:
 notifylist.extend(mm_cfg.GLOBALBANLIST_NOTIFY)

or you could get fancy and test if it's a list or a string and extend or
append accordingly.



Sorry about not responding earlier, I was on vacation.  That is exactly 
what I intended.  I didn't see extend when I was looking up information 
on lists.  Thanks.


I was thinking about what you mentioned in an earlier message about this 
handler not catching the command requests like subscribe, unsubscribe, 
etc.  I think I have a possible solution but I want to think about it 
some more then ask about it.


Thanks for the help on the handler,
Chris
--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Writing a custom handler

[Chris Nulk]

On 7/2/2013 5:23 PM, Mark Sapiro wrote:

On 07/02/2013 03:09 PM, Chris Nulk wrote:

In the updated code, I did change the populating of the banlist in the
Read_GlobalBan_File function.  Now, it strips and lowercases the
addresses before it checks if the address is in the banlist. Before, it
checked then populated a stripped, lowercase version. This could have
resulted in duplicates if the address in the banlist but the address
checked had a different case profile.



 # Go through possible senders.  Check if any of them are
 #   on the global ban list
 for sender in msg.get_senders():
 if sender.lower() in banlist:


Not necessary to lowercase sender here as msg.get_senders() always
returns lowercased addresses unless called with a preserve_case argument
with a True value.


Thanks.  It was more of a just to make sure thing.

Actually, I code I was referring to was in Read_GlobalBan_File:

def Read_GlobalBan_File(ban_file):
try:
with open(ban_file) as f:
for addr in f:
# strip and lowercase addr
addr = addr.lower().strip()
# if addr is not in banlist, add it - to avoid 
duplicates, should not add blank addresses

if (addr) and (addr not in banlist):
banlist.append(addr)

In previous versions, lower() and strip() were in the banlist.append not 
where they are now.  That would have allowed the possibility of 
duplicates if the same addr was presented with different cases (i.e. 
j...@example.com and j...@example.com)



# copied almost verbatim from Mailman/Handlers/Moderate.py
def do_discard_globalban(mlist, msg, sender):
 # forward auto-discards to list owners?
 if mlist.forward_auto_discards:


Note that no forwarding of the discard occurs unless the list's Privacy
options... -> Sender filters -> forward_auto_discards is set to Yes.
Otherwise, no notice is sent and the only indication is a "Message
discarded, msgid: ..." entry in the 'vette' log.

You may wish to remove the above condition.


I will let the list-owners decide if they want the messages. Granted, it 
will be all or nothing for the discards.



 lang = mlist.preferred_language
 nmsg = Message.UserNotification(mlist.GetOwnerEmail(),


mlist.GetOwnerEmail() returns the LISTNAME-owner address which is the
recipient of this mail. This argument can be a string with a single
address as it's value as it is here or a list of strings, each of which
is an address. So the above line could be changed to

 nmsg = Message.UserNotification([
  mlist.GetOwnerEmail(),
  'y...@example.com',
  ],

with whatever list of addresses you want, or maybe since list owners
aren't much involved with this ban list

 nmsg = Message.UserNotification('y...@example.com',



Okay.  Makes sense.  Here is the modified do_discard_globalban code:

def do_discard_globalban(mlist, msg, sender):
# forward discarded message to site administrator(s) if defined
#in mm_cfg.GLOBALBANLIST_NOTIFY
notifylist = []
if mm_cfg.GLOBALBANLIST_NOTIFY:
notifylist.append(mm_cfg.GLOBALBANLIST_NOTIFY)

# forward auto-discards to list owners?
if mlist.forward_auto_discards:
notifylist.append(mlist.GetOwnerEmail())

# Send message if notifylist is not empty
if notifylist:
lang = mlist.preferred_language
nmsg = Message.UserNotification(notifylist,
mlist.GetBouncesEmail(),
_('Global Ban List Auto-discard 
notification'),

lang=lang)
nmsg.set_type('multipart/mixed')
text = MIMEText(Utils.wrap(_("""\
The sender - %(sender)s - of the attached message is on the Global Ban 
list.  Therefore, the message

has been automatically discarded.""")),
_charset=Utils.GetCharSet(lang))
nmsg.attach(text)
nmsg.attach(MIMEMessage(msg))
nmsg.send(mlist)
# Discard the message
raise Errors.DiscardMessage


Thanks again for the help,
Chris
--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Writing a custom handler

[Chris Nulk]

On 7/2/2013 2:37 PM, Mark Sapiro wrote:

If you raise some exception other than the Mailman.Errors exceptions
DiscardMessage, HoldMessage or RejectMessage, it will be the same as if
you didn't catch the exception, i.e., the exception will be logged in
the 'error' log with a traceback and the message will be shunted.

I can see that you don't really want that because if your ban file is
unreadable or you can't stat it for some reason, you'd be shunting every
message for a reason that has really nothing to do with processing the
message. So, I think you do want to catch the exception, log it and
return as that is the only way to continue processing messages.

Just watch your 'error' log as you'll get an entry per message if
something is wrong.



Thank you for the help.   Hopefully a last question.  In the 
do_discard_globalban function, how can I send the discard message to 
mailman list?  Or should I?  As it stands now, each list owner/admin 
would be getting the message for their list.  I am thinking that as the 
site administrator and maintainer of the global ban list, I should also 
get the message.


In the updated code, I did change the populating of the banlist in the 
Read_GlobalBan_File function.  Now, it strips and lowercases the 
addresses before it checks if the address is in the banlist. Before, it 
checked then populated a stripped, lowercase version. This could have 
resulted in duplicates if the address in the banlist but the address 
checked had a different case profile.


Thanks,
Chris

  Updated Global Ban list Custom Handler 



#!/usr/bin/env python
#

"""This is a custom handler that will check all the sender addresses of
a message against a global ban list.  If any of the sender addresses are
on the global ban list, the message will get logged and discarded.
"""

import sys
import os

from Mailman import mm_cfg
from Mailman import Utils
from Mailman import Message
from Mailman import Errors
from Mailman.i18n import _
from Mailman.Logging.Syslog import syslog

# Global variables
#   Initialize the banlist
banlist = []

#   Keep the Global Ban lists modification time
# set to -1 to indicate ban file hasn't been read
ban_mtime = -1

# Define ban_file
#   mm_cfg.GLOBALBANLIST_FILENAME is defined in mm_cfg and should
#   be the full path to the file.
ban_file = mm_cfg.GLOBALBANLIST_FILENAME

def process(mlist, msg, msgdata):
# Upstream pipeline handler marked message approved -
#   respect the decision or message has 'Approved: ' header
if msgdata.get('approved'):
return

# ban_file gets its value from mm_cfg.GLOBALBANLIST_FILENAME. If
#   mm_cfg.GLOBALBANLIST_FILENAME is not defined, then neither is
#   ban_file, so simply return.
if not ban_file:
return

# Read in the global ban list of email addresses
if Ban_File_Changed(ban_file, ban_mtime):
# Global Ban list has changed (or ban_mtime = -1),
#   read in the changes
if not Read_GlobalBan_File(ban_file):
# Problems reading the GlobalBan list
return

# Check if banlist has anything, if not, no need to go further
if not banlist:
return

# Go through possible senders.  Check if any of them are
#   on the global ban list
for sender in msg.get_senders():
if sender.lower() in banlist:
break
else:
# None of the sender addresses were in the global ban
#   list so return and continue with the next pipeline
#   handler
return

# A sender was found on the global ban list.  Log it and
#   discard the message notifying the list owner
if sender:
# Log banned sender to the vette log
syslog('vette', '%s is banned by the global ban list', sender)
# Perform message discard
do_discard_globalban(mlist, msg, sender)
else:
assert 0, 'Bad sender in GlobalBan.py'


# Stat the ban file to get the modification time and compare it to the
#   last time the file was changed.  If a changed occured, update
#   ban_mtime to current change time
def Ban_File_Changed(ban_file, ban_mtime):
try:
statinfo = os.stat(ban_file)
except IOError, e:
# cannot stat the global ban list for whatever reason
# log it and continue with the next pipeline handler
syslog('error',
   "Can't stat %s: %s" % (ban_file, e)
   )
return False
except:
# unspecified error
# log it and continue with the next pipeline handler
syslog('error',
   'ERROR: %s: %s' % (sys.exc_info()[0], sys.exc_info()[1])
  )
return False

# if ban_mtime = -1, statinfo.st_mtime should always be greater, this
#   is a special case for when the code is first loaded and run
if statinfo.st_mtime > ban_mtime:
# ban_file has changed or it's the special case
ban_mtime = statinfo.st_mtime

Re: [Mailman-Users] Writing a custom handler

[Chris Nulk]

On 7/2/2013 12:17 PM, Mark Sapiro wrote:

On 07/02/2013 11:38 AM, Chris Nulk wrote:


I did forget about some of my other questions.  I plan on writing
another custom handler for a list-specific issue.  Where would I look if
I wanted to intercept messages related to subscribing, unsubscribing,
and options processing?


These are messages to the -request, -join, -subscribe, -leave and
-unsubscribe addresses and are all processed by CommandRunner. Only
IncomingRunner processes a pipeline of handlers. There is no comparable
processing or custom handler option for CommandRunner. You would have to
modify Mailman/Queue/CommandRunner.py itself.


Hmmm.  I have a few lists where unsubscribes are not allowed (not my 
decision).  On one of them, a persistent member keeps trying to 
unsubscribe.  I wanted to setup an automated response and discard mechanism.






How would I tell the difference between a
subscribe message and an unsubscribe message? Also, is there a
difference if the person does the subscribing or unsubscribing via the
web?  Can I trap those?


The web functions are handled by modules in Mailman/Cgi. They don't
involve qrunners at all.


I have made enough mods to Mailman.  Don't want make any more if I don't 
have to so I will just keep discarding the unsubscribe requests.



Below is the latest update incorporating your suggestions.


See some inline comments.


def process(mlist, msg, msgdata):
 # Upstream pipeline handler marked message approved -
Or message contained an Approved:  header.


Yep.




 #   respect the decision
 if msgdata.get('approved'):
 return

 # ban_file gets its value from mm_cfg.GLOBALBANLIST_FILENAME. If
 #   mm_cfg.GLOBALBANLIST_FILENAME is not defined neither is
 #   ban_file, so simply return.
 if not ban_file:
 return

 # Read in the global ban list of email addresses
 if Ban_File_Changed(ban_file, ban_mlist):


ban_mtime, not ban_mlist.


Yeah, I fixed it after I sent the message.





 # Global Ban list has changed (or ban_mlist = -1),


ban_mtime.


Same here.





 #   read in the changes
 rc = Read_GlobalBan_File(ban_file)
 if not rc:
 # Problems reading the GlobalBan list
 return


Or just
 if not Read_GlobalBan_File(ban_file):
 return


I actually had more code here.  As I reduced the code, I tightened the 
code.  Missed this one.



# Stat the ban file to get the modification time and compare it to the
#   last time the file was changed.  If a changed occured, update
#   ban_mtime to current change time
def Ban_File_Changed(ban_file, ban_mtime):
 try:
 statinfo = os.stat(ban_file)
 except IOError, e:
 # cannot stat the global ban list for whatever reason
 # log it and continue with the next pipeline handler
 syslog('error',
"Can't stat %s: %s" % (ban_file, e)
)
 return False
 except:
 # unspecified error
 # log it and continue with the next pipeline handler
 syslog('error',
'ERROR: %s: %s' % (sys.exc_info()[0], sys.exc_info()[1])
   )
 return False


Do you really want to do this? The Mailman philosophy is all caught and
handled exceptions should be explicit. An unanticipated exception will
be caught at the top qrunner level which will log the error with
traceback in the 'error' log and shunt the message rather than
continuing processing with a message that triggered an unanticipated
exception.


Instead of returning, should I raise an exception?  For the IOError or 
any error, whether or not the file can be read or perform a stat isn't a 
big deal.  I would rather log the error, clear the exception and keep 
going.  If there is a problem in this function, returning False just 
means the ban_file hasn't changed so I will still be running sender 
addresses against a potentially older banlist.


# Read the Global Ban file and populate the banlist.
def Read_GlobalBan_File(ban_file):
 try:
 with open(ban_file) as f:
 for addr in f:
 # if addr is not in banlist, add it - to avoid duplicates
 if addr not in banlist:
 banlist.append(addr.lower().strip())
 except IOError, e:
 # cannot open the global ban list for whatever reason
 # log it and continue with the next pipeline handler
 syslog('error',
"Can't open %s: %s" % (ban_file, e)
)
 return False
 except:
 # unspecified error
 # log it and continue with the next pipeline handler
 syslog('error',
'ERROR: %s: %s' % (sys.exc_info()[0], sys.exc_info()[1])
   )
 return False


See comment above about generic

Re: [Mailman-Users] Writing a custom handler

[Chris Nulk]

On 7/2/2013 10:16 AM, Mark Sapiro wrote:

On 07/02/2013 08:48 AM, Chris Nulk wrote:


You could change your code as follows:

1) Make the banlist global by putting

# First, initialize the banlist
banlist = []

ahead of

def process(mlist, msg, msgdata):

and remove that from the process() definition:

Then make the code inside process() that loads the banlist conditional
on banlist being the empty list, e.g.,

 if not banlist:
 ...

Then once the global banlist is loaded with the file data, the file
won't be read again.



What I would like to do is read the global ban list file once to build
the ban list but update the ban list if there has been a change in the
global ban list file.


You have a couple of choices:

1) Do the above and just restart Mailman if you update the banlist.

2) add code to test and remember (in a global) the mod time of the
banlist file and read it only if the file is newer.

Method 2) may involve comparable overhead to just reading the file each
time so may not be worth it. I would either do 1) or just read the file
every time based on how often I expect the file to change.

Note that if posts are infrequent, the overhead of reading the file each
time doesn't much matter, and if posts are frequent, the file will
probably be in an OS disk cache anyway.


Mark,

Thanks again for all the help.  I choose method 2.  I have a few 
processes that monitor the log files.  I can modify them to 
automatically update the ban list file based on whatever parameters I 
want to set.  With method 2, my Mailman instance is updated on the next 
message that comes in without me having to do anything.  Our site has 
some activity but not a lot as some places do so I am not worried about 
the overhead.


I did forget about some of my other questions.  I plan on writing 
another custom handler for a list-specific issue.  Where would I look if 
I wanted to intercept messages related to subscribing, unsubscribing, 
and options processing?  How would I tell the difference between a 
subscribe message and an unsubscribe message? Also, is there a 
difference if the person does the subscribing or unsubscribing via the 
web?  Can I trap those?


Below is the latest update incorporating your suggestions.

Thanks again,
Chris

 Updated Global Ban Custom Handler 
--


#!/usr/bin/env python
#

"""This is a custom handler that will check all the sender addresses of
a message against a global ban list.  If any of the sender addresses are
on the global ban list, the message will get logged and discarded.
"""

import sys
import os

from Mailman import mm_cfg
from Mailman import Utils
from Mailman import Message
from Mailman import Errors
from Mailman.i18n import _
from Mailman.Logging.Syslog import syslog

# Global variables
#   Initialize the banlist
banlist = []

#   Keep the Global Ban lists modification time
# set to -1 to indicate ban file hasn't been read
ban_mtime = -1

# Define ban_file
#   mm_cfg.GLOBALBANLIST_FILENAME is defined in mm_cfg and should
#   be the full path to the file.
ban_file = mm_cfg.GLOBALBANLIST_FILENAME

def process(mlist, msg, msgdata):
# Upstream pipeline handler marked message approved -
#   respect the decision
if msgdata.get('approved'):
return

# ban_file gets its value from mm_cfg.GLOBALBANLIST_FILENAME. If
#   mm_cfg.GLOBALBANLIST_FILENAME is not defined neither is
#   ban_file, so simply return.
if not ban_file:
return

# Read in the global ban list of email addresses
if Ban_File_Changed(ban_file, ban_mlist):
# Global Ban list has changed (or ban_mlist = -1),
#   read in the changes
rc = Read_GlobalBan_File(ban_file)
if not rc:
# Problems reading the GlobalBan list
return

# Check if banlist has anything, if not, no need to go further
if not banlist:
return

# Go through possible senders.  Check if any of them are
#   on the global ban list
for sender in msg.get_senders():
if sender.lower() in banlist:
break
else:
# None of the sender addresses were in the global ban
#   list so return and continue with the next pipeline
#   handler
return

# A sender was found on the global ban list.  Log it and
#   discard the message notifying the list owner
if sender:
# Log banned sender to the vette log
syslog('vette', '%s is banned by the global ban list', sender)
# Perform message discard
do_discard_globalban(mlist, msg, sender)
else:
assert 0, 'Bad sender in GlobalBan.py'


# Stat the ban file to get the modification time and compare it to the
#   last time the file was changed.  If a changed occured, update
#   ban_mtime to current change time
def Ban_File_Changed(ban_file, ban_mtime):
try:

Re: [Mailman-Users] Writing a custom handler

[Chris Nulk]

Thank you Mark for your time.

I went through and made the syslog change, removed the usenet bit, and 
correct my typing error.


On 7/1/2013 3:35 PM, Mark Sapiro wrote:

On 7/1/2013 10:24 AM, Chris Nulk wrote:

Hello user,

I am writing a custom handler to globally ban email address from sending
messages sent to Mailman.  I know I can use Mark's add_banned.py script
to add an address to all lists.  However, if I add an address to be
banned, as the administrator for all lists, I don't want a list admin to
remove a pest from their list(s).  I banned an address for being a pest
to all lists (or a majority of them), therefore, the address stays banned.


Note that the ban_list only prevents the address from subscribing. If
the address was already a member when banned, it can still post.


I know the ban_list only prevents the address from subscribing.  My goal 
for this custom handler is to stop a pesky addresses from subscribing, 
unsubscribing, posting, or having any interaction with any of my lists.  
I have add a few addresses try to subscribe to all the lists on my 
site.  I highly doubt those are all legit so I want to ban and have 
nothing to do with those addresses in the future.



from Mailman import Errors
from Mailman.i18n import _
from Mailman.Logging.Syslog import syslog
from Mailman.MailList import MailList

You don't actually use MailList so there's no need to import it. Also
Utils, Message, Errors and _ are used only by the discard code you
copied. You could instead just

 From Mailman.Handlers.Moderate import do_discard

and use that if it wasn't important to have specific messages. OTOH, see
below.


I removed the MailList line.  As for the do_discard, I actually was 
going to import it, however, I decided I wanted to have a different 
discard message indicating the global ban list caused the discard. Then, 
the list owners would know it wasn't something with their list 
configuration.




 # Go through possible senders.  Check if any of them are
 #   on the global ban list
 for sender in msg.get_senders():
 if sender.lower() in banlist:
 break
 else:
 # None of the sender addresses were in the global ban
 #   list so return and continue with the next pipeline
 #   handler
 return

 # A sender was found on the global ban list.  Log it and
 #   discard the message notifying the list owner
 if sender:


How can this be False?


If I understand the code properly, sender should never be False or None 
but if something happens that I do not expect, I have a way to catch 
it.  Or, am I missing something?


You could just import do_discard from Mailman.Handlers.Moderate and 
use that, but you may want the custom messages. If so, you may also 
want do_discard_globalban(mlist, msg, sender)


# copied almost verbatim from Mailman/Handlers/Moderate.py
def do_discard_globalban(mlist, msg):
 sender = msg.get_sender


I take it that should I change the line -

do_discard_globalban(mlist, msg, sender)

I should also update the def for it to -

def do_discard_globalban(mlist, msg, sender)

and I can remove the line -

sender = msg.get_sender()


I have a few more questions.  My plan for the custom handler is install 
it as the first handler in the global pipeline for all lists.  I have 
seen how to do that in the faqs.   I also believe that once it is 
running with Mailman, it will be in memory and executable - no loading 
and unloading of the handler.  Once Mailman loads, it loads all the 
handlers and is ready to go.


My handler reads a file every time it runs.  Not very efficient. Once I 
have read in the banlist from the file, will the banlist always be 
available for future iterations through the code?


What I would like to do is read the global ban list file once to build 
the ban list but update the ban list if there has been a change in the 
global ban list file.


Thanks for any assistance,

Chris


--- Updated Global Ban list custom handler 



#!/usr/bin/env python
#

"""This is a custom handler that will check all the sender addresses of
a message against a global ban list.  If any of the sender addresses are
on the global ban list, the message will get logged and discarded.
"""

import sys

from Mailman import mm_cfg
from Mailman import Utils
from Mailman import Message
from Mailman import Errors
from Mailman.i18n import _
from Mailman.Logging.Syslog import syslog

def process(mlist, msg, msgdata):
# added because it was in Mailman/Handlers/Moderate.py
#   I am guessing it has to due in part with an upstream
#   pipeline handler marking the message approved and/or
#   because the handler can be moved to different parts
#   of the pipeline.
if msgdata.get('approved'):
return

# First, initialize the banlist
banlist = []

# Read in the glob

[Mailman-Users] Writing a custom handler

[Chris Nulk]

Hello user,

I am writing a custom handler to globally ban email address from sending 
messages sent to Mailman.  I know I can use Mark's add_banned.py script 
to add an address to all lists.  However, if I add an address to be 
banned, as the administrator for all lists, I don't want a list admin to 
remove a pest from their list(s).  I banned an address for being a pest 
to all lists (or a majority of them), therefore, the address stays banned.


Before I put the custom handler in place and screw up my lists, I 
thought I would post it here so others more knowledgeable can review it 
and let me know if it will work, correct it, and/or improve it.


Thanks,
Chris

--- Custom Handler Below 
---


#!/usr/bin/env python
#

"""This is a custom handler that will check all the sender addresses of
a message against a global ban list.  If any of the sender addresses are
on the global ban list, the message will get logged and discarded.
"""

import sys

from Mailman import mm_cfg
from Mailman import Utils
from Mailman import Message
from Mailman import Errors
from Mailman.i18n import _
from Mailman.Logging.Syslog import syslog
from Mailman.MailList import MailList

def process(mlist, msg, msgdata):
# added because it was in Mailman/Handlers/Moderate.py
#   I am guessing it has to due in part with an upstream
#   pipeline handler marking the message approved and/or
#   because the handler can be moved to different parts
#   of the pipeline.
#   But, I have been wrong before.
if msgdata.get('approved') or msgdata.get('fromusenet'):
return

# First, initialize the banlist
banlist = []

# Read in the global ban list of email addresses
#  mm_cfg.GLOBALBANLIST_FILENAME is defined in mm_cfg and should
#  be the full path to the file.
try:
with open(mm_cfg.GLOBALBANLIST_FILENAME) as f:
for addr in f:
banlist.append(addr.lower().strip())
except IOError:
# cannot open the global ban list for whatever reason
# log it and continue with the next pipeline handler
syslog('error', 'An error occurred opening the global ban list')
return
except:
# unspecified error
# log it and continue with the next pipeline handler
syslog('error', "ERROR: Unknown error: ", sys.exc_info()[0])
return

# Go through possible senders.  Check if any of them are
#   on the global ban list
for sender in msg.get_senders():
if sender.lower() in banlist:
break
else:
# None of the sender addresses were in the global ban
#   list so return and continue with the next pipeline
#   handler
return

# A sender was found on the global ban list.  Log it and
#   discard the message notifying the list owner
if sender:
# Log banned sender to the vette log
syslog('vette', '%s is banned by the global ban list', sender)
# Perform message discard
do_discard_globalban(mlist, msg)
else:
assert 0, 'Bad sender in GlobalBan.py'


# copied almost verbatim from Mailman/Handlers/Moderate.py
def do_discard_globalban(mlist, msg):
sender = msg.get_sender
# forward auto-discards to list owners?
if mlist.forward_auto_discards:
lang = mlist.preferred_language
# is varhelp used anywhere?
varhelp = '%s/?VARHELP=privacy/sender/discard_these_nonmembers' % \
  mlist.GetScriptURL('admin', absolute=1)
nmsg = Message.UserNotification(mlist.GetOwnerEmail(),
mlist.GetBouncesEmail(),
_('Global Ban List Auto-discard 
notification'),

lang=lang)
nmsg.set_type('multipart/mixed')
text = MIMEText(Utils.wrap(_("""\
The sender of the attached message is on the Global Ban list. Therefore, 
the message

has been automatically discarded.""")),
_charset=Utils.GetCharSet(lang))
nmsg.attach(text)
nmsg.attach(MIMEMessage(msg))
nmsg.send(mlist)
# Discard the message
raise Errors.DiscardMessage

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Problems with Mailman LDAPMemberships Adapter

[Chris Nulk]

Hi Mark,

I don't know about others but I still use the LDAP adapter here. And I 
still appreciate the assistance you gave to get it working.


Thanks again,
Chris

On 6/5/2013 5:47 PM, Mark Sapiro wrote:

On 06/05/2013 08:29 AM, Lennart Barfod wrote:

Hi Mark,

I'm using the LDAPMembership.py Version 0.63.

Yes I restartet mailman some times after adding the extend.py.

Ok, so I should reinstall mailman and Python-ldap? Or where seems to be the 
problem?


There seems to be a serious coding error in LDAPMemberships.py, version
0.63. Namely, at line 274-275, the code

 def __ldap_get_members(self):
 return self.__ldap_get_regularmembers() +
self.__ldap_get_digestmembers()

(watch out for wrapped line)

should be

 def __ldap_get_members(self):
 return self.__ldap_get_regular_members() +
self.__ldap_get_digest_members()


I wonder how many people are actually using this?

If you fix that and restart Mailman, I think you'll be OK.



--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Subject line message filtering

[Chris Nulk]

On 2/11/2013 6:40 PM, Mark Sapiro wrote:

Chris Nulk wrote:

{SNIP}

The only issue I have now is the boss wants to be able to send a reject
message if a message is rejected when it matches a rule.  The only
option I have is to hold the message and let the moderator send the
rejection notice.  And since, the header rule is for our global lists, I
am the lucky moderator.


If you mean you want to reject the message with something other than
the "Message rejected by filter rule match" message, then you are
correct unless you want to modify the message in the SpamDetect.py
module, i.e. change the message in

 raise Errors.RejectMessage(
 _('Message rejected by filter rule match'))

Making it something like:

 raise Errors.RejectMessage(
 _("""Some long winded,
   Multiple line
   Rejection message
   """))

However, I hope these are just legitimate notifications that shouldn't
have been sent to a list and not spam. It is a very bad idea to reject
anything that might be spam because the rejection almost certainly
goes to some innocent 3rd party whose address was spoofed in the
message's headers. This is known as backscatter and is bad.


The messages I am trying to keep from getting to the list are Google 
Calendar invitations (and cancellations).  For these messages or 
specifically targeted non-spam type messages where I have to use the 
spam filters to get rid of them, I would like to send a rejection notice 
giving a reason why the posting is not allowed.  Any spam filters I set 
up for known spam and like messages, I usually just discard the 
message.  I certainly don't want to do any backscatter.


Thanks, Mark.

Chris
--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Subject line message filtering

[Chris Nulk]

On 2/8/2013 4:28 PM, Mark Sapiro wrote:

Chris Nulk wrote:


My regex filter is:

   ^Subject:[\s\+]*\[.*\]\s+(Invitation|Canceled Event):.*


I know you've resolved this and that the issue was that
header_filter_rules are processed way before the subject_prefix is added
so you didn't want the '\[.*\]'part of the pattern, but I'm confused by
the '[\s\+]*' part of the pattern which says match 0 or more occurrences
of the character class consisting of any white space character (\s) and
the plus sign (either \+ or just + would be equivalent here, + loses its
special meaning inside []).

So, is [\s\+]* a typo or did you just mean \s+ or \s*?



The [\s\+]* was deliberate.  Most of our lists but the list name in 
brackets ('[', ']'), however, we have a few lists that are members of 
another list.  For those lists the list subject prefix is a plus-sign 
('+').   For example, we have a student list that has the [Student] 
prefix when the messages go out.  Faculty and staff are not members of 
the list and don't receive those messages.  Yet, there are a few faculty 
and staff who have an interest in seeing the messages so those 
faculty/staff are put on a "listening" list which is a member of the 
student list.  When a message goes to the student list, the prefix 
[Student] is added to the subject line.  The message is then sent to the 
"listening" list and it adds its own subject prefix.  Instead of having 
'[Student-Listen][Student]' show in the subject line, I changed it to a 
'+'.  The long and short of it is that I was trying to match both types 
of lines and the [\s\+]* was the bit for matching the "listening" list's 
subject prefix.






Also, can any of the sender/recipient filters cause the message to bypass the 
spam filters?  The message pipeline is unchanged.


No. header_filter_rules are processed by SpamDetect which is the first
module in the pipeline. As you discovered, the CookHeaders module which
adds the subject_prefix comes much later, and the sender/recipient
filters are processed by Moderate which also comes later.


Great information to know, thanks.

The only issue I have now is the boss wants to be able to send a reject 
message if a message is rejected when it matches a rule.  The only 
option I have is to hold the message and let the moderator send the 
rejection notice.  And since, the header rule is for our global lists, I 
am the lucky moderator.



Thanks for all the great help,
Chris
--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Subject line message filtering

[Chris Nulk]

On 2/8/2013 11:44 AM, Richard Damon wrote:

On 2/8/13 12:52 PM, Chris Nulk wrote:

Hello all,

I am trying to stop a particular group of messages from going to
several of my lists.  I am looking at the Subject line of the message
to do the filtering and I am using the header_filter_rules in Spam
Filtering.  My regex filter is:

  ^Subject:[\s\+]*\[.*\]\s+(Invitation|Canceled Event):.*

and the Subject lines I am trying to catch and reject the message look
like:

  Subject: [MYLIST] Invitation: Test event @ Thu Feb 28,
  2013 10:00 - 11:00 (myli...@example.com)

or

  Subject: [MYLIST] Canceled Event: Test event @ Thu Feb 28,
  2013 10:00 - 11:00 (myl...@example.com)


I have set the action for the rule to reject, however, the messages
still get posted to the list.

I have tested the regex above using the python interactive interpreter
and the regex does match the various inputs used.

Can anyone tell me what the problem is with my regex?

Also, can any of the sender/recipient filters cause the message to
bypass the spam filters?  The message pipeline is unchanged.

Thanks,
Chris

Make sure you have set the "Action" for the filter to some other than
Defer or Accept.


I did set the "Action" to "Reject".



A previous Accept that matches the message will override this filter.


Not a problem.  Yet.  I only have the one filter.



Also, note that I think the filter it tested BEFORE the [MYLIST] is
added to the subject by the list, so is only there to be matched if it
was in the original email.

This was the answer.  Thank you.  I should have realized that but I just 
wanted to get it done so I can move on to other critical work. Taking 
out the [MYLIST] check from the regex worked and I could add back in 
addition constraints to the regex to make it more targeted.


Thanks again Richard.

Chris
--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

[Mailman-Users] Subject line message filtering

[Chris Nulk]

Hello all,

I am trying to stop a particular group of messages from going to several 
of my lists.  I am looking at the Subject line of the message to do the 
filtering and I am using the header_filter_rules in Spam Filtering.  My 
regex filter is:


 ^Subject:[\s\+]*\[.*\]\s+(Invitation|Canceled Event):.*

and the Subject lines I am trying to catch and reject the message look like:

 Subject: [MYLIST] Invitation: Test event @ Thu Feb 28,
 2013 10:00 - 11:00 (myli...@example.com)

or

 Subject: [MYLIST] Canceled Event: Test event @ Thu Feb 28,
 2013 10:00 - 11:00 (myl...@example.com)


I have set the action for the rule to reject, however, the messages still get 
posted to the list.

I have tested the regex above using the python interactive interpreter and the 
regex does match the various inputs used.

Can anyone tell me what the problem is with my regex?

Also, can any of the sender/recipient filters cause the message to bypass the 
spam filters?  The message pipeline is unchanged.

Thanks,
Chris

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

[Mailman-Users] Using set_mod.py

[Chris Nulk]

Hello all,

I am trying to change moderation settings on some lists via a script so 
I can automate the list management for several lists (approx. 30 lists).


In the script I use the following to run set_mod.py:

  /usr/lib/mailman/bin/withlist --run /usr/lib/mailman/bin/set_mod 
LISTNAME --unset addr...@example.com


however, I get the following error back:

 Importing /usr/lib/mailman/bin/set_mod...
 Traceback (most recent call last):
   File "/usr/lib/mailman/bin/withlist", line 297, in ?
 main()
   File "/usr/lib/mailman/bin/withlist", line 266, in main
 mod = __import__(module)
 ImportError: No module named /usr/lib/mailman/bin/set_mod

Just to make sure I am running withlist and set_mod correctly, I did 
change into the mailman directory (/usr/lib/mailman) and run the command 
as recommended in the documentation:


  bin/withlist --run bin/set_mod LISTNAME --unset addr...@example.com

and, I get back the following error:

 Importing bin/set_mod...
 Running bin/set_mod.bin/set_mod()...
 Traceback (most recent call last):
   File "bin/withlist", line 297, in ?
 main()
   File "bin/withlist", line 269, in main
 func = getattr(mod, callable)
 AttributeError: 'module' object has no attribute 'bin/set_mod'

I tried adding the .py at the end of set_mod but I get a different error.

As a matter of fact, things went from bad to worse.  All the different 
trys I did may have corrupted my test list.  I had to delete the list 
and recreate it.


Is there a magic incantation so I can run set_mod.py in a script like my 
first command above?
Is there an easy way to modify set_mod.py so I can run it without using 
withlist (like add_members, remove_members, etc.)?


Thanks for an help and any advice,
Chris
--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Information about config_list and change_pw

[Chris Nulk]

On 1/28/2013 10:53 AM, Mark Sapiro wrote:

Chris Nulk wrote:

It seems to make a little more sense to use scripts/programs that are
more targeted to specific changes I want to make to a list and I can
document what I am doing a little easier.


If you know what you are doing, you can do anything to a list with
config_list or withlist. These are powerful tools that also have the
ability to break things badly if you don't know what you're doing or
make a mistake.

A targeted script is more limited in what it can do, but if it is well
tested, probably won't break anything in an irrecoverable way.

Most of the scripts in my scripts/ directory were created in response
to "how do I" questions on this list that I thought were of some wider
interest. They exist so I can point people at them rather than
explaining how to do it each time it comes up. Others are things I
actually use myself.

Ultimately, its a matter of taste, style and frequency of use that
determines the choice of a targeted script vs. a more general tool to
accomplish a specific task.



Great information, Mark.  I appreciate your time and knowledge for 
replying.


I agree both config_list and withlist have a lot of power.  I think 
using the targeted scripts will work better for me since they are more 
limited.  I also appreciate you having created some very useful scripts 
for others to use.


Thanks again,
Chris
--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Information about config_list and change_pw

[Chris Nulk]

On 1/25/2013 6:10 PM, Mark Sapiro wrote:

Chris Nulk wrote:

Can I create a file (properly formatted) to input into config_list
containing only the values I want to change?  Or, do I have to dump the
entire file, make the changes, then reload?


You can include in the input to config_list only those things you wish
to change. Attributes not defined in the config_list input will be
unchanged.



Next, I see the change_pw allows me to change a list's password.  Is
that the owner's password, the moderator's password, or both?  If it is
only the owners password, is there a way to change the moderator's
password through some other mechanism/program?


It's only the list owner/admin password. You can use config_list to
change either the admin or moderator password or both.

This three-line input to config_list

from Mailman.Utils import sha_new
password = sha_new('abc').hexdigest()
mod_password = sha_new('def').hexdigest()

will set the admin password to abc and the moderator poassword to def.
When run, it will give this output

Non-standard property restored: password
Non-standard property restored: mod_password
attribute "sha_new" ignored

That is expected. If you don't want to set both passwords, just omit
the corresponding line from the input file.

The lines must be exactly as shown except for the actual passwords.

You can add additional settings in the same input file.



Thanks for the information, Mark.   While I was waiting for a response 
from the list, I searched the archives for the discussion about 
setting/unsetting the moderation bit on list members.  I remember there 
was a discussion and a script to it.  I found the discussion (and 
script) but also saw your website with Mailman scripts.  After reviewing 
the scripts on your site, would it be better to use the various scripts 
to make changes.


For example, should I use:
1. non_members script to make entry changes to the *_these_nonmembers,
2. change_admins.py to change the owner/moderator of a list,
3. change_pw to change the list owner/moderator password,
4. and set_mod.py to change the moderation bit for list members.

I know I will have to do some coding to add the ability to change the 
moderator password to the change_pw program.  I figured I would add a 
-m/--mod_password option and duplicate the -p option with appropriate 
corrections.


It seems to make a little more sense to use scripts/programs that are 
more targeted to specific changes I want to make to a list and I can 
document what I am doing a little easier.


Thanks for the information and any advice,
Chris

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

[Mailman-Users] Information about config_list and change_pw

[Chris Nulk]

Hello all,

Before I go and completely savage the lists we have here, I thought I 
would try to get a little more information about using config_list.


I get the list information from a file sent to me.  I want to use a 
script to process the file, output the processed data to a file, then 
use config_list to make the changes to the appropriate list(s).  The 
only values I want to change are the list's owner, moderator, and the 
*_these_nonmembers.


Can I create a file (properly formatted) to input into config_list 
containing only the values I want to change?  Or, do I have to dump the 
entire file, make the changes, then reload?



Next, I see the change_pw allows me to change a list's password.  Is 
that the owner's password, the moderator's password, or both?  If it is 
only the owners password, is there a way to change the moderator's 
password through some other mechanism/program?



Thanks,
Chris
--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Email command to find out what lists I am on

[Chris Nulk]

Thanks, Mark.  Given that I couldn't find anything, I suspected that was the 
case.
 
Chris

>>> On Tuesday, November 09, 2010 at 4:07 PM, Mark Sapiro  
>>> wrote:

C Nulk wrote:
>
>Is there a command I can send to Mailman VIA EMAIL so that Mailman send
>a list of all the lists of which I am a member back to me VIA EMAIL?


No.

--
Mark Sapiro The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] How to set up notificationsforMembershipManagement?

[Chris Nulk]

I am not asking for nor suggesting any changes but it might be easier for Ulf 
(if he is doing the mods) to use a bit mask.
 
Have the admin notify option use the following bits for differentiating between 
the subscribes and unsubscribes:
 
0x00: no notifications
0x01: notify subscribes
0x10: notify unsubscribes
0x11: notify both
 
In the code instead of looking for a yes/no/true/false, look for the 
appropriate bit.
 
Just my take (FWIW),
chris

>>> Mark Sapiro  Tuesday, June 09, 2009 11:23 AM >>>
Ulf Dunkel wrote:
>
>So wouldn't it be a good idea to introduce separate settings for both 
>welcome and goodbye events, e.g.
>- admin_notify_subscribes
>- admin_notify_unsubscribes


Mailman already has too many settings and options. I don't think it is
a good idea to add more, unless there is a clear demand.

I think (no evidence - just a guess) that as many people would think
this should be one option as would think it should be two.


>Should I write a feature request for this?


You are of course welcome to submit an RFE, but unless there is a
large, positive response, you probably won't see it implemented.

-- 
Mark Sapiro The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan

--
Mailman-Users mailing list
Mailman-Users@python.org 
http://mail.python.org/mailman/listinfo/mailman-users 
Mailman FAQ: http://wiki.list.org/x/AgA3 
Searchable Archives: http://www.mail ( http://www.mail/ 
)-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/cnulk%40scu.edu 

Security Policy: http://wiki.list.org/x/QIA9
--
Mailman-Users mailing list
Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Users] Mailman password reminders - Resolved

[Chris Nulk]

With the excellent help of Mark Sapiro, I was able to resolve my issue with 
password reminders.  Basically, we wanted for the majority of our lists to have 
the normal monthly reminders.  For the select few lists, we wanted quarterly 
reminders.  I made a few modifications to mailpasswds which Mark cleaned 
up/refined to allow on option to exclude a list from password reminders.  This 
was needed since my select few lists need to leave the "send_reminders" setting 
to on but I didn't want the monthly reminders to go out for them.  Now, my cron 
job for the monthly reminders can exclude the select few lists and I can add 
additional cron jobs to send reminders to only the select few lists.  E.G.
 
# send reminders to all lists except for list-a, list-b, and list-c
0 5 1 * * mailman /usr/lib/mailman/cron/mailpasswds -x list-a -x list-b -x 
list-c
# send reminders to list-a and list-c in Jan, Apr, and Oct
0 5 1 1,4,10 * mailman /usr/lib/mailman/cron/mailpasswds -l list-a -l list-c
# send reminders to list-b in Aug
0 5 1 8 * mailman /usr/lib/mailman/cron/mailpasswds -l list-b
 
The new option/switch added is -x or --exclude-list (in long form).  I am 
including a diff of the changes from Mailman v2.1.9 below.  The options of -l 
and -x are mutually exclusive.
 
Thanks,
Chris
 
--- Diff Begins ---
34c34,40
< allowed.
---
> allowed.  This and the -x/--exclude-list option are mutually 
> exclusive.
> 
> -x listname
> --exclude-list
> Do not send password reminders for the excluded list.  If omitted,
> reminders are sent for all lists.  Multiple -x/--exclude-list options
> are allowed.  This and the -l/--listname option are mutually 
> exclusive.
91,92c97,98
< opts, args = getopt.getopt(sys.argv[1:], 'l:h',
<['listname=', 'help'])
---
> opts, args = getopt.getopt(sys.argv[1:], 'l:x:h',
>['listname=', 'exclude-list=', 'help'])
99a106
> excludedlists = None
103a111,112
> if excludedlists:
> usage(1, '-l/--listname and -x/--exclude-list are mutually 
> exclusive')
107a117,123
> if opt in ('-x', '--exclude-list'):
> if listnames:
> usage(1, '-l/--listname and -x/--exclude-list are mutually 
> exclusive')
> if excludedlists is None:
> excludedlists = [arg]
> else:
> excludedlists.append(arg)
111a128,130
> if excludedlists:
> listnames = set(listnames) - set(excludedlists)
> 
--- Diff Ends --

>>> Mark Sapiro  Thursday, April 02, 2009 2:22 PM >>>
Chris Nulk wrote:
> 
>I know I can stop password reminders from going out to list members by turning 
>off the reminders in the list configuration.  I also know I can comment out 
>the cron entry for mailpasswd to stop it for all lists.  In addition, I know I 
>can send reminders using mailpasswd  to a specific list(s).  But, can I 
>exclude sending reminders to specifics lists with mailpasswd.  What I would 
>like to do is send the normal monthly reminders to most of our lists.  There 
>is a small group of lists that I want reminders to be sent to but on a 
>quarterly/yearly basis instead.
>
>Can I do this using mailpasswd or some other way?  Or should I pull out the 
>big hacking sword, close my eyes, and have a go at mailpasswd?  And, does 
>anyone really want to see the results if I do?


I don't recommend the hack away with eyes closed method, but it
shouldn't be too difficult to add a -x/--exclude option to mailpasswds.

On the other hand you could do something like

#!/bin/sh
excludes=`cat /path/to/lists/to/exclude`
for list in `/path/to/bin/list_lists --bare` ; do
  skip=0
  for xlist in $excludes ; do
if [ $list == $xlist ] ; then skip=1 ; fi
  done
  if [ $skip == 0 ] ; then
/path/to/cron/mailpasswds -l $list
  fi
done

-- 
Mark Sapiro The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan

--
Mailman-Users mailing list
Mailman-Users@python.org 
http://mail.python.org/mailman/listinfo/mailman-users 
Mailman FAQ: http://wiki.list.org/x/AgA3 
Searchable Archives: http://www.mail ( http://www.mail/ 
)-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/cnulk%40scu.edu 

Security Policy: http://wiki.list.org/x/QIA9
--
Mailman-Users mailing list
Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

[Mailman-Users] Mailman password reminders

[Chris Nulk]

Hello all,
 
I know I can stop password reminders from going out to list members by turning 
off the reminders in the list configuration.  I also know I can comment out the 
cron entry for mailpasswd to stop it for all lists.  In addition, I know I can 
send reminders using mailpasswd  to a specific list(s).  But, can I exclude 
sending reminders to specifics lists with mailpasswd.  What I would like to do 
is send the normal monthly reminders to most of our lists.  There is a small 
group of lists that I want reminders to be sent to but on a quarterly/yearly 
basis instead.

Can I do this using mailpasswd or some other way?  Or should I pull out the big 
hacking sword, close my eyes, and have a go at mailpasswd?  And, does anyone 
really want to see the results if I do?

Thanks,
Chris

--
Mailman-Users mailing list
Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

[Mailman-Users] LDAP and case-sensitive memberlist - UPDATE

[Chris Nulk]

Hello all,

Earlier this month, I asked about an issue I was having with the LDAPMembership 
adaptor and case-sensitivity.  Mark Sapiro most graciously helped me with the 
problem including making several changes to the LDAPMemberships.py file.  Mark 
was an outstanding help and resource for myself, knowing as little as I do of 
Python.

While somewhat documented in the file, I will mention some of the changes here.

1. __members is now a dictionary keyed by LCE with CPE as the values (straight 
from what Mark told me :) )

2. the getMemberName() method returns None if the member has no 'cn' value.

3. I added several additional fields (which we use here) to be retrieved from 
the LDAP database - GivenName, PreferredName, sn, and FullName.   GivenName and 
sn should probably common to most LDAP databases since they generally refer to 
a person's given/first name and their surname/lastname.  PreferredName and 
FullName may not be generally available.  PreferredName is what a person 
prefers to be called (i.e.  Jim instead of James).  FullName is simply the 
Given/First name and the Surname/lastname.

4. To help support 3. above, new mm_cfg.py settings were created - 
LDAP_DEFAULT_GIVENNAME, LDAP_NAME_SEPARATOR, and LDAP_SURNAME_FIRST.   These 
settings only apply to the 'displayable' name (__member_names).
   LDAP_DEFAULT_GIVENNAME allows you to provide a default givenname for an 
entry without one defined (say 'Unknown') [Default value = ''].  
   LDAP_NAME_SEPARATOR allows you to specify what string of characters you want 
to use to separate the Given/Preferred name from the Surname/Lastname.  
[Default value = ''].
   LDAP_SURNAME_FIRST allows you to specify whether the surname/lastname or the 
given/preferred name should be first.  [Default value = 0/False].

   When building the value for __member_names, the code first checks for 'sn'.  
If found, it then checks 'preferredname', 'givenname', and LDAP_DEFAULT_NAME in 
order and uses the first available choice.  With both a preferred/given name 
and a surname/lastname, the __member_names is built according to 
LDAP_NAME_SEPARATOR and LDAP_SURNAME_FIRST settings.  If 'sn' is not found, 
then 'fullname' is checked and used if available.  If 'fullname' is not 
available, then 'cn' is checked and used if available.  Otherwise None is 
returned.

Those are the basics to the changes.

I am attaching the latest version of LDAPMemberships.py to this message.  It 
should be text/plain but who knows.  If the attachment is stripped and several 
ask for the file, I will repost with the file included in the message (if 
allowed by the moderator).

Thanks to all that help,

Chris

--
Mailman-Users mailing list
Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

[Mailman-Users] LDAP and case-sensitive memberlist

[Chris Nulk]

Hello all,

I have recently set up Mailman v2.1.19 and added the LDAPMemberAdaptor I picked 
up from

>> See
>> .
>> 

I did check the FAQ and search the list for the LDAP stuff.  I configured it as 
per the docs and it works fine.  When I look at the member list (via Mailman's 
web interface), all the email addresses are there with proper capitalization 
(it is how it is defined in our LDAP and I CANNOT change the LDAP entries).

The problem is when anyone sends to the list, their posts are held for 
moderation because they are not a member of the list.  I have looked at the 
detailed message source on the moderator's page to view the message source.  It 
shows the sender's email address with the correct capitalization.  However, 
Mailman appears to be comparing a lowercase version of the sender's email 
address to the properly capitalized member list derived from the 
LDAPMemberAdapter.

Is there a way to force the comparison to be case-insensitive?  Or, modify the 
LDAPMemberAdaptor to lowercase all the email addresses?

Or, should I subscribe to the Mailman Developers list and repost there?

Thank you for your consideration,

Chris


--
Mailman-Users mailing list
Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9