Re: [Mailman-Users] utf-8 error with all lists

[Dimitri Maziuk via Mailman-Users]

On 5/13/19 10:19 AM, Mark Sapiro wrote:
> On 5/13/19 7:46 AM, Jason Morrill wrote:
>>

>> It appears that lowercase does return an apparent properly lowercased string 
>> of letters. So, I'm uncertain why the hand-edited code would function 
>> differently.
> 
> 
> This is what we were unable to solve in the original thread starting at
> 
> for 10 posts and then continuing at
> 
> for another 13 posts.

Just pipe it through

lambda fixstr s : return unicodedata.normalize('NFKD',
 s.decode('utf-8')).encode('ascii','ignore'))

or something along those lines first and be done with it.

-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu



signature.asc
Description: OpenPGP digital signature
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] utf-8 error with all lists

[Mark Sapiro]

On 5/13/19 7:46 AM, Jason Morrill wrote:
> 
> Now that I have my Python-fu working I'm retesting Mark's suggestion:
> 
> webserv:~# python
> Python 2.7.13 (default, Sep 26 2018, 18:42:22)
> [GCC 6.3.0 20170516] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
 import string
 string.lowercase
> 'abcdefghijklmnopqrstuvwxyz'

> 
> It appears that lowercase does return an apparent properly lowercased string 
> of letters. So, I'm uncertain why the hand-edited code would function 
> differently.


This is what we were unable to solve in the original thread starting at

for 10 posts and then continuing at

for another 13 posts.

When the OP put in the logging suggested at
, we
discovered that in this Mailman/Archiver/pipermail.py at least, string
.lowercase returns the iso-8859-1 encoding of
'abcdefghijklmnopqrstuvwxyzµßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþÿ' (see
),
but we were never able to determine where this came from.

If you are curious, you could modify your current
/var/lib/mailman/Mailman/Archiver/pipermail.py which i think will contain

lowercase = 'abcdefghijklmnopqrstuvwxyz' at line 13.

If you add following that, the two lines

import string
syslog('error', 'string: %r', string)

You should then see (after at least one post) messages in Mailman's
error log like

 string: The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] unshunt does nothing

[Jason Morrill]

I believe this issue is related to my other trouble with utf-8.
https://mail.python.org/pipermail/mailman-users/2019-May/084435.html

Once I hand-edited /var/lib/mailman/Mailman/Archiver/pipermail.py, per the 
suggestion in that message. I restarted Mailman and ran the unshunt command 
again.

The shunted messages cleared out and no errors appeared. Though there was no 
positive affirmation the shunted messages were cleared either. Is the 'shunt' 
command meant to display anything?

 Per Mark

No. All unshunt does is move the .pck files from the shunt queue to the
original queue from which they were shunted. It does this silently.
The requeued entry is then processed again by the respective qrunner,
and this process either succeeds or possibly throws the same exception
that caused the shunting in the first place, in which case the message
is shunted again.
=

Thanks,

  Jason Morrill

  IT Director

  Child and Family Agency

  860-443-2896 x1422

  http://www.childandfamilyagency.org/


From: Mailman-Users 
[mailman-users-bounces+morrillj=childandfamilyagency@python.org] on behalf 
of Mark Sapiro [m...@msapiro.net]
Sent: Saturday, May 11, 2019 1:45 PM
To: mailman-users@python.org
Subject: Re: [Mailman-Users] unshunt does nothing

On 5/10/19 12:46 PM, Jason Morrill wrote:
> I have messages accumulating in the qfiles/shunt folder.
>
> Running unshunt does nothing. Literally.
>
> No error message, no progress message. The only change I see is the timestamp 
> on the shunted messages is updated to when I ran the unshunt command.


This is because the message is restored to it's original queue and
processed and throws the same exception that occurred before and is
shunted again.

Are there new entries in Mailman's error log when you unshunt?


> This is on Debian 9.6 with Python 2.7.13 and Mailman 2.1.23 , all installed 
> from the Debian package manager.
>
> BTW - I've been using Mailman for 10+ years on this machine. It's not a new 
> install and these are not new lists.


What changed? Was the Debian package updated?

--
Mark Sapiro The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/morrillj%40childandfamilyagency.org
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] utf-8 error with all lists

[Jason Morrill]

Realizing I didn't respond to the list. So scroll down for my response. Then 
come back here to continue reading. 

Now that I have my Python-fu working I'm retesting Mark's suggestion:

webserv:~# python
Python 2.7.13 (default, Sep 26 2018, 18:42:22)
[GCC 6.3.0 20170516] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import string
>>> string.lowercase
'abcdefghijklmnopqrstuvwxyz'
>>>

It appears that lowercase does return an apparent properly lowercased string of 
letters. So, I'm uncertain why the hand-edited code would function differently.

Thanks,

  Jason Morrill

  IT Director

  Child and Family Agency

  860-443-2896 x1422

  http://www.childandfamilyagency.org/


From: Jason Morrill
Sent: Monday, May 13, 2019 10:29 AM
To: Mark Sapiro
Subject: RE: [Mailman-Users] utf-8 error with all lists

Mark, Thanks for the suggestions and link to the remainder of that prior 
message thread.

I too get the same results related to Python and lowercase.

Now, I'm not a Python programmer but can certainly stumble through to sort 
things out. Here's what I'm seeing:

webserv:~# python
Python 2.7.13 (default, Sep 26 2018, 18:42:22)
[GCC 6.3.0 20170516] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import string
>>> lowercase
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'lowercase' is not defined
>>> string

>>>

Per your suggestion I modified /var/lib/mailman/Mailman/Archiver/pipermail.py 
to set lowercase = 'abcdefghijklmnopqrstuvwxyz'

And indeed the issue worked itself out. No more errors and I was able to clear 
the qfiles/shunt folder too.

Per the Python docs, and your info, the string returned by lowercase should be 
a-z. And it depends upon the locale of the server. Something I've never 
purposefully changed.

In Debian it looks at /etc/locale.gen

In that file I see my locale set to be
en_US ISO-8859-1

Could that somehow be related to this issue?

Thanks,

  Jason Morrill

  IT Director

  Child and Family Agency

  860-443-2896 x1422

  http://www.childandfamilyagency.org/


From: Mailman-Users 
[mailman-users-bounces+morrillj=childandfamilyagency@python.org] on behalf 
of Mark Sapiro [m...@msapiro.net]
Sent: Saturday, May 11, 2019 2:13 PM
To: mailman-users@python.org
Subject: Re: [Mailman-Users] utf-8 error with all lists

On 5/10/19 12:37 PM, Jason Morrill wrote:
> Since the beginning of May I've had a problem with all messages going through 
> Mailman. They are being delivered to the lists properly but copies are 
> staying in qfiles/shunt
>
> The errors appearing in the Mailman error.1 log file indicate
> "
> May 05 10:16:29 2019 (1167) Uncaught runner exception: 'utf8' codec can't 
> decode byte 0xaa in position 26: invalid start byte
...
>   File "/var/lib/mailman/Mailman/Archiver/pipermail.py", line 63, in fixAuthor
> while i>0 and (L[i-1][0] in lowercase or
> UnicodeDecodeError: 'utf8' codec can't decode byte 0xaa in position 26: 
> invalid start byte
>
> May 05 10:16:29 2019 (1167) SHUNTING: 
> 1557065787.914554+25c37ef6e662cc98d41ac9d45855fe04f151e11a
>
> "
>
> This is on Debian 9.6 with Python 2.7.13 and Mailman 2.1.23 , all installed 
> from the Debian package manager.
>
> I've spent several hours hunting around for a potential solution. Then 
> stumbled upon this:
> https://mail.python.org/pipermail/mailman-users/2019-March/084243.html


If you read the entire thread (which is split in the archive with the
second part beginning at
),
you will see that I think the issue is that whatever changed in your
system at the beginning of May is causing the Python

from string import lowercase

in /var/lib/mailman/Mailman/Archiver/pipermail.py to import a lowercase
object which should be the string 'abcdefghijklmnopqrstuvwxyz', but in
fact is longer and has non-ascii accented characters following 'z'.

The OP in that thread couldn't verify that that was the case, but I am
convinced that it is.

There doesn't seem to be a Debian patch to mailman 1:2.1.23-1+deb9u4
that would affect this so it is apparently something in Python or the OS.

I am certain that if you start with the base
/var/lib/mailman/Mailman/Archiver/pipermail.py and change

from string import lowercase

at line 13 to

lowercase = 'abcdefghijklmnopqrstuvwxyz'

the problem will be fixed.

--
Mark Sapiro The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: