Re: [Mailman-Users] Uncaught runner exception

2019-03-12 Thread Lothar Schilling 
> On 3/8/19 8:17 AM, Mark Sapiro wrote:

>> On 3/8/19 12:05 AM, Lothar Schilling wrote:
>>> Am 07.03.2019 um 23:54 schrieb Mark Sapiro:
 I would still like to find the underlying issue. I have been unable to
 duplicate it locally. I would appreciate it if before you run
 bin/unshunt, you could copy a few of the .pck files aside and send them
 to me off-list.
>>> Will send some .pck files to your private address. Some may hold content
>>> which is not meant to be publicly available, they are for your eyes only :-)
>>
>> Thank you. I will treat them confidentially.
>
> I have set up a test wherein I symlinked the qfiles path from the
> metadata in the shunted .pck files to the qfiles directory in the test
> installation, created the 5 lists that the shunted messages were
> destined to with preferred language = German and unshunted the 5 messages.
>
> The test Mailman is the head of the
> https://code.launchpad.net/~mailman-coders/mailman/2.1 branch, but this
> differs from the 2.1.29 release in minor ways, none of which have to do
> with archiving.
>
> Each of the unshunted messages was properly archived with no exceptions
> being thrown and qfiles/shunt/ was empty after unshunting.
>
> I then removed the 5 lists and started over, this time with
>
> ARCHIVER_OBSCURES_EMAILADDRS = No
>
> which is the only setting I see offhand that might affect this, but the
> result was the same, no error.
>
> Also, I note that some of the shunted messages had no non-ascii in From:
>
> I don't know what's going on in your installation, but I can't duplicate
> the issue. It may have something to do with list configuration or your
> German message catalog or templates if you have any changes in that area
> (i.e. any templates/site/de/* templates/HOST_NAME/de/* or
> templates/lists/LIST_NAME/de/* files not properly encoded as
> iso-8859-1). It could also be something in the
> archives/private/LIST_NAME/database/* files, although this would
> probably only affect the period in which you upgraded.
>
> In order to go any further with this, I'd need to see complete error and
> traceback information from Mailman's error log for the recently shunted
> messages.
>

An example of a complete error message:

Mar 11 17:02:22 2019 (15400) SHUNTING:
1552320069.504329+15fb4f71a5404fd1af3dafbe2780486d5b474cc7
Mar 11 17:22:37 2019 (15400) Uncaught runner exception: 'ascii' codec
can't decode byte 0xb5 in position 26: ordinal not in range(128)
Mar 11 17:22:37 2019 (15400) Traceback (most recent call last):
  File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 119, in _oneloop
    self._onefile(msg, msgdata)
  File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 190, in _onefile
    keepqueued = self._dispose(mlist, msg, msgdata)
  File "/usr/lib/mailman/Mailman/Queue/ArchRunner.py", line 77, in _dispose
    mlist.ArchiveMail(msg)
  File "/usr/lib/mailman/Mailman/Archiver/Archiver.py", line 216, in
ArchiveMail
    h.processUnixMailbox(f)
  File "/usr/lib/mailman/Mailman/Archiver/pipermail.py", line 602, in
processUnixMailbox
    self.add_article(a)
  File "/usr/lib/mailman/Mailman/Archiver/pipermail.py", line 646, in
add_article
    author = fixAuthor(article.decoded['author'])
  File "/usr/lib/mailman/Mailman/Archiver/pipermail.py", line 63, in
fixAuthor
    while i>0 and (L[i-1][0] in lowercase or
UnicodeDecodeError: 'ascii' codec can't decode byte 0xb5 in position 26:
ordinal not in range(128)

The problem is site-wide, not just for one or two lists. Anyway, there
are no altered templates or messages for a specific list. The only files
deviating from the default configuration are:
Defaults.py
mm_cfg.py

Of course Defaults.py has
DEFAULT_SERVER_LANGUAGE = 'en'.

So in mm_cfg.py I added
DEFAULT_SERVER_LANGUAGE = 'de'
add_language('en', 'English (USA)', 'iso-8859-1', 'ltr')

But I cannot see how this would cause the problem.

--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Uncaught runner exception

2019-03-12 Thread Mark Sapiro 
On 3/12/19 2:37 AM, Lothar Schilling wrote:
> 
> An example of a complete error message:
> 
> Mar 11 17:02:22 2019 (15400) SHUNTING:
> 1552320069.504329+15fb4f71a5404fd1af3dafbe2780486d5b474cc7
> Mar 11 17:22:37 2019 (15400) Uncaught runner exception: 'ascii' codec
> can't decode byte 0xb5 in position 26: ordinal not in range(128)
> Mar 11 17:22:37 2019 (15400) Traceback (most recent call last):
>   File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 119, in _oneloop
>     self._onefile(msg, msgdata)
>   File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 190, in _onefile
>     keepqueued = self._dispose(mlist, msg, msgdata)
>   File "/usr/lib/mailman/Mailman/Queue/ArchRunner.py", line 77, in _dispose
>     mlist.ArchiveMail(msg)
>   File "/usr/lib/mailman/Mailman/Archiver/Archiver.py", line 216, in
> ArchiveMail
>     h.processUnixMailbox(f)
>   File "/usr/lib/mailman/Mailman/Archiver/pipermail.py", line 602, in
> processUnixMailbox
>     self.add_article(a)
>   File "/usr/lib/mailman/Mailman/Archiver/pipermail.py", line 646, in
> add_article
>     author = fixAuthor(article.decoded['author'])
>   File "/usr/lib/mailman/Mailman/Archiver/pipermail.py", line 63, in
> fixAuthor
>     while i>0 and (L[i-1][0] in lowercase or
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xb5 in position 26:
> ordinal not in range(128)

Again, this is the code trying to "Canonicalize a name into Last, First
format". The complete statement is

while i>0 and (L[i-1][0] in lowercase or
   L[i-1].lower() in smallNameParts):
i = i - 1

smallNameParts is all lower case ascii, i.e.

= ['van', 'von', 'der', 'de']

and L is the list returned by

L = author.split()

and none of the shunted .pck files you provided me have any name pieces
longer than 14 bytes and only 2 of the 5 even have any non-ascii, but
the decode error refers to byte 0xb5 in position 26 (B5 in iso-8859-1 is µ).

I know you said in your original post

> I am using Python 2.6.6 on a Centos 6 system. The output of
> /usr/lib64/python >>> import string >>> string.lowercase is
> 'abcdefghijklmnopqrstuvwxyz' (as it is supposed to be).

but see below.

> The problem is site-wide, not just for one or two lists. Anyway, there
> are no altered templates or messages for a specific list. The only files
> deviating from the default configuration are:
> Defaults.py
> mm_cfg.py
> 
> Of course Defaults.py has
> DEFAULT_SERVER_LANGUAGE = 'en'.
> 
> So in mm_cfg.py I added
> DEFAULT_SERVER_LANGUAGE = 'de'
> add_language('en', 'English (USA)', 'iso-8859-1', 'ltr')
> 
> But I cannot see how this would cause the problem.

Neither can I, but I tried again with those settings just to be sure,
and got no issues.

All of this leads me to conclude that the issue must be non-ascii
following the 'z' in string.lowercase.

What is the path to the Python interpreter that Mailman is using?

What do you get if in that Python you do

import string
lowercase
string

For example, I get

>>> import string
>>> lowercase
'abcdefghijklmnopqrstuvwxyz'
>>> string


-- 
Mark Sapiro The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org