Re: [Mailman-Developers] Problem with runner logger with unicode

2013-12-03 Thread nicolas
Basicaly it was not hyperkitty archiver, but mailman itself (or in fact 
Storm localized error message strings), cause here is the output :


Dec 03 12:37:49 2013 (23760) ACCEPT: 
CAMYTGwj+v6q+Utc2=wk_7nloeokc+r5ctb7pyc95fddmq0l...@mail.gmail.com

Dec 03 12:37:50 2013 (23762) conf loaded
Dec 03 12:37:50 2013 (23762) Uncaught runner exception: ERREUR:  la 
relation \xc2\xab listarchiver \xc2\xbb n\'existe pas\nLINE 1: 
...tarchiver.mailing_list_id, listarchiver.name FROM listarchiv...\n 
   ^\nDec 03 
12:37:50 2013 (23762) Traceback (most recent call last):\n  File 
/home/mailman/mailman/src/mailman/core/runner.py, line 169, in 
_one_iteration\nself._process_one_file(msg, msgdata)\n  File 
/home/mailman/mailman/src/mailman/core/runner.py, line 252, in 
_process_one_file\nkeepqueued = self._dispose(mlist, msg, msgdata)\n 
File /home/mailman/mailman/src/mailman/runners/pipeline.py, line 36, 
in _dispose\nprocess(mlist, msg, msgdata, pipeline)\n  File 
/home/mailman/mailman/src/mailman/core/pipelines.py, line 66, in 
process\nhandler.process(mlist, msg, msgdata)\n  File 
/home/mailman/mailman/src/mailman/handlers/rfc_2369.py, line 120, in 
process\nprocess(mlist, msg, msgdata)\n  File 
/home/mailman/mailman/src/mailman/handlers/rfc_2369.py, line 87, in 
process\narchiver_set = IListArchiverSet(mlist)\n  File 
/home/mailman/py27/local/lib/python2.7/site-packages/zope.component-4.1.0-py2.7.egg/zope/component/hookable.py, 
line 33, in __call__\nreturn self.__implementation(*args, **kw)\n  
File 
/home/mailman/py27/local/lib/python2.7/site-packages/zope.component-4.1.0-py2.7.egg/zope/component/_api.py, 
line 156, in adapter_hook\nreturn sitemanager.queryAdapter(object, 
interface, name, default)\n  File 
/home/mailman/py27/local/lib/python2.7/site-packages/zope.interface-4.0.5-py2.7-linux-x86_64.egg/zope/interface/registry.py, 
line 229, in queryAdapter\nreturn 
self.adapters.queryAdapter(object, interface, name, default)\n  File 
/home/mailman/mailman/src/mailman/model/mailinglist.py, line 590, in 
__init__\nListArchiver.name == archiver_name)).one()\n  File 
/home/mailman/py27/local/lib/python2.7/site-packages/storm-0.20-py2.7-linux-x86_64.egg/storm/store.py, 
line 1158, in one\nresult = 
self._store._connection.execute(select)\n  File 
/home/mailman/py27/local/lib/python2.7/site-packages/storm-0.20-py2.7-linux-x86_64.egg/storm/databases/postgres.py, 
line 266, in execute\nreturn Connection.execute(self, statement, 
params, noresult)\n  File 
/home/mailman/py27/local/lib/python2.7/site-packages/storm-0.20-py2.7-linux-x86_64.egg/storm/database.py, 
line 241, in execute\nraw_cursor = self.raw_execute(statement, 
params)\n  File 
/home/mailman/py27/local/lib/python2.7/site-packages/storm-0.20-py2.7-linux-x86_64.egg/storm/databases/postgres.py, 
line 276, in raw_execute\nreturn Connection.raw_execute(self, 
statement, params)\n  File 
/home/mailman/py27/local/lib/python2.7/site-packages/storm-0.20-py2.7-linux-x86_64.egg/storm/database.py, 
line 374, in raw_execute\nself._run_execution(raw_cursor, args, 
params, statement)\n  File 
/home/mailman/py27/local/lib/python2.7/site-packages/storm-0.20-py2.7-linux-x86_64.egg/storm/database.py, 
line 388, in _run_execution\n
self._check_disconnect(raw_cursor.execute, *args)\n  File 
/home/mailman/py27/local/lib/python2.7/site-packages/storm-0.20-py2.7-linux-x86_64.egg/storm/database.py, 
line 454, in _check_disconnect\nreturn function(*args, 
**kwargs)\nProgrammingError: ERREUR:  la relation \xc2\xab listarchiver 
\xc2\xbb n\'existe pas\nLINE 1: ...tarchiver.mailing_list_id, 
listarchiver.name FROM listarchiv...\n   
 ^\n\nDec 03 12:37:50 2013 (23762) SHUNTING: 
1386070670.221361+6120e5d3eaedee1815abe8fc6bf7dd306961f1dd


and the real problem origin is missing schema upgrades for postgres

Cheers

Le 2013-12-03 12:09, nicolas a écrit :

Hi,

I encountered a subtil problem when a component logs unicode strings
(present case was the hyperkitty archiver) :

(py27)mailman@mail:~$ Traceback (most recent call last):
  File /home/mailman/mailman/src/mailman/core/logging.py, line 76, 
in emit

msg = self.format(record)
  File /usr/lib/python2.7/logging/__init__.py, line 723, in format
return fmt.format(record)
  File /usr/lib/python2.7/logging/__init__.py, line 464, in format
record.message = record.getMessage()
  File /usr/lib/python2.7/logging/__init__.py, line 328, in 
getMessage

msg = msg % self.args
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position
21: ordinal not in range(128)
Logged from file runner.py, line 261
Traceback (most recent call last):
  File /home/mailman/mailman/src/mailman/core/logging.py, line 76, 
in emit

msg = self.format(record)
  File /usr/lib/python2.7/logging/__init__.py, line 723, in format
return fmt.format(record)
  File 

Re: [Mailman-Developers] Problem with runner logger with unicode

2013-12-03 Thread Aurelien Bompard
So, if I understand correctly, Storm add to its exception the error
message it gets from PostgreSQL, which is a bytestream (encoded in
UTF-8 on your system). But Mailman sends its logs to the logger as
unicode, thus the exception must be converted to unicode, and that's
where the UnicodeDecodeError is raised.
What can we do? Should we try to decode the Storm exception using the
default system encoding before sending it to the logger?

Aurélien
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9