Re: [Zope] UnicodeDecodeError
Hi Yuri, does this error occur with all browsers? I once had a similar issue that was caused only by some browsers because they do not send a HTTP_ACCEPT_CHARSET header. Maybe the following bug reports helps you in solving the issue: https://bugs.launchpad.net/zope2/+bug/160968 I'm using a workaround similar to that in the bug report in my environment (Zope 2.13.13 and a mixture of utf-8 and iso8859). Without the workaround the issue does still occur for me. Greetings, Stefan Am 03.04.2012 09:40, schrieb Yuri: Hi! I've an Zope application, basically postgres + zpt. The postgres db has been upgraded to 8.5, the data inside is utf-8 (dumped and checkd it), the pg adapeter is the latest, and the *Z Psycopg 2 Database Connection is configured to use utf-8.* Zope is 2.12.20 + latest security fix. When I try to do: span tal:replace=result/?column / (result came from an ZSQL method) I get an UnicodeDecodeError: Traceback (innermost last): Module ZPublisher.Publish, line 127, in publish Module ZPublisher.mapply, line 77, in mapply Module ZPublisher.Publish, line 47, in call_object Module Shared.DC.Scripts.Bindings, line 324, in __call__ Module Shared.DC.Scripts.Bindings, line 361, in _bindAndExec Module Products.PageTemplates.ZopePageTemplate, line 335, in _exec Module Products.PageTemplates.ZopePageTemplate, line 432, in pt_render Module Products.PageTemplates.PageTemplate, line 80, in pt_render Module zope.pagetemplate.pagetemplate, line 113, in pt_render Module zope.tal.talinterpreter, line 271, in __call__ Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 888, in do_useMacro Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 946, in do_defineSlot Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 533, in do_optTag_tal Module zope.tal.talinterpreter, line 518, in do_optTag Module zope.tal.talinterpreter, line 513, in no_tag Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 821, in do_loop_tal Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 821, in do_loop_tal Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 852, in do_condition Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 376, in do_startEndTag Module zope.tal.talinterpreter, line 405, in do_startTag Module zope.tal.talinterpreter, line 502, in attrAction_tal UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 38: ordinal not in range(128) if I do: span tal:replace=python:str(result['column']).decode('utf-8') / it works. (I use str because I've also dates in the record, otherwise I could not decode being it not a string) I've several zpt to fix, and I would like it to work as it should. My analisys is that result['column'] is seen as a plain string with \xx characters inside, and decode just transform it in something that Zope recognize as utf-8 when it prints it. Anyway, any idea? ___ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev ) -- Stefan Loidl Phone: +49 89 35831 8882 Leibniz-Rechenzentrum Fax:+49 89 35831 8682 Boltzmannstr. 1 mailto:lo...@lrz.de 85748 Garchinghttp://www.lrz.de ___ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] UnicodeDecodeError in Zope 2.10.4 (upgrade from 2.8.4)
--On 17. Juli 2007 10:33:39 +0200 Maciej Wisniowski [EMAIL PROTECTED] wrote: I've upgraded my installation from Zope2.8.4 to Zope 2.10.4 (by copying data.fs, Products/ etc.). I have publisher encoding and management_page_charset set to utf-8. Also system default encoding is utf-8 Did you read http://www.zope.org/Products/Zope/2.10.4/Zope-2.10.4-released ? -aj pgpSGCPsV0oEi.pgp Description: PGP signature ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] UnicodeDecodeError in Zope 2.10.4 (upgrade from 2.8.4)
--On 17. Juli 2007 10:52:05 +0200 Andreas Jung [EMAIL PROTECTED] wrote: --On 17. Juli 2007 10:33:39 +0200 Maciej Wisniowski [EMAIL PROTECTED] wrote: I've upgraded my installation from Zope2.8.4 to Zope 2.10.4 (by copying data.fs, Products/ etc.). I have publisher encoding and management_page_charset set to utf-8. Also system default encoding is utf-8 Did you read http://www.zope.org/Products/Zope/2.10.4/Zope-2.10.4-released If the documented hints don't work, please submit a bugreport with a reproducable testcase in form of a unittest. -aj pgpT3AJebGwqh.pgp Description: PGP signature ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] UnicodeDecodeError in Zope 2.10.4 (upgrade from 2.8.4)
Did you read http://www.zope.org/Products/Zope/2.10.4/Zope-2.10.4-released Yes. The migration code should auto-detect ISO-8859-15 and UTF-8 encoded page templates. For other encodings you must set the environment variable ZPT_PREFERRED_ENCODING. The migration code applies to ZopePageTemplate instances only. I have utf-8 encoded ZPTs so I didn't used ZPT_PREFERRED_ENCODING When you download ZPT content through FTP or WebDAV the content is converted using the output_encoding property of the corresponding ZopePageTemplate instance. So far I didn't even try ftp or webdav with 2.10.4. The problems are with TTW. The encoding of a rendered ZPT (through HTTP) is determined through charset= within the content-type HTTP response header etc/zope.conf: default-zpublisher-encoding default: iso-8859-15 default-zpublisher-encoding is set to utf-8 In order to deal with UnicodeDecodeErrors in a reasonable way, we added a configurable Unicode conflict resolver. As I said before resolver is not even called when using 'structure' keyword. -- Maciej Wisniowski ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] UnicodeDecodeError in Zope 2.10.4 (upgrade from 2.8.4)
If the documented hints don't work, please submit a bugreport with a reproducable testcase in form of a unittest. Bug submitted with tests attached at: http://www.zope.org/Collectors/Zope/2339 -- Maciej Wisniowski ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] UnicodeDecodeError in Zope 2.10.4 (upgrade from 2.8.4)
Both em tal:content=python: 'żółć'template id/em. and em tal:content=structure python: 'żółć'template id/em is like playing with fire. Don't do it. What you've got there is unicode characters written down without any encoding information. It will work if you set the internal ZPT encoding to be the same as it was entered into the template which I can see is not ASCII which is what your system defaults to. This is internally what happens: x=\xc5 unicode(x) Traceback (most recent call last): File stdin, line 1, in module UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 0: ordinal not in range(128) import sys; sys.getdefaultencoding() 'ascii' Maciej Wisniowski wrote: I've upgraded my installation from Zope2.8.4 to Zope 2.10.4 (by copying data.fs, Products/ etc.). I have publisher encoding and management_page_charset set to utf-8. Also system default encoding is utf-8. Zope 2.10 is said to have better Unicode support with UnicodeEncodeConflictResolver. It is but unfortunatelly in some cases this is not enough. Seems that code that is 'protected' by Resolver is not the only code that may be affected by non unicode strings. Simple example is with 'structure' keyword. eg: This works (resolver resolves conflict): em tal:content=python: 'żółć'template id/em. This doesn't work: em tal:content=structure python: 'żółć'template id/em. Also if you have Folder instance and set it's Title to the string that contains some i18n characters you're not able to even add page template inside it. Error traceback in both cases is same: Error Type: UnicodeDecodeError Error Value: 'ascii' codec can't decode byte 0xc5 in position 200: ordinal not in range(128) Traceback (innermost last): Module ZPublisher.Publish, line 119, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 42, in call_object Module Shared.DC.Scripts.Bindings, line 313, in __call__ Module Shared.DC.Scripts.Bindings, line 350, in _bindAndExec Module Products.PageTemplates.PageTemplateFile, line 129, in _exec Module Products.PageTemplates.PageTemplate, line 89, in pt_render Module zope.pagetemplate.pagetemplate, line 117, in pt_render Module zope.tal.talinterpreter, line 271, in __call__ Module zope.tal.talinterpreter, line 346, in interpret Module zope.tal.talinterpreter, line 534, in do_optTag_tal Module zope.tal.talinterpreter, line 516, in no_tag Module zope.tal.talinterpreter, line 346, in interpret Module zope.tal.talinterpreter, line 754, in do_insertStructure_tal UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 200: ordinal not in range(128) problem is with: text = unicode(structure) at Module zope.tal.talinterpreter, line 754, in do_insertStructure_tal sitecustomize.py with setdefaultencoding('utf-8') solves this but it is not nice solution. I wonder whether this should be submitted as a bug, or maybe there is different solution that I've missed? For the record: one more thing that was wrong for me during migration was 'expand' attribute of ZPT that in case of few ZPT objects was set to true in 2.10.4 while oryginally in 2.8.4 it was false. This caused that tales expressions disappeared from ZPTs under 2.10.4. I've written a script that explicitly set expand=0 to all ZPT instances in Zope 2.8.4 and then, after migration, everything is OK. -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] UnicodeDecodeError in Zope 2.10.4 (upgrade from 2.8.4)
Both em tal:content=python: 'żółć'template id/em. and em tal:content=structure python: 'żółć'template id/em is like playing with fire. Don't do it. What you've got there is unicode characters written down without any encoding information. This is legacy code from zope 2.8.4 that I have to deal with during migration. All new code is supposed to use u'' strings. In fact, above is only an example but in real case we have properties set for folders, eg. 'title' that contains national characters. You can easily check it yourself if you want. Just create a Folder and set it's 'title' property to one that contains some unicode characters. You'll not be even able to add ZPT object into that folder I think (at last it is not possible for me). It will work if you set the internal ZPT encoding to be the same as it was entered into the template which I can see is not ASCII which is what your system defaults to. What do you mean by internal ZPT encoding? Docs says: Starting Zope 2.10.2 the ZPT implementation uses unicode as internal representation so there should be no encoding at this level, I think. This is internally what happens: x=\xc5 unicode(x) Traceback (most recent call last): File stdin, line 1, in module UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 0: ordinal not in range(128) import sys; sys.getdefaultencoding() 'ascii' I already realized same thing and used sitecustomize.py with sys.setdefaultencoding('utf-8') as a quick fix. This works but is not nice solution. I rather expected Zope 2.10 resolver to deal with this. -- Maciej Wisniowski ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] UnicodeDecodeError inside ZMI
Remy Pinsonnault wrote at 2006-9-5 11:18 -0400: With Zope 2.8.7 and Python 2.3.5 (Plone 2.1.3 installer), when I search inside the ZMI for the spanish ñ character (inside a Plone site object), I get an UnicodeDecodeError: Site Error An error was encountered while publishing this resource. *UnicodeDecodeError* Sorry, a site error occurred. Traceback (innermost last): ... - Module OFS.FindSupport, line 112, in ZopeFind UnicodeDecodeError: 'ascii' codec can't decode byte 0xf1 in position 0: ordinal not in range(128) This means that unicode and (non ASCII) byte strings are mixed together. Search the mailing list for UnicodeDecodeError (or setdefaultencoding) to find out how you can approach this problem. -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] UnicodeDecodeError
Luiz Fernando Bernardes Ribeiro wrote at 2006-8-14 11:03 -0300: Due to project demands I had to convert the html entities in all TEXT fields in my database to the real international characters like á, ç and so on. Good! The problem is that all my dynamic generated XML started to fail with UnicodeDecodeError. The XML encoding is set to utf-8 and the file itself is correctly encoded. If I inject the international characters using a Python Script with something like u'çáé' it works, but passing the TEXT field from the database generates the error. Then, this means that the value from the database is not unicode. Convert it to unicode by value.decode(the_database_encoding). -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] UnicodeDecodeError
Dieter Maurer wrote: Only the default is latin-1 (as specified by the HTML specification). Indeed, but that should be configurable... Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] UnicodeDecodeError
Joern Wallstabe wrote: Seem Zopes ZPT system is trying to do the encoding, depending on on setHeader() method. So if this header is set you can simply return unicode strings. It's actually lower down in the stack... ZPublisher encodes the response, and has a stoopid hard coded default of 'latin-1' that it returns ;-) cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] UnicodeDecodeError
Chris Withers wrote at 2005-5-25 20:54 +0100: Joern Wallstabe wrote: Seem Zopes ZPT system is trying to do the encoding, depending on on setHeader() method. So if this header is set you can simply return unicode strings. It's actually lower down in the stack... ZPublisher encodes the response, and has a stoopid hard coded default of 'latin-1' that it returns ;-) You are wrong: ZPublisher uses what it finds in the charset parameter of the Content-Type response header. Only the default is latin-1 (as specified by the HTML specification). -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] UnicodeDecodeError
--On Montag, 23. Mai 2005 10:35 Uhr +0200 Joern Wallstabe [EMAIL PROTECTED] wrote: I'm always getting a UnicodeDecodeError. To every error message belongs a traceback where you can see *where* the error occurs...another try? -aj pgpo0qQ4ukTs7.pgp Description: PGP signature ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] UnicodeDecodeError
Andreas Jung wrote: --On Montag, 23. Mai 2005 10:35 Uhr +0200 Joern Wallstabe [EMAIL PROTECTED] wrote: I'm always getting a UnicodeDecodeError. To every error message belongs a traceback where you can see *where* the error occurs...another try? -aj the script decodes unicode 'Spécification' - utf8 'Sp\xc3\xa9cification' the error is raised during zpt processing. by the way python default default encoding is set to ascii. Traceback (innermost last): Module ZPublisher.Publish, line 101, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Products.CMFFormController.FSControllerPageTemplate, line 98, in __call__ Module Products.CMFFormController.BaseControllerPageTemplate, line 42, in _call Module Shared.DC.Scripts.Bindings, line 306, in __call__ Module Shared.DC.Scripts.Bindings, line 343, in _bindAndExec Module Products.CMFCore.FSPageTemplate, line 188, in _exec Module Products.CMFCore.FSPageTemplate, line 127, in pt_render Module Products.PageTemplates.PageTemplate, line 97, in pt_render FSControllerPageTemplate at /cmf/edit_form used for /cmf/Members/wal/myid Module StringIO, line 203, in getvalue UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 95: ordinal not in range(128) regards joern smime.p7s Description: S/MIME Cryptographic Signature ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] UnicodeDecodeError
--On Montag, 23. Mai 2005 11:18 Uhr +0200 Joern Wallstabe [EMAIL PROTECTED] wrote: the script decodes unicode 'Spécification' - utf8 'Sp\xc3\xa9cification' the error is raised during zpt processing. by the way python default default encoding is set to ascii. hm...I can't reproduce this behaviour :-/ ANdreas pgpNQ9wMM1V1g.pgp Description: PGP signature ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] UnicodeDecodeError
Am Montag, den 23.05.2005, 10:35 +0200 schrieb Joern Wallstabe: Hello, i would like to feed a zpt form with data from a python dictionary via a python script zpt snippet: textarea name=title:record:ulines rows=3 cols=65 tal:span content=python:context.printDict(here.dict)/ This is wrong anyway. Use tal:content in the textarea instead. /textarea python script prinDict: keys_ = dict.keys() keys_.sort() lines=[] for key in keys_: lines.append(u%s:%s\n %(key ,dict[key])) uni_str = u.join(lines) return uni_str.encode('utf-8') I'm always getting a UnicodeDecodeError. I' using Zope 2.7.5, CMF 1.5.1, Python 2.3.5 HTML meta tag is set to meta http-equiv=Content-Type content=text/html; charset=utf-8 CMF default_charset = utf-8 meta charset does not count btw. CMF default should, but it really depends on the real content-type header you send. Make sure its the encoding you want there (setHeader() ) And its important to watch the encoding of the strings you work with. Where does the dictionary come from? ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] UnicodeDecodeError
Thank you all for replying - now it works I got utf-8 string in my form. Seem Zopes ZPT system is trying to do the encoding, depending on on setHeader() method. So if this header is set you can simply return unicode strings. Strange setting cmf default charset to utf-8 doesn't work :-( Am Montag, den 23.05.2005, 10:35 +0200 schrieb Joern Wallstabe: Hello, i would like to feed a zpt form with data from a python dictionary via a python script zpt snippet: textarea name=title:record:ulines rows=3 cols=65 tal:span content=python:context.printDict(here.dict)/ This is wrong anyway. Use tal:content in the textarea instead. o.k. replace that with tal:content content=python:context.printDict(here.dict)/ /textarea python script prinDict: keys_ = dict.keys() keys_.sort() lines=[] for key in keys_: lines.append(u%s:%s\n %(key ,dict[key])) uni_str = u.join(lines) return uni_str.encode('utf-8') function now returns unicode string without utf-8 encoding I'm always getting a UnicodeDecodeError. I' using Zope 2.7.5, CMF 1.5.1, Python 2.3.5 HTML meta tag is set to meta http-equiv=Content-Type content=text/html; charset=utf-8 CMF default_charset = utf-8 meta charset does not count btw. CMF default should, but it really depends on the real content-type header you send. Make sure its the encoding you want there (setHeader() ) That was the important hint: added to cmf main_header_template: tal:define dummy python: request.RESPONSE.setHeader('content-type', 'text/html;; charset=utf-8'); And its important to watch the encoding of the strings you work with. Where does the dictionary come from? python dictionary was result form parsing xml data into that dictionary (all unicode) Thanx again :-) smime.p7s Description: S/MIME Cryptographic Signature ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] UnicodeDecodeError
Joern Wallstabe wrote at 2005-5-23 11:18 +0200: ... the error is raised during zpt processing. by the way python default default encoding is set to ascii. FSControllerPageTemplate at /cmf/edit_form used for /cmf/Members/wal/myid Module StringIO, line 203, in getvalue UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 95: ordinal not in range(128) Your page template mixes unicode and non-unicode (which cannot be decoded with the default encoding). -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )