[Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7
Hi All, After a recent upgraded to Zope 2.12, I'm now seeing errors like the following when using IE (version 8) or Safari (version 4.0.3) to view page templates. Chrome and Firefox work fine. Traceback (innermost last): Module ZPublisher.Publish, line 127, in publish Module ZPublisher.mapply, line 77, in mapply Module ZPublisher.Publish, line 47, in call_object Module Shared.DC.Scripts.Bindings, line 324, in __call__ Module Shared.DC.Scripts.Bindings, line 361, in _bindAndExec Module Products.PageTemplates.ZopePageTemplate, line 335, in _exec Module Products.PageTemplates.ZopePageTemplate, line 432, in pt_render Module Products.PageTemplates.PageTemplate, line 80, in pt_render Module zope.pagetemplate.pagetemplate, line 115, in pt_render Module zope.tal.talinterpreter, line 271, in __call__ Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 751, in do_insertStructure_tal UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position 366: ordinal not in range(128) Has anyone seen this or anything similar before? Any solutions spring to mind? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7
On Tue, Aug 31, 2010 at 11:40 AM, Chris Withers wrote: > Hi All, > > After a recent upgraded to Zope 2.12, I'm now seeing errors like the > following when using IE (version 8) or Safari (version 4.0.3) to view > page templates. Chrome and Firefox work fine. > > Traceback (innermost last): > Module ZPublisher.Publish, line 127, in publish > Module ZPublisher.mapply, line 77, in mapply > Module ZPublisher.Publish, line 47, in call_object > > Module Shared.DC.Scripts.Bindings, line 324, in __call__ > Module Shared.DC.Scripts.Bindings, line 361, in _bindAndExec > Module Products.PageTemplates.ZopePageTemplate, line 335, in _exec > Module Products.PageTemplates.ZopePageTemplate, line 432, in pt_render > Module Products.PageTemplates.PageTemplate, line 80, in pt_render > Module zope.pagetemplate.pagetemplate, line 115, in pt_render > Module zope.tal.talinterpreter, line 271, in __call__ > > Module zope.tal.talinterpreter, line 343, in interpret > Module zope.tal.talinterpreter, line 751, in do_insertStructure_tal > UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position > 366: ordinal not in range(128) > > Has anyone seen this or anything similar before? > Any solutions spring to mind? > > Hi Chris. No solution, actually, but I would suggest to sniff the traffic between you and the server; I suspect different headers cause problems, here. Regards Marco > cheers, > > Chris > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > ___ > Zope-Dev maillist - Zope-Dev@zope.org > https://mail.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > https://mail.zope.org/mailman/listinfo/zope-announce > https://mail.zope.org/mailman/listinfo/zope ) > -- Marco Bizzarri http://code.google.com/p/qt-asterisk/ http://notenotturne.blogspot.com/ http://iliveinpisa.blogspot.com/ ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7
Marco Bizzarri wrote: > > No solution, actually, but I would suggest to sniff the traffic between > you and the server; I suspect different headers cause problems, here. Of that I have no doubt. What does Zope2's publisher do with requested encodings and the like now? Why would this be causing problems? Errors that affect all browsers I'd be inclined to attribute to my code, ones that randomly affect specific browsers appear more likely to be a bug in Zope for me... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7
Am 31.08.2010, 12:05 Uhr, schrieb Chris Withers : > What does Zope2's publisher do with requested encodings and the like now? > Why would this be causing problems? I can't remember exactly where but preferred_charset is looked for. It isn't set by either IE or Safari and, although it should default to UTF-8 it seems they get served Latin 1 and you get badda boom pretty quickly. > Errors that affect all browsers I'd be inclined to attribute to my code, > ones that randomly affect specific browsers appear more likely to be a > bug in Zope for me... It is and I thought it had been squashed. Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226 ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7
Charlie Clark wrote: > Am 31.08.2010, 12:05 Uhr, schrieb Chris Withers : > >> What does Zope2's publisher do with requested encodings and the like now? >> Why would this be causing problems? > > I can't remember exactly where but preferred_charset is looked for. It > isn't set by either IE or Safari and, although it should default to UTF-8 > it seems they get served Latin 1 and you get badda boom pretty quickly. That's weird, the content in this app should be fine to be served latin-1. >> Errors that affect all browsers I'd be inclined to attribute to my code, >> ones that randomly affect specific browsers appear more likely to be a >> bug in Zope for me... > > It is and I thought it had been squashed. Any idea what I should be looking for and in what changelog? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7
Am 31.08.2010, 12:51 Uhr, schrieb Chris Withers :
> Any idea what I should be looking for and in what changelog?
This my horribly patched version of charset.py which we just use via an
override. I know it's wrong but it was the easiest thing to do at a time.
class HTTPCharsets(object):
"""This is a complete copy of zope.publisher.http.HTTPCharset
except if there is no charset specified. Zope defaults to
ISO-8859-1 and we use the portal default_charset"""
adapts(IHTTPRequest)
implements(IUserPreferredCharsets)
def __init__(self, request):
self.request = request
def getPreferredCharsets(self):
'''See interface IUserPreferredCharsets'''
charsets = []
sawstar = sawiso88591 = 0
header_present = 'HTTP_ACCEPT_CHARSET' in self.request
for charset in self.request.get('HTTP_ACCEPT_CHARSET',
'').split(','):
charset = charset.strip().lower()
if charset:
if ';' in charset:
charset, quality = charset.split(';')
if not quality.startswith('q='):
# not a quality parameter
quality = 1.0
else:
try:
quality = float(quality[2:])
except ValueError:
continue
else:
quality = 1.0
if quality == 0.0:
continue
if charset == '*':
sawstar = 1
if charset == 'iso-8859-1':
sawiso88591 = 1
charsets.append((quality, charset))
# Quoting RFC 2616, $14.2: If no "*" is present in an
Accept-Charset
# field, then all character sets not explicitly mentioned get a
# quality value of 0, except for ISO-8859-1, which gets a quality
# value of 1 if not explicitly mentioned.
# And quoting RFC 2616, $14.2: "If no Accept-Charset header is
# present, the default is that any character set is acceptable."
if not sawstar and not sawiso88591 and header_present:
ptool = getUtility(IPropertiesTool)
default_charset = ptool.getProperty('default_charset', None)
charsets.append((1.0, default_charset))
# charsets.append((1.0, 'iso-8859-1'))
# UTF-8 is **always** preferred over anything else.
# Reason: UTF-8 is not specific and can encode the entire unicode
# range , unlike many other encodings. Since Zope can easily use
very
# different ranges, like providing a French-Chinese dictionary, it
is
# always good to use UTF-8.
# charsets.sort(sort_charsets)
charsets = [charset for quality, charset in charsets]
if sawstar and charsets[0] != 'utf-8':
if 'utf-8' in charsets:
idx = charsets.index('utf-8')
del charsets[idx]
charsets.insert(0, 'utf-8')
return charsets
The related bug in zope.publisher may or may not have found its way into
the current release.
https://bugs.launchpad.net/zope2/+bug/143873
In addition, but I don't know how relevant this is in your case, Jens
improved FSPageTemplate
https://bugs.launchpad.net/zope-cmf/+bug/322263
Charlie
--
Charlie Clark
Managing Director
Clark Consulting & Research
German Office
Helmholtzstr. 20
Düsseldorf
D- 40215
Tel: +49-211-600-3657
Mobile: +49-178-782-6226
___
Zope-Dev maillist - Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7
Maybe not the best solution but I use this hack:
zope.tal-3.5.2-py2.7.egg/zope/tal/talinterpreter.py
line 751
change:
text = unicode(structure)
to
try:
text = unicode(structure)
except:
text=unicode(structure.decode('utf-8'))
Vlad
2010/8/31 Chris Withers :
> Hi All,
>
> After a recent upgraded to Zope 2.12, I'm now seeing errors like the
> following when using IE (version 8) or Safari (version 4.0.3) to view
> page templates. Chrome and Firefox work fine.
>
> Traceback (innermost last):
> Module ZPublisher.Publish, line 127, in publish
> Module ZPublisher.mapply, line 77, in mapply
> Module ZPublisher.Publish, line 47, in call_object
>
> Module Shared.DC.Scripts.Bindings, line 324, in __call__
> Module Shared.DC.Scripts.Bindings, line 361, in _bindAndExec
> Module Products.PageTemplates.ZopePageTemplate, line 335, in _exec
> Module Products.PageTemplates.ZopePageTemplate, line 432, in pt_render
> Module Products.PageTemplates.PageTemplate, line 80, in pt_render
> Module zope.pagetemplate.pagetemplate, line 115, in pt_render
> Module zope.tal.talinterpreter, line 271, in __call__
>
> Module zope.tal.talinterpreter, line 343, in interpret
> Module zope.tal.talinterpreter, line 751, in do_insertStructure_tal
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position
> 366: ordinal not in range(128)
>
> Has anyone seen this or anything similar before?
> Any solutions spring to mind?
>
> cheers,
>
> Chris
>
> --
> Simplistix - Content Management, Batch Processing & Python Consulting
> - http://www.simplistix.co.uk
> ___
> Zope-Dev maillist - zope-...@zope.org
> https://mail.zope.org/mailman/listinfo/zope-dev
> ** No cross posts or HTML encoding! **
> (Related lists -
> https://mail.zope.org/mailman/listinfo/zope-announce
> https://mail.zope.org/mailman/listinfo/zope )
>
--
Best Regards
Vlad Vorobiev
___
Zope-Dev maillist - Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7
Vladislav Vorobiev wrote: > Maybe not the best solution but I use this hack: > > zope.tal-3.5.2-py2.7.egg/zope/tal/talinterpreter.py Indeed, certainly not a good solution. This implies that things should be unicode by the time they get here. Now, the decoding shouldn't be dependent on anything the browser sends, should it? I would have thought only the encoding side of things should matter.. ...so why on earth does the browser used have an effect on the decoding (or not, as seems to be the case with the bug) of the structure? (and where is this decoding done?) cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7
Hi Chris, On 08/31/2010 11:40 AM, Chris Withers wrote: > After a recent upgraded to Zope 2.12, I'm now seeing errors like the > following when using IE (version 8) or Safari (version 4.0.3) to view > page templates. Chrome and Firefox work fine. This may be related to IE and Safari not sending an accept-charset header and Zope then falling back to the Python default (often ASCII) for encoding, which then fails. I use the following patch to http.py (tested up to 2.12.3): --- lib/python/zope/publisher/http.py.orig 2009-05-02 00:49:35.0 -0700 +++ lib/python/zope/publisher/http.py 2009-06-12 02:12:50.0 -0700 @@ -965,8 +965,11 @@ # value of 1 if not explicitly mentioned. # And quoting RFC 2616, $14.2: "If no Accept-Charset header is # present, the default is that any character set is acceptable." -if not sawstar and not sawiso88591 and header_present: +# ROC: the following checked also for header_present which seemed wrong +if not sawstar and not sawiso88591: charsets.append((1.0, 'iso-8859-1')) +# ROC: if there is no header we assume star (to get utf-8) +sawstar = 1 # UTF-8 is **always** preferred over anything else. # Reason: UTF-8 is not specific and can encode the entire unicode # range , unlike many other encodings. Since Zope can easily use very Best Robert ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7
On Tue, Aug 31, 2010 at 10:40:46AM +0100, Chris Withers wrote: > After a recent upgraded to Zope 2.12, I'm now seeing errors like the > following when using IE (version 8) or Safari (version 4.0.3) to view > page templates. Chrome and Firefox work fine. > > Traceback (innermost last): >Module ZPublisher.Publish, line 127, in publish >Module ZPublisher.mapply, line 77, in mapply >Module ZPublisher.Publish, line 47, in call_object > >Module Shared.DC.Scripts.Bindings, line 324, in __call__ >Module Shared.DC.Scripts.Bindings, line 361, in _bindAndExec >Module Products.PageTemplates.ZopePageTemplate, line 335, in _exec >Module Products.PageTemplates.ZopePageTemplate, line 432, in pt_render >Module Products.PageTemplates.PageTemplate, line 80, in pt_render >Module zope.pagetemplate.pagetemplate, line 115, in pt_render >Module zope.tal.talinterpreter, line 271, in __call__ > >Module zope.tal.talinterpreter, line 343, in interpret >Module zope.tal.talinterpreter, line 751, in do_insertStructure_tal > UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position > 366: ordinal not in range(128) > > Has anyone seen this or anything similar before? Yes. Basically, every Zope site I've upgraded to 2.12 has suffered from this. Annoyingly this only happens with MSIE, so developers/admins tend not to notice. Clarifying the "only with MSIE" comment: this happens if and only if the HTTP request doesn't explicitly include 'UTF-8' in Accept-Charset. IIRC. > Any solutions spring to mind? Magically sprinkle .encode() / .decode() in TAL expressions/Python scripts, where appropriate. Most of the data stored in the ZODB is UTF-8 raw strings (things like DTML Documents and DTML Methods don't support storing Unicode data). Page templates now suddenly (since Zope 2.11) refuse to deal with raw strings and insist on having Unicode. Which means you need to do explicit conversions not at your system boundaries, but inside your Zope site, as you pass data between Python scripts and page templates. Marius Gedminas -- http://pov.lt/ -- Zope 3/BlueBream consulting and development signature.asc Description: Digital signature ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7
Marius Gedminas wrote: >> Has anyone seen this or anything similar before? > > Yes. Basically, every Zope site I've upgraded to 2.12 has suffered from > this. Annoyingly this only happens with MSIE, so developers/admins tend > not to notice. ...well, and Safari, but no-one uses that, right? ;-) > Clarifying the "only with MSIE" comment: this happens if and only if > the HTTP request doesn't explicitly include 'UTF-8' in Accept-Charset. > IIRC. OK, but I'll ask this again; what has the present of an Accept-Charset header got to do with whether or not rending a page template to a unicode string succeeds? Surely, up to that point, it really shouldn't matter what the browser wants or doesn't want, we're just getting a unicode string ready from whatever data we have around, right? >> Any solutions spring to mind? > > Magically sprinkle .encode() / .decode() in TAL expressions/Python > scripts, where appropriate. I want to get to the bottom of why the Accept-Charset header is getting involved at this point. Anyone able to point me at the code that does this? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7
Hi Chris, 2010/9/1 Chris Withers : > Marius Gedminas wrote: >>> Has anyone seen this or anything similar before? >> >> Yes. Basically, every Zope site I've upgraded to 2.12 has suffered from >> this. Annoyingly this only happens with MSIE, so developers/admins tend >> not to notice. > > ...well, and Safari, but no-one uses that, right? ;-) I think of this problem like a topic same as following URLs. * https://mail.zope.org/pipermail/zope-dev/2009-August/037565.html * https://bugs.launchpad.net/zope2/+bug/160968 * https://bugs.launchpad.net/zope2/+bug/143873 * http://dev.plone.org/plone/ticket/8185 And why would this problem appear again? Will this be regression? -- Takayuki SHIMIZUKAWA ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
