Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity
On Wed, Sep 22, 2010 at 09:24:11AM +0100, Chris Withers wrote: > On 21/09/2010 23:59, Marius Gedminas wrote: > > If this means Zope2-the-application becomes unusable out of the box if > > you've non-ASCII data in your database, then I'm -1. > > I'd call having the odd '?' shown in pages and a warning in your logs > telling you about the problem in clear terms versus sporadic total > failures by way of unicode exceptions a lot more usable, rather than the > current situation, which is conceptually insane *and* unusable ;-) There are people who install Zope 2 and then build their websites through the web exclusively. Asking them to figure out how to write and register a utility when they upgrade their 2.12 Zope site to 2.13 is a bit too much, in my opinion. Why replace a bad situation with a different bad situation? How would you like a piece of software that r?pl?c?d ?ll th? v?w?ls w?th q??st??n m?rks ??t ?f th? b?x unless you did a highly technical voodoo dance under the hood? I would be happy with the other version you pasted in this thread, the one that tries context.management_page_charset, UTF-8, and Latin-1. Let's do that instead. Marius Gedminas -- http://pov.lt/ -- Zope 3/BlueBream consulting and development signature.asc Description: Digital signature ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity
On Wed, Sep 22, 2010 at 09:34:59AM +0100, Chris Withers wrote:
> On 22/09/2010 03:50, Andreas Jung wrote:
> > I also strongly object such a change in the default behavior.
>
> I'm sorry, but WHAT?
>
> > I did a
> > lot of testing of the current implementation and configuration with
> > various browsers and Zope applications and add-ons in order to make the
> > Zope 3 ZPT engine in Zope 2 work out-of-the-box for most applications.
Did "various browsers" include Safari and MSIE, I wonder? Because those
are the ones causing me pain and UnicodeDecodeErrors.
> A less insane version of that default would be:
>
> class BasicEncodingConflictResolver(object):
>implements(IUnicodeEncodingConflictResolver)
>
>def resolve(self, context, text, expression):
>logging.warn('You should register an '
> 'IUnicodeEncodingConflictResolver that matches '
> 'your content')
>encodings = ['utf-8','latin-1']
>mpc = getattr(context,'management_page_charset',None)
>if mpc:
>encodings.insert(0,mpc)
>for enc in encodings:
>try:
>return unicode(text,enc)
>except UnicodeDecorError:
>pass
>return unicode(text,sys.getdefaultencoding(),'replace')
I would prefer this to current implementation.
Marius Gedminas
--
http://pov.lt/ -- Zope 3/BlueBream consulting and development
signature.asc
Description: Digital signature
___
Zope-Dev maillist - Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chris Withers wrote: > On 22/09/2010 09:58, Andreas Jung wrote: >> If you think it is crap then I cam happy to accept that (and I don't >> mind much because Zope 2 is not my radar right now). You are committer >> and you can fix my crap. Blame me for having written this crap but you >> could also blame me for having on this whole Zope 3 ZPT engine >> integration :-) > > Can you please answer me one thing though: in what way could it ever be > a good idea to consult the browser's list of acceptable character sets > to decide how to decode python strings into python unicode objects in a > database that the browser has never had anything to do with? Check with Launchpad - neither I can remember the details nor am I interested right now thinking about. There were reasons *and* real-world tests with browsers obviously required that. As said: the encoding handling is a game you can only lose inside Zope 2. - -aj -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQGUBAEBAgAGBQJMmcfoAAoJEADcfz7u4AZjQ90LwLQoiWZzKwnLhYJBaO/j4Hs5 RXxURlWAPX2VB1iNbzmzZgadffYdF4Tigmh/kuZFZwZLizJw+SnYwPwJJgAHM/dn aq2Q9JV3atjlu0p/OGe/54BY5ZfMWSE/AjS2gKSJExNdp9rKZbvOdKWg/J+utzq3 M/U7E2AXvOtG3YWZYggQF1QZlY3PZu08XlXHUCNdiK3fAu9dtJQstRGGdggdxhuQ QiheKAs2Tu/hKWsVPURlxes9zGtylpDAQxt5ycRxTxYToJyCQtFdSzBsbLC1rkGG fMDOOL/fPR3UlTzyhvB87ugaiBH7owng+n8tia3Ro06LzGp0oy5PdytrhfQ2BGn2 RzPre7tGavUFn6sxYSEZ6akIuHwk1KQgvMFN2ngsa/F1N1xqZpQUXwF1ALrEFzWc qg4CeO4UDdkh1V8ZlawbKgbih/0eaNYKreCg0vCb2xNqXvsl5OL0QERj0v4u/gWp pJ1zGB+BVHbtZifhHSHkKYmx9Dgpl3I= =EwM7 -END PGP SIGNATURE- <>___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chris Withers wrote: > On 21/09/2010 21:18, Andreas Jung wrote: >> I can not recall all details but at the time of the integration of the >> Zope 3 ZPT engine the unicode resolver approach was the best thinkable >> solution for dealing all possible edgecase where encoding issues may pop >> *for the sake of backwards compatibility* > > ...where it miserably failed. The projects I've hit this on have all > been running flawlessly on Zope 2.1(ish) through to Zope 2.9. > Zope 2.11 onwards, in my experience, phail, spectacularly... Sorry, not my problem! The Zope 3 ZPT engine together with the resolver is now available for at least two or three yearswe had a *very long* beta phase for the ZPT integration..have you tested? Obviously no...blaming me for doing a lot of testing while having seen zero contributions in terms of code and testing in this particular case is laughable. Such allegations were one of the reasons for quitting as Zope 2 release manager. If the release manager is the idiot doing the dirty work and taking the beating from people not contributing to the project that I was happy being the Zope 2 idiot for years but I am no longer interested dealing with a minority of people knowing better and constantly playing the niggler. Andreas -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQGUBAEBAgAGBQJMmcdSAAoJEADcfz7u4AZjn+4LvA2JtT++GJD+taeZZ+B95UGt fB7fINT1nUyXNWxDV6OyoZd2zgJoCX2Yg3V3FibVaz5uiQjGNkZnPvLooLP+9ESn lSyc6D7tYgHzPhNU/OitwUp8pjZeOWhLXShR0PH1TVtv9I8PAx/0NRDNZDLpE9Cx v1TbTlRztY+crRDkGK4xf9runTNZwVw5yyBk6NJXnKr1yMIOhZ72Fz0LAReiIHq0 vkgSakS7vhYde2acsPi4lXC4lS6s7T67/GjbcPWQAtUw4f4Xc4linYERvFCHUYOT w4z2CDahhyu7ZTfzguGCuUEQZAffi9OlY5LbFlXMSF9crFjrCrqxvPhGmAQXN5O+ z6tYlAXGCZpuV3gMNIPjJBej6UujYP/mnjT8SC5BYkGlYrriNfoBttVynK5TJpMk z5s90fDZa4hETaIXAxTkpCiBfX0IFUp7aqb+lU7HsSs4rHj331cknbemX/iq+S7O ohb5SMgDqJg20UTxxNFH/2ZLSdWsOm8= =Wyxr -END PGP SIGNATURE- <>___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity
On 22/09/2010 09:58, Andreas Jung wrote: > If you think it is crap then I cam happy to accept that (and I don't > mind much because Zope 2 is not my radar right now). You are committer > and you can fix my crap. Blame me for having written this crap but you > could also blame me for having on this whole Zope 3 ZPT engine > integration :-) Can you please answer me one thing though: in what way could it ever be a good idea to consult the browser's list of acceptable character sets to decide how to decode python strings into python unicode objects in a database that the browser has never had anything to do with? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Chris Withers wrote:
> On 22/09/2010 03:50, Andreas Jung wrote:
>> I also strongly object such a change in the default behavior.
>
> I'm sorry, but WHAT?
>
>> I did a
>> lot of testing of the current implementation and configuration with
>> various browsers and Zope applications and add-ons in order to make the
>> Zope 3 ZPT engine in Zope 2 work out-of-the-box for most applications.
>
> Er, you can't be serious...
>
> The idea of asking the user agent what encoding is used to encode data
> in the database is like telling an obese person they can have their foot
> amputated in order to lose weight.
>
> A less insane version of that default would be:
>
> class BasicEncodingConflictResolver(object):
> implements(IUnicodeEncodingConflictResolver)
>
> def resolve(self, context, text, expression):
> logging.warn('You should register an '
>'IUnicodeEncodingConflictResolver that matches '
>'your content')
> encodings = ['utf-8','latin-1']
> mpc = getattr(context,'management_page_charset',None)
> if mpc:
> encodings.insert(0,mpc)
> for enc in encodings:
> try:
> return unicode(text,enc)
> except UnicodeDecorError:
> pass
> return unicode(text,sys.getdefaultencoding(),'replace')
>
>
>> However with the given constraints there will be always applications or
>> bad written code
>
> PreferredCharsetResolver *is* bad code.
>
> I'll say it again, 'cos the message doesn't seem to be getting through:
>
> What on earth do the character sets accepted by the browser have to do
> with the python string encoding used in the data on the server, which up
> until that request, most likely has never been anywhere near that
> browser before?!
>
>> resolver. That's why it is pluggable.
>
> Yes, and well documented and advertised too... not!
>
>> existing resolver smarter and more efficient: +1 for tuning the
>> implementation but the proposed solution above does not solve anything
>> and will break a bunch of apps.
>
> bull crap.
>
> Chris
Please calm down - I am not up to reply to a posting written in a tone
that is even far beyond my personal style when ranting on other peoples
crap.
As said: the discussion is documented in some Launchpad ticket.
If you think it is crap then I cam happy to accept that (and I don't
mind much because Zope 2 is not my radar right now). You are committer
and you can fix my crap. Blame me for having written this crap but you
could also blame me for having on this whole Zope 3 ZPT engine
integration :-)
Cheers,
Andreas
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQGUBAEBAgAGBQJMmcU5AAoJEADcfz7u4AZj1wMLvRQ9riRMebLtqZe4VY9U0SCg
v8S8uS/gZXvBDlDodoNoPurumWGlabCXORD5GRgVZhGiSub1GmTXqslHD5Hro1xf
FeGr3skMlTlAis9xjH3qUuBUbivk5+ebk75ii/jPKJmxRg87a3Gwhi1eFnnCZBTg
AGWzYu5QFmKI/iHgWpYjdQiJ1fS2MtRrisF6JCVuduBJs3kenHAV8huw8/7azv6l
uI/tUQjg2N49s14NWZbIObe/cDY6SkSoiu4ifINTJX68aJxgywVqzGu1aiqR33t3
/j2v347/4TCwdfJGFBopvHLBdoI0WIHA4DB9C2VMuAYO/eOQSrH84gGLqbWeRBku
tfh7KqLa5mulyufXVc3YtgF14rk2kJ5TvRohM4YuR1RFB/0WGU6p0FzD8KZ23pB+
CXk9puJi/FTsIO+eSgDcuPZhoc6ZJoPgA2s03TD+8ghspnWSXU4hbZ2DncYzlDOu
Ss+ri+thcf5APSrvfCb4B7xLq/tW2do=
=yS8z
-END PGP SIGNATURE-
<>___
Zope-Dev maillist - Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity
On 21/09/2010 21:18, Andreas Jung wrote: > I can not recall all details but at the time of the integration of the > Zope 3 ZPT engine the unicode resolver approach was the best thinkable > solution for dealing all possible edgecase where encoding issues may pop > *for the sake of backwards compatibility* ...where it miserably failed. The projects I've hit this on have all been running flawlessly on Zope 2.1(ish) through to Zope 2.9. Zope 2.11 onwards, in my experience, phail, spectacularly... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity
Am 22.09.2010, 10:24 Uhr, schrieb Chris Withers : > I'd call having the odd '?' shown in pages and a warning in your logs > telling you about the problem in clear terms versus sporadic total > failures by way of unicode exceptions a lot more usable, rather than the > current situation, which is conceptually insane *and* unusable Next Wednesday is bug day - a good chance for a concerted operation on this. Chris, your description that the current implementation contravenes interface specification means that it is a bug. But no change should simply break existing applications. Can we come up with some tests (ZODB data and presumably client encodings) and a workaround to enable the fudge? I don't know how much non-ASCII you have but the "odd ?" might turn out to be quite a lot for some people. Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226 ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity
On 22/09/2010 03:50, Andreas Jung wrote:
> I also strongly object such a change in the default behavior.
I'm sorry, but WHAT?
> I did a
> lot of testing of the current implementation and configuration with
> various browsers and Zope applications and add-ons in order to make the
> Zope 3 ZPT engine in Zope 2 work out-of-the-box for most applications.
Er, you can't be serious...
The idea of asking the user agent what encoding is used to encode data
in the database is like telling an obese person they can have their foot
amputated in order to lose weight.
A less insane version of that default would be:
class BasicEncodingConflictResolver(object):
implements(IUnicodeEncodingConflictResolver)
def resolve(self, context, text, expression):
logging.warn('You should register an '
'IUnicodeEncodingConflictResolver that matches '
'your content')
encodings = ['utf-8','latin-1']
mpc = getattr(context,'management_page_charset',None)
if mpc:
encodings.insert(0,mpc)
for enc in encodings:
try:
return unicode(text,enc)
except UnicodeDecorError:
pass
return unicode(text,sys.getdefaultencoding(),'replace')
> However with the given constraints there will be always applications or
> bad written code
PreferredCharsetResolver *is* bad code.
I'll say it again, 'cos the message doesn't seem to be getting through:
What on earth do the character sets accepted by the browser have to do
with the python string encoding used in the data on the server, which up
until that request, most likely has never been anywhere near that
browser before?!
> resolver. That's why it is pluggable.
Yes, and well documented and advertised too... not!
> existing resolver smarter and more efficient: +1 for tuning the
> implementation but the proposed solution above does not solve anything
> and will break a bunch of apps.
bull crap.
Chris
--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
___
Zope-Dev maillist - Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity
On 21/09/2010 23:59, Marius Gedminas wrote:
> On Tue, Sep 21, 2010 at 07:39:46PM +0100, Chris Withers wrote:
>> So, would anyone object if I change the default, on both trunk and the
>> 2.12 branch to some variant of:
>>
>> class BasicEncodingConflictResolver(object):
>>implements(IUnicodeEncodingConflictResolver)
>>
>>def resolve(self, context, text, expression):
>>logging.warn('You should register an '
>> 'IUnicodeEncodingConflictResolver that matches '
>> 'your content')
>>return text.decode('ascii','replace')
>>
>> ...and delete that devil spawn turd that is PreferredCharsetResolver?
>
> If this means Zope2-the-application becomes unusable out of the box if
> you've non-ASCII data in your database, then I'm -1.
I'd call having the odd '?' shown in pages and a warning in your logs
telling you about the problem in clear terms versus sporadic total
failures by way of unicode exceptions a lot more usable, rather than the
current situation, which is conceptually insane *and* unusable ;-)
Chris
--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
___
Zope-Dev maillist - Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Marius Gedminas wrote:
> On Tue, Sep 21, 2010 at 07:39:46PM +0100, Chris Withers wrote:
>> So, would anyone object if I change the default, on both trunk and the
>> 2.12 branch to some variant of:
>>
>> class BasicEncodingConflictResolver(object):
>> implements(IUnicodeEncodingConflictResolver)
>>
>> def resolve(self, context, text, expression):
>> logging.warn('You should register an '
>>'IUnicodeEncodingConflictResolver that matches '
>>'your content')
>> return text.decode('ascii','replace')
>>
>> ...and delete that devil spawn turd that is PreferredCharsetResolver?
>
> If this means Zope2-the-application becomes unusable out of the box if
> you've non-ASCII data in your database, then I'm -1.
>
>
I also strongly object such a change in the default behavior. I did a
lot of testing of the current implementation and configuration with
various browsers and Zope applications and add-ons in order to make the
Zope 3 ZPT engine in Zope 2 work out-of-the-box for most applications.
However with the given constraints there will be always applications or
bad written code or specific requirements were you need to change the
resolver. That's why it is pluggable. There is is room for making the
existing resolver smarter and more efficient: +1 for tuning the
implementation but the proposed solution above does not solve anything
and will break a bunch of apps.
Andreas
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQGUBAEBAgAGBQJMmW78AAoJEADcfz7u4AZjs5MLv1TK/Y4gzX3SgnE0//ZPgx2g
FSMcLkli2AT2//83U6w9NTDxre4OEkpfCZVaQbMGAdoooCPBcgLCV1Mvd44aVI7Q
f6ff+1B22d+BxDEIJZOU05hFjJgxh4EQE7i8/wTVAoz+vq/HVbppXV5btu6Cyy9+
RsjdIUaqqIJ1FPDPAIG7BF+c2h+amcPZjoErN5/vxc4HSkNU4vOUG67bCh4Ea8eL
6RCiBT1Hr3zkNYGsdwc7m9fsTjz3jAi2o20qWCk0xeYaz0SQxvAkSPMrvbz3fL+P
GMAaP5tU/ZBbUgIGD6OfUlrx1uaTeoNKRP1tA1FavQPXYaIuPUVoTXhV9c/5gp9h
zZQbFQofSv8hX7PfDp3kGCiA11lBk6XAAiB7nO6PqdC/uDD1pqMuIa6h2Gzo71B6
kTTrKs2M9m7IKTwuq58SkGmJu7zUchnn7fcgXUjxv/T7z52tyN/rMemp5YaxG7O3
644kz3G4zelQpLeH5TOZwmaBcSWJ10g=
=ZPTw
-END PGP SIGNATURE-
<>___
Zope-Dev maillist - Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity
On Tue, Sep 21, 2010 at 07:39:46PM +0100, Chris Withers wrote:
> So, would anyone object if I change the default, on both trunk and the
> 2.12 branch to some variant of:
>
> class BasicEncodingConflictResolver(object):
> implements(IUnicodeEncodingConflictResolver)
>
> def resolve(self, context, text, expression):
> logging.warn('You should register an '
>'IUnicodeEncodingConflictResolver that matches '
>'your content')
> return text.decode('ascii','replace')
>
> ...and delete that devil spawn turd that is PreferredCharsetResolver?
If this means Zope2-the-application becomes unusable out of the box if
you've non-ASCII data in your database, then I'm -1.
Marius Gedminas
--
http://pov.lt/ -- Zope 3/BlueBream consulting and development
signature.asc
Description: Digital signature
___
Zope-Dev maillist - Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 First, cool down for a moment. Second, I discussed all those issues with my special friend in some Launchpad ticket years ago (+ a huge amount of private discussions). I can not recall all details but at the time of the integration of the Zope 3 ZPT engine the unicode resolver approach was the best thinkable solution for dealing all possible edgecase where encoding issues may pop *for the sake of backwards compatibility* and with all possible combinations where encodings have to be considered (including stuff like the management_page_charset property). The implementation is now working for years - so it can't be that bad. If you have a better implementation you should be able to override the utility or to fix the code directly. I won't comment after years on the details below (check Launchpad for the related ticket). Andreas Chris Withers wrote: > Hi All, > > It's been a a while since I had a good rant on a Zope list, but this > really takes the biscuit. > > Andreas, what on *earth* were you thinking with PreferredCharsetResolver?! > > I like the idea of IUnicodeEncodingConflictResolver, but > PreferredCharsetResolver is in the realms of totally batshit crazy. > > Why? well, because what *on earth* have the character sets accepted by > the user agent got to do with how strings are encoded in *my database*?! > > Nevermind the logic holes of the following: > > - reverting to management_page_charset *only* if there's no REQUEST > (there's always a REQUEST, this is Zope 2, the world collapses if there > isn't!) > > - returning the original text if the bizarre dance fails, violating the > contract of IUnicodeEncodingConflictResolver to *return unicode* > > How about the *insane performance overhead* of doing two utility lookups > *and* iterating over a load of encodings trying to find one that works > for *every single string substituition* done by a ZPT?! > > Why has no-one noticed this? Well, my guess is because when browsers are > explicitly supplying headers, as quite a few do (IE and Safari excepted, > who are legitimately going "we can handle anything you can throw at > us"), this moronic piece of code happily loops through them all, and no > doubt gets a hit on either utf-8 or latin-1, or, if you're really lucky, > ascii. > > To boot, when things go wrong, nobody suspects this miserable little > turd because it's hides itself nicely by just returning the original > text, leaving the bemused reader to wonder why some UA's fail and some > succeed, pointing the finger in totally the wrong direction (hence > Charlie's hacked up getPreferredCharsets and poor Vlad's desperate > attempts). > > Thankfully, I guess I can register an override that doesn't bark at the > moon and froth at the mouth... Here, have an invoice for the hours of my > life you've needlessly wasted... > > Farq, > > Chris ;-) > - -- ZOPYX Limited | zopyx group Charlottenstr. 37/1 | The full-service network for Zope & Plone D-72070 Tübingen| Produce & Publish www.zopyx.com | www.produce-and-publish.com - E-Publishing, Python, Zope & Plone development, Consulting -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQGUBAEBAgAGBQJMmRMqAAoJEADcfz7u4AZjThgLvjIWRhiVWoV/0IG7qadEliG0 Dk1+v+hu3ocu0DURn9XK5nGrHkkR+cFIjYdyjArDfXTqoNXWPii90QbtypK7Lce8 PD2rPp+fnwyMSne9iz9s/LtYYrDO/gbxWx9robjyjGmvICthKZD/qZk3cM47SSzq rhZLRV+r6GwVemn8sQEQ/y3gXKMgP7txz/Co1reWdWgrRH2HX4qpzvKHVLykXIrT lx4D9X0UfxGO+64p7MCEOgg41W3EzNM3qIEGp4PUTTwjcp/Use+AP5kS+Q6vQEWV n4c3RdaIX35EMNjVaunh0SMj+75rizMTarkGJ9e5lxekQLxqrveTiKqgGWtManD4 fjCSXpaobvy1Ym02CSLSLmSA4Y8ETTvdM87covKBvXHfaDyn5zxZaCmJ66L7Y+1i OwnbtFIke4tLFGhsEQDcbSBQDMD6TNuA1h/SQJ5AXAT3kSBPPvhWS/C8tWvoEdft dhzLEvXmpqMyA/4EgrMh6zP5fuGGkuM= =WUsF -END PGP SIGNATURE- <>___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity
Am 21.09.2010, 20:39 Uhr, schrieb Chris Withers :
> So, would anyone object if I change the default, on both trunk and the
> 2.12 branch to some variant of:
> class BasicEncodingConflictResolver(object):
> implements(IUnicodeEncodingConflictResolver)
> def resolve(self, context, text, expression):
> logging.warn('You should register an '
>'IUnicodeEncodingConflictResolver that matches '
>'your content')
> return text.decode('ascii','replace')
This sounds like a very good idea. Might benefit from some brief
documentation, to facilitate testing.
Charlie
--
Charlie Clark
Managing Director
Clark Consulting & Research
German Office
Helmholtzstr. 20
Düsseldorf
D- 40215
Tel: +49-211-600-3657
Mobile: +49-178-782-6226
___
Zope-Dev maillist - Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity
On 21/09/2010 19:32, Hanno Schlichting wrote:
> Because there's not all that many people left using plain Zope 2 and
> upgrading to new versions.
Well, or they assume they must be doing something wrong because of this
default.
> The rest is using Plone, which has its own version of this:
>
> def _unicode_replace(structure):
> if isinstance(structure, str):
> text = structure.decode('utf-8', 'replace')
> else:
> text = unicode(structure)
> return text
>
> class UTF8EncodingConflictResolver(object):
> implements(IUnicodeEncodingConflictResolver)
>
> def resolve(self, context, text, expression):
> return _unicode_replace(text)
Right, although the 'replace' is a lame cop out too ;-)
> We can do this, as we force manage_page_charset and
> default-zpublisher-encoding to be utf-8 and only support utf-8 and
> Unicode in the database.
Fundamentally, the developer must know what encoding is being used in
their data. So, they should be able to register an appropriate decoder
BUT...
> Zope 2 itself tries to support arbitrary encodings, at which point it
> can only guess badly or fail.
...having this default, which makes ABSOLUTELY NO SENSE in this
universe, or likely any other, is fucking crazy.
So, would anyone object if I change the default, on both trunk and the
2.12 branch to some variant of:
class BasicEncodingConflictResolver(object):
implements(IUnicodeEncodingConflictResolver)
def resolve(self, context, text, expression):
logging.warn('You should register an '
'IUnicodeEncodingConflictResolver that matches '
'your content')
return text.decode('ascii','replace')
...and delete that devil spawn turd that is PreferredCharsetResolver?
cheers,
Chris
--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
___
Zope-Dev maillist - Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity
On Tue, Sep 21, 2010 at 8:17 PM, Chris Withers wrote:
> I like the idea of IUnicodeEncodingConflictResolver, ...
[...]
> Why has no-one noticed this?
Because there's not all that many people left using plain Zope 2 and
upgrading to new versions.
The rest is using Plone, which has its own version of this:
def _unicode_replace(structure):
if isinstance(structure, str):
text = structure.decode('utf-8', 'replace')
else:
text = unicode(structure)
return text
class UTF8EncodingConflictResolver(object):
implements(IUnicodeEncodingConflictResolver)
def resolve(self, context, text, expression):
return _unicode_replace(text)
We can do this, as we force manage_page_charset and
default-zpublisher-encoding to be utf-8 and only support utf-8 and
Unicode in the database.
Zope 2 itself tries to support arbitrary encodings, at which point it
can only guess badly or fail.
Hanno
___
Zope-Dev maillist - Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity
Am 21.09.2010, 20:17 Uhr, schrieb Chris Withers : > To boot, when things go wrong, nobody suspects this miserable little > turd because it's hides itself nicely by just returning the original > text, leaving the bemused reader to wonder why some UA's fail and some > succeed, pointing the finger in totally the wrong direction (hence > Charlie's hacked up getPreferredCharsets and poor Vlad's desperate > attempts). Chris, rant aside it sounds like you've identified a bug. Can you submit a bug so we can hopefully fix it in 2.13? Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226 ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] IUnicodeEncodingConflictResolver moronosity
Hi All, It's been a a while since I had a good rant on a Zope list, but this really takes the biscuit. Andreas, what on *earth* were you thinking with PreferredCharsetResolver?! I like the idea of IUnicodeEncodingConflictResolver, but PreferredCharsetResolver is in the realms of totally batshit crazy. Why? well, because what *on earth* have the character sets accepted by the user agent got to do with how strings are encoded in *my database*?! Nevermind the logic holes of the following: - reverting to management_page_charset *only* if there's no REQUEST (there's always a REQUEST, this is Zope 2, the world collapses if there isn't!) - returning the original text if the bizarre dance fails, violating the contract of IUnicodeEncodingConflictResolver to *return unicode* How about the *insane performance overhead* of doing two utility lookups *and* iterating over a load of encodings trying to find one that works for *every single string substituition* done by a ZPT?! Why has no-one noticed this? Well, my guess is because when browsers are explicitly supplying headers, as quite a few do (IE and Safari excepted, who are legitimately going "we can handle anything you can throw at us"), this moronic piece of code happily loops through them all, and no doubt gets a hit on either utf-8 or latin-1, or, if you're really lucky, ascii. To boot, when things go wrong, nobody suspects this miserable little turd because it's hides itself nicely by just returning the original text, leaving the bemused reader to wonder why some UA's fail and some succeed, pointing the finger in totally the wrong direction (hence Charlie's hacked up getPreferredCharsets and poor Vlad's desperate attempts). Thankfully, I guess I can register an override that doesn't bark at the moon and froth at the mouth... Here, have an invoice for the hours of my life you've needlessly wasted... Farq, Chris ;-) -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
