Marc-Andre Lemburg <m...@egenix.com> added the comment: Amaury Forgeot d'Arc wrote: > I feel uneasy to change the default unicode-escape encoding. > I think that we mix two features here; to transfer a unicode string between > two points, programs must agree on where the data ends, and how characters > are represented as bytes. > All codecs including unicode-escape only dealt with byte conversion; > (unicode-escape converts everything to printable 7bit ascii); > these patches want to add a feature related to the "where does the string > end" issue, and is only aimed at "python code" containers. Other transports > and protocols may choose different delimiters. > > My point is that unicode-escape used to not change printable 7-bit ascii > characters, and the patches will change this. > > And actually this will break existing code. It did not take me long to find > two examples of programs which embed unicode_escape-encoded text between > quotes, and take care themselves of escaping quotes. First example generates > javascript code, the second generates SQL statements: > http://github.com/chriseppstein/pywebmvc/blob/master/src/code/pywebmvc/tools/searchtool.py#L450 > http://gitweb.sabayon.org/?p=entropy.git;a=blob;f=libraries/entropy/db/__init__.py;h=2d818455efa347f35b2e96d787fefd338055d066;hb=HEAD#l6463
Ouch... these codecs should not have been used outside Python. I wonder why these applications don't use repr(text) to format the JavaScript/SQL strings. I guess this is the result of documenting them in http://docs.python.org/library/codecs.html#standard-encodings Too bad that the docs actually say "Produce a string that is suitable as Unicode literal in Python source code." The codecs main intent was to *decode* Unicode literals in Python source code to Unicode objects... The naming in the examples you mention also suggest that the programmers used the table from the docs - they use "unicode_escape" as codec name, not the standard "unicode-escape" name which we use throughout the Python code. The fact that the demonstrated actual use already does apply the extra quote escaping suggests that we cannot easily add this to the existing codecs. It would break those applications, since they'd be applying double-escaping. > This does not prevent the creation of a new codec, let's call it > 'python-unicode-escape' [ or 'repr' :-) ] I think that's a good idea to move forward. Python 3.x comes with a new Unicode repr() format which we could turn into a new codec: it automatically adds the quotes, processes the in-string quotes and backslashes and also escapes \t, \n and \r as well as all non-printable code points. As for naming the new codec, I'd suggest "unicode-repr" since that's what it implements. ---------- _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue7615> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
