[issue45472] Add public C API for partial "unicode-escape" and "raw-unicode-escape" decoding

Change by Serhiy Storchaka : -- keywords: +patch pull_requests: +27244 stage: -> patch review pull_request: https://github.com/python/cpython/pull/28955 ___ Python tracker

[issue45472] Add public C API for partial "unicode-escape" and "raw-unicode-escape" decoding

New submission from Serhiy Storchaka : The proposed PR adds C API functions PyUnicode_DecodeUnicodeEscapeStateful() and PyUnicode_DecodeRawUnicodeEscapeStateful() for partially decoding with "unicode-escape" and "raw-unicode-escape" encodings. It were the only multibyte t

[issue28750] Replace string with bytes in doc of unicode-escape an raw-unicode-escape

Changes by Serhiy Storchaka : -- resolution: -> fixed stage: commit review -> resolved status: open -> closed ___ Python tracker

[issue28750] Replace string with bytes in doc of unicode-escape an raw-unicode-escape

Roundup Robot added the comment: New changeset 059b8e15b738 by Serhiy Storchaka in branch '3.5': Issue #28750: Fixed docs of of unicode-escape an raw-unicode-escape C API. https://hg.python.org/cpython/rev/059b8e15b738 New changeset 0c6fccf04a79 by Serhiy Storchaka in branch '3.6': Issue #28750

[issue28750] Replace string with bytes in doc of unicode-escape an raw-unicode-escape

Serhiy Storchaka added the comment: LGTM. -- assignee: docs@python -> serhiy.storchaka ___ Python tracker ___

[issue28750] Replace string with bytes in doc of unicode-escape an raw-unicode-escape

Julien Palard added the comment: So, lgtm. -- ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue28750] Replace string with bytes in doc of unicode-escape an raw-unicode-escape

Xiang Zhang added the comment: > But I think the word "Python" in "Python bytes object" is redundant. It was > needed in "Python string object" to distinguish from "C string" and "Python > Unicode object". Make sense. This "Pyt

[issue28750] Replace string with bytes in doc of unicode-escape an raw-unicode-escape

Serhiy Storchaka added the comment: For PyUnicode_AsCharmapString and PyUnicode_EncodeCharmap see issue28749. -- ___ Python tracker ___

[issue28750] Replace string with bytes in doc of unicode-escape an raw-unicode-escape

Serhiy Storchaka added the comment: Good catch Xiang! But I think the word "Python" in "Python bytes object" is redundant. It was needed in "Python string object" to distinguish from "C string" and "Python Unicode

[issue28750] Replace string with bytes in doc of unicode-escape an raw-unicode-escape

string() which was used to implement PyUnicode_EncodeUnicodeEscape() too), because once repr() is required to return unicode objects it needs its own implementation anyway. (PyUnicode_EncodeUnicodeEscape was the old name for PyUnicode_AsUnicodeEscapeString

[issue28750] Replace string with bytes in doc of unicode-escape an raw-unicode-escape

Changes by Emanuel Barry : -- stage: patch review -> commit review ___ Python tracker ___

[issue28750] Replace string with bytes in doc of unicode-escape an raw-unicode-escape

Changes by Xiang Zhang <angwe...@126.com>: Added file: http://bugs.python.org/file45559/unicode-escape-doc_v2.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python

[issue28750] Replace string with bytes in doc of unicode-escape an raw-unicode-escape

Changes by Emanuel Barry : -- nosy: +ebarry ___ Python tracker ___ ___ Python-bugs-list

[issue28750] Replace string with bytes in doc of unicode-escape an raw-unicode-escape

New submission from Xiang Zhang: The docs of the encoders of unicode-escape and raw-unicode-escape still tell the result of the encoding is Python string object. It should be Python bytes object. -- assignee: docs@python components: Documentation files: unicode-escape-doc.patch

[issue16334] Faster unicode-escape and raw-unicode-escape codecs

STINNER Victor added the comment: Feel free to document the change. It's not my patch, it's yours :-) -- ___ Python tracker ___

[issue16334] Faster unicode-escape and raw-unicode-escape codecs

Serhiy Storchaka added the comment: Thanks Victor! I benchmarked your patch. There is no regression in comparison with my patch. In few cases your patch is even faster! UnpatchedPatch v.4Patch v.5 148 (+76%) 235 (+11%) 260decode unicode-escape 'A'*1 197 (+30%) 257 (+0

[issue16334] Faster unicode-escape and raw-unicode-escape codecs

STINNER Victor added the comment: Since it's almost the 3.6 beta 1, I chose to push the change right now. I'm sure that it's faster, I trust your benchmarks ;-) Thanks Serhiy for this nice enhancement. > > * Rename WRITECHAR macro to WRITE_ASCII_CHAR() > This is not correct name. This macro

[issue16334] Faster unicode-escape and raw-unicode-escape codecs

Roundup Robot added the comment: New changeset ad5a28ace615 by Victor Stinner in branch 'default': Optimize unicode_escape and raw_unicode_escape https://hg.python.org/cpython/rev/ad5a28ace615 -- nosy: +python-dev ___ Python tracker

[issue16334] Faster unicode-escape and raw-unicode-escape codecs

STINNER Victor added the comment: > Did you benchmark this change? I afraid that this inflates execution code > size and can have negative impact on the performance. I consider that readability (maintainability) matters more than such micro optimization. --

[issue16334] Faster unicode-escape and raw-unicode-escape codecs

Serhiy Storchaka added the comment: > Unicode escape encodecs were modified by the issue #25353 to use the > _PyBytesWriter API. Sadly, I didn't benchmark my change before pushing it > :-/ You can benchmark it now by checking out revisions with your patch and just before.

[issue16334] Faster unicode-escape and raw-unicode-escape codecs

only returns valid unicode characters (<= MAX_UNICODE) * For \U format: since MAX_UNICODE is 0x10, hardcode the first two digits as 0, and add an assertion on MAX_UNICODE value * PEP 7: add {...} on if/else blocks -- Added file: http://bugs.python.org/file44348/faster

[issue16334] Faster unicode-escape and raw-unicode-escape codecs

STINNER Victor added the comment: Unicode escape encodecs were modified by the issue #25353 to use the _PyBytesWriter API. Sadly, I didn't benchmark my change before pushing it :-/ Your patch basically reverts my change. > Py3.2Py3.3Py3.6Py3.6+patch > 195 (+136%

[issue16334] Faster unicode-escape and raw-unicode-escape codecs

Serhiy Storchaka added the comment: Victor's patch harvested most fruits, but there is a place for further optimization. Benchmark results for new patch: Py3.2Py3.3Py3.6Py3.6+patch 451 (-47%) 77 (+209%) 140 (+70%) 238decode unicode-escape 'A'*1 269

[issue25353] Use _PyBytesWriter for unicode escape and raw unicode escape encoders

Roundup Robot added the comment: New changeset 8e27f8398a4f by Victor Stinner in branch 'default': Issue #25353: Optimize unicode escape and raw unicode escape encoders to use https://hg.python.org/cpython/rev/8e27f8398a4f -- nosy: +python-dev

[issue25353] Use _PyBytesWriter for unicode escape and raw unicode escape encoders

Changes by STINNER Victor : -- resolution: -> fixed status: open -> closed ___ Python tracker ___

[issue25353] Use _PyBytesWriter for unicode escape and raw unicode escape encoders

STINNER Victor added the comment: A few more encoders should be updated to use _PyBytesWriter API: * Code Page (Windows only) * Charmap * UTF-7 * UTF-16 * UTF-32 -- ___ Python tracker

[issue25353] Use _PyBytesWriter for unicode escape and raw unicode escape encoders

STINNER Victor added the comment: The _PyBytesWriter API was added in the issue #25318. See also the issue #25349 which optimized bytes % args. -- ___ Python tracker

[issue25353] Use _PyBytesWriter for unicode escape and raw unicode escape encoders

New submission from STINNER Victor: Attached patch modifies unicode escape and raw unicode escape encoders to use the new _PyBytesWriter API. The patch is optimized to encode Latin1 characters: encoding Latin1 characters when no character is escaped should not have to call _PyByte_Resize

[issue16334] Faster unicode-escape and raw-unicode-escape codecs

STINNER Victor added the comment: unicode-escape and raw-unicode-escape decoders now use the PyUnicodeWriter API. Can you please compare performances of your patch to PyUnicodeWriter API? Decoders overallocate the buffer. According to a comment in the decoder, overallocating is never needed

[issue16334] Faster unicode-escape and raw-unicode-escape codecs

New submission from Serhiy Storchaka: The proposed patch optimizes unicode-escape and raw-unicode-escape codecs. Coders still slower than in 3.2, but much faster than in 3.3. Further speedup is possible with the use of stringlib, but I think that this is enough. The code unified

Convert unicode escape sequences to unicode in a file

I have a file that has unicode escape sequences, i.e., J\u00e9r\u00f4me and I want to replace all of them in a file and write the results to a new file. The simple script I've created is copied below. However, I am getting the following error: UnicodeEncodeError: 'ascii' codec can't encode

Re: Convert unicode escape sequences to unicode in a file

On Jan 11, 8:53 pm, Jeremy jlcon...@gmail.com wrote: I have a file that has unicode escape sequences, i.e., J\u00e9r\u00f4me and I want to replace all of them in a file and write the results to a new file.  The simple script I've created is copied below.  However, I am getting

Re: Convert unicode escape sequences to unicode in a file

a Unicode escape sequence. Open the file using a text editor or hex editor and look at the value at offset 947 to be sure. If so, you need to replace 'unicode-escape' with the actual encoding of the file. Yeah, I'm sure that's what the file contains. In fact, I solved my own problem while

Re: To unicode or not to unicode

re: You should never have to rely on the default encoding. You should explicitly decode and encode data. What is the best practice for 1) doing this in Python and 2) for unicode support ? I want to standardize on unicode and want to put into place best Python practice so that we don't have

Re: To unicode or not to unicode

On Sun, Feb 22, 2009 at 1:39 AM, Ross Ridge rri...@csclub.uwaterloo.ca wrote: Ross Ridge (Sat, 21 Feb 2009 18:06:35 -0500) I understand what Unicode and MIME are for and why they exist. Neither their merits nor your insults change the fact that the only current standard governing the content

choosing a default text-encoding in Python programs (was: To unicode or not to unicode)

Denis Kasak denis.ka...@gmail.com writes: Python assumes ASCII and if the decodes/encoded text doesn't fit that encoding it refuses to guess. Which is reasonable given that Python is programming language where it's better to have more conservative assumption about encodings so errors

Re: choosing a default text-encoding in Python programs (was: To unicode or not to unicode)

character sets were deliberately restricted to scripts that would fit in 8 bits. So Chinese, Japanese, Korean and Vietnamese aren't included. Note that Chinese and Japanese already each had *multiple* legacy (i.e. non-Unicode) character sets ... they (and the rest the world) don't want/need yet another

Re: To unicode or not to unicode

=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?= mar...@v.loewis.de wrote: I don't think that was the complaint. Instead, the complaint was that the OP's original message did not have a Content-type header, and that it was thus impossible to tell what the byte in front of Wiki meant. To properly post either

Re: To unicode or not to unicode

* Ross Ridge (Sat, 21 Feb 2009 12:22:36 -0500) =?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?= mar...@v.loewis.de wrote: I don't think that was the complaint. Instead, the complaint was that the OP's original message did not have a Content-type header, and that it was thus impossible to tell what the

Re: To unicode or not to unicode

Thorsten Kampe thors...@thorstenkampe.de wrote: RFC 1036 doesn't require nor give a meaning to a Content-Type header in a Usenet message Well, /maybe/ the reason for that is that RFC 1036 was written in 1987 and the first MIME RFC in 1992...? Obviously. Son of RFC 1036 mentions MIME more

Re: To unicode or not to unicode

: http://groups.google.com/group/comp.lang.python/msg/828fefd7040238bc Your understanding of the principles of Unicode is as least as non- existant as the OP's. I could just as easily argue that assuming ISO 8859-1 is the defacto standard, and that its your newsreader that's broken

Re: To unicode or not to unicode

of Unicode is as least as non- existant as the OP's. The link demonstrates that Google Groups doesn't assume ASCII like Python does. Since popular newsreaders like Google Groups and Outlook Express can display the message correctly without the MIME headers, but your obscure one can't, there's a much

Re: To unicode or not to unicode

unicode encoding on my Python installation is ascii.  So I'm trying to decide on a course of action.  There seem to be three possibilities: 1.  Change the code to properly support unicode.  Preliminary investigations indicate that this is going to be a colossal pain in the ass. 2.  Change

Re: To unicode or not to unicode

a much stronger case to made that it's your newsreader that's broken. *sigh* I give up on you. You didn't even read the Joel on Software article. The whole why and what for of Unicode and MIME will always be a complete mystery to you. T. -- http://mail.python.org/mailman/listinfo/python-list

Re: To unicode or not to unicode

stronger case to made that it's your newsreader that's broken. Thorsten Kampe thors...@thorstenkampe.de wrote: *sigh* I give up on you. You didn't even read the Joel on Software article. The whole why and what for of Unicode and MIME will always be a complete mystery to you. I understand what

Re: To unicode or not to unicode

a much stronger case to made that it's your newsreader that's broken. Thorsten Kampe thors...@thorstenkampe.de wrote: *sigh* I give up on you. You didn't even read the Joel on Software article. The whole why and what for of Unicode and MIME will always be a complete mystery to you. I

Re: To unicode or not to unicode

Ross Ridge (Sat, 21 Feb 2009 18:06:35 -0500) I understand what Unicode and MIME are for and why they exist. Neither their merits nor your insults change the fact that the only current standard governing the content of Usenet posts doesn't require their use. Thorsten Kampe thors

Re: To unicode or not to unicode

* Ross Ridge (Sat, 21 Feb 2009 19:39:42 -0500) Thorsten Kampe thors...@thorstenkampe.de wrote: That's right. As long as you use pure ASCII you can skip this nasty step of informing other people which charset you are using. If you do use non ASCII then you have to do that. That's the way

Re: To unicode or not to unicode

it correctly too: http://groups.google.com/group/comp.lang.python/msg/828fefd7040238bc Your understanding of the principles of Unicode is as least as non- existant as the OP's. I could just as easily argue that assuming ISO 8859-1 is the defacto standard, and that its your

Re: To unicode or not to unicode

Since when is Google Groups a newsreader? So far as I know, all the display/formatting is handled by my web browser and GG merely stuffs messages into an HTML wrapper... It also transmits this HTML wrapper via HTTP, where it claims that the charset of the HTML is UTF-8. To do that, it

Re: To unicode or not to unicode

Ross Ridge rri...@csclub.uwaterloo.ca writes: It's all about declaring your charset. In Python as well as in your newsreader. If you don't declare your charset it's ASCII for you - in Python as well as in your newsreader. Except in practice unlike Python, many newsreaders don't assume

Re: To unicode or not to unicode

* Ron Garret (Thu, 19 Feb 2009 18:57:13 -0800) I'm writing a little wiki that I call µWiki. That's a lowercase Greek mu at the beginning (it's pronounced micro-wiki). No, it's not. I suggest you start your Unicode adventure by configuring your newsreader. Thorsten -- http://mail.python.org

Re: To unicode or not to unicode

Thorsten Kampe wrote: * Ron Garret (Thu, 19 Feb 2009 18:57:13 -0800) I'm writing a little wiki that I call µWiki. That's a lowercase Greek mu at the beginning (it's pronounced micro-wiki). No, it's not. I suggest you start your Unicode adventure by configuring your newsreader. It looked

Re: To unicode or not to unicode

-wiki). No, it's not. I suggest you start your Unicode adventure by configuring your newsreader. It looked like mu to me, but you're correct: it's MICRO SIGN, not GREEK SMALL LETTER MU. Heh, I didn't know that those two things were distinct. Learn something new every day. rg -- http

Re: To unicode or not to unicode

MRAB wrote: Thorsten Kampe wrote: * Ron Garret (Thu, 19 Feb 2009 18:57:13 -0800) I'm writing a little wiki that I call µWiki. That's a lowercase Greek mu at the beginning (it's pronounced micro-wiki). No, it's not. I suggest you start your Unicode adventure by configuring your newsreader

Re: To unicode or not to unicode

). No, it's not. I suggest you start your Unicode adventure by configuring your newsreader. It looked like mu to me, but you're correct: it's MICRO SIGN, not GREEK SMALL LETTER MU. I don't think that was the complaint. Instead, the complaint was that the OP's original message did

Re: To unicode or not to unicode

Ron Garret wrote: In article 499f0cf0.8070...@v.loewis.de, Martin v. Löwis mar...@v.loewis.de wrote: I'm the OP. I'm using MT-Newswatcher 3.5.1. I thought I had it configured properly, but I guess I didn't. Probably you did. However, it then means that the newsreader is crap. Under

To unicode or not to unicode

I'm writing a little wiki that I call µWiki. That's a lowercase Greek mu at the beginning (it's pronounced micro-wiki). It's working, except that I can't actually enter the name of the wiki into the wiki itself because the default unicode encoding on my Python installation is ascii. So I'm

Re: To unicode or not to unicode

Ron Garret rNOSPAMon at flownet.com writes: I'm writing a little wiki that I call µWiki. That's a lowercase Greek mu at the beginning (it's pronounced micro-wiki). It's working, except that I can't actually enter the name of the wiki into the wiki itself because the default unicode

Re: env parameter to popen won't accept Unicode on Windows - minor Unicode bug

John Nagle wrote: Benjamin wrote: On Jan 14, 6:26 pm, Bjoern Schliessmann usenet- [EMAIL PROTECTED] wrote: John Nagle wrote: It turns out that the strings in the env parameter have to be ASCII, not Unicode, even though Windows fully supports Unicode in CreateProcess. That's of course

RE: env parameter to popen won't accept Unicode on Windows -minor Unicode bug

Diez B. Roggisch wrote: Sure thing, python will just magically convert unicode to the encoding the program YOU invoke will expect. Right after we introduced the solve_my_problem() built-in-function. Any other wishes? There's no reason to be rude. Anyway, at least on Windows it makes

RE: env parameter to popen won't accept Unicode on Windows -minor Unicode bug

Brian Smith wrote: Diez B. Roggisch wrote: Sure thing, python will just magically convert unicode to the encoding the program YOU invoke will expect. Right after we introduced the solve_my_problem() built-in-function. Any other wishes? There's no reason to be rude. If you'd know

RE: env parameter to popen won't accept Unicode on Windows -minor Unicode bug

Brian Smith wrote: popen() knows that it is running on Windows, and it knows what encoding Windows needs for its environment (it's either UCS2 or UTF-16 for most Windows APIs). At least when it receives a unicode string, it has enough information to apply the conversion automatically

Re: env parameter to popen won't accept Unicode on Windows - minor Unicode bug

Diez B. Roggisch wrote: John Nagle wrote: Benjamin wrote: On Jan 14, 6:26 pm, Bjoern Schliessmann usenet- [EMAIL PROTECTED] wrote: John Nagle wrote: It turns out that the strings in the env parameter have to be ASCII, not Unicode, even though Windows fully supports Unicode

Re: env parameter to popen won't accept Unicode on Windows -minor Unicode bug

Diez B. Roggisch wrote: Brian Smith wrote: Diez B. Roggisch wrote: Sure thing, python will just magically convert unicode to the encoding the program YOU invoke will expect. Right after we introduced the solve_my_problem() built-in-function. Any other wishes? There's no reason

Re: env parameter to popen won't accept Unicode on Windows -minor Unicode bug

John Nagle wrote: The problem is that only the NT-derived Microsoft systems talk Unicode. The DOS/Win16/Win9x family did not. But they did have CreateProcess. So the current code will handle Win9x, but not Unicode. Please explain, I don't understand. If you try using Windows system functions

env parameter to popen won't accept Unicode on Windows - minor Unicode bug

I passed a dict for the env variable to Popen with Unicode strings for the dictionary values. Got: File D:\Python24\lib\subprocess.py, line 706, in _execute_child TypeError: environment can only contain strings It turns out that the strings in the env parameter have

Re: env parameter to popen won't accept Unicode on Windows - minor Unicode bug

John Nagle wrote: It turns out that the strings in the env parameter have to be ASCII, not Unicode, even though Windows fully supports Unicode in CreateProcess. Are you sure it supports Unicode, not UTF8 or UTF16? Probably using something like uthestring.encode(utf16) will help. Regards

Re: env parameter to popen won't accept Unicode on Windows - minor Unicode bug

On Jan 14, 6:26 pm, John Nagle [EMAIL PROTECTED] wrote: I passed a dict for the env variable to Popen with Unicode strings for the dictionary values. Got: File D:\Python24\lib\subprocess.py, line 706, in _execute_child TypeError: environment can only contain strings

Re: env parameter to popen won't accept Unicode on Windows - minor Unicode bug

On Jan 14, 6:26 pm, Bjoern Schliessmann usenet- [EMAIL PROTECTED] wrote: John Nagle wrote: It turns out that the strings in the env parameter have to be ASCII, not Unicode, even though Windows fully supports Unicode in CreateProcess. Are you sure it supports Unicode, not UTF8 or UTF16

Re: env parameter to popen won't accept Unicode on Windows - minor Unicode bug

Benjamin wrote: On Jan 14, 6:26 pm, Bjoern Schliessmann usenet- [EMAIL PROTECTED] wrote: John Nagle wrote: It turns out that the strings in the env parameter have to be ASCII, not Unicode, even though Windows fully supports Unicode in CreateProcess. Are you sure it supports Unicode

Getting unicode escape sequence from unicode character?

Given a Python unicode character (string of length one), how would I find out the \u escape sequence for it? This isn't obvious from the docs I've been looking through. Thanks, Ken -- http://mail.python.org/mailman/listinfo/python-list

Re: Getting unicode escape sequence from unicode character?

Kenneth McDonald wrote: Given a Python unicode character (string of length one), how would I find out the \u escape sequence for it? This isn't obvious from the docs I've been looking through. You can use the ord builtin, or the encode method with unicode_escape: a = u'\u1234' a u