Re: Python & Unicode decimal interpretation
Scott David Daniels wrote:
> In reading over the source for CPython's PyUnicode_EncodeDecimal,
> I see a dance to handle characters which are neither dec-equiv nor
> in Latin-1. Does anyone know about the intent of such a conversion?
To support this:
>>> int(u"\N{DEVANAGARI DIGIT SEVEN}")
7
> As far as I can tell, error handling is one of:
> strict, replace, ignore, xmlcharrefreplace, or something_else
> What I don't understand is whether, in the ignore or something_else
> cases, there is any chance that digits will show up anywhere that
> they would not if these characters were treated as a character like '?'?
>
> Can someone either give me definitive "why not" or (preferably) give
> me a test case that shows where that interpretation does not hold.
In the "ignore" case, no output is produced at all, for the unencodable
character; this is the same way that '?' would be treated (it is
also unencodable).
In the something_else case, a user-defined exception handler could
treat the error in any way it liked, e.g. encoding all letters
u'A' to digit '0'. This might be different from the way this error
handler would treat '?'.
Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python & Unicode decimal interpretation
Martin v. Löwis wrote:
> Scott David Daniels wrote:
>> In reading over the source for CPython's PyUnicode_EncodeDecimal,
>> I see a dance to handle characters which are neither dec-equiv nor
>> in Latin-1. Does anyone know about the intent of such a conversion?
>
> To support this:
>
> >>> int(u"\N{DEVANAGARI DIGIT SEVEN}")
> 7
OK, That much I have handled. I am fiddling with direct-to-number
conversions and wondering about cases like
>>> int(u"\N{DEVANAGARI DIGIT SEVEN}" + XXX
+ u"\N{DEVANAGARI DIGIT SEVEN}")
Where XXX does not pass the digit test, but must either:
(A) be dropped, giving a result of 77
or (B) get translated (e.g. to u'234') giving 72347
or (C) get translated (to u'2' + YYY + u'4') where YYY will
require further handling ...
I don't really understand how the "ignore" or "something_else"
cases get caused by python source [where they come from]. Are they
only there for C-program access?
> In the "ignore" case, no output is produced at all, for the unencodable
> character; this is the same way that '?' would be treated (it is
> also unencodable).
If I understand you correctly -- I can consider the digit stream to stop
as soon as I hit a non-digit (except for handling bases 11-36).
> In the something_else case, a user-defined exception handler could
> treat the error in any way it liked, e.g. encoding all letters
> u'A' to digit '0'. This might be different from the way this error
> handler would treat '?'.
--Scott David Daniels
[EMAIL PROTECTED]
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python & Unicode decimal interpretation
Scott David Daniels wrote:
>> >>> int(u"\N{DEVANAGARI DIGIT SEVEN}")
>> 7
>
> OK, That much I have handled. I am fiddling with direct-to-number
> conversions and wondering about cases like
>>>> int(u"\N{DEVANAGARI DIGIT SEVEN}" + XXX
>+ u"\N{DEVANAGARI DIGIT SEVEN}")
int() passes NULL as error mode, equalling strict. So if you get an
unencodable character, you get the UnicodeError.
> I don't really understand how the "ignore" or "something_else"
> cases get caused by python source [where they come from]. Are they
> only there for C-program access?
Neither, nor. This code is dead.
>> In the "ignore" case, no output is produced at all, for the unencodable
>> character; this is the same way that '?' would be treated (it is
>> also unencodable).
>
> If I understand you correctly -- I can consider the digit stream to stop
> as soon as I hit a non-digit (except for handling bases 11-36).
No. In "ignore" mode, a codec doesn't stop at the unencodable character.
Instead, it skips it, continuing with the next character.
I mistakenly said that this would happen to '?' (question mark) also;
this is incorrect: PyUnicode_EncodeDecimal copies all Latin-1 characters
to the output, latin-1-encoded. So '?' would appear in the output,
even in "ignore" mode.
Handling of bases is not done in the function at all. Instead, the
callers of PyUnicode_EncodeDecimal will deal with number formats
(base, prefix, exponent syntax, etc.) They will assume ASCII
bytes.
Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list
