Re: 'Straße' ('Strasse') and Python 2
On Thu, Jan 16, 2014 at 11:43 AM, Steven D'Aprano wrote: > Worse, linguists sometimes disagree as to what counts as a grapheme. For > instance, some authorities consider the English "sh" to be a separate > grapheme. As a native English speaker, I'm not sure about that. Certainly > it isn't a separate letter of the alphabet, but on the other hand I can't > think of any words containing "sh" that should be considered as two > graphemes "s" followed by "h". Wait, no, that's not true... compound > words such as "glasshouse" or "disheartened" are counter examples. Digression: When I was taught basic English during my school days, my mum used Spalding's book and the 70 phonograms. 25 of them are single letters (Q is not a phonogram - QU is), and the others are mostly pairs (there are a handful of 3- and 4-letter phonograms). Not every instance of "s" followed by "h" is the phonogram "sh" - only the times when it makes the single sound "sh" (which it doesn't in "glasshouse" or "disheartened"). Thing is, you can't define spelling and pronunciation in terms of each other, because you'll always be bitten by corner cases. Everyone knows how "Thames" is pronounced... right? Well, no. There are (at least) two rivers of that name, the famous one in London p1[ and another one further north [2]. The obscure one is pronounced the way the word looks, the famous one isn't. And don't even get started on English family names... Majorinbanks, Meux and Cholmodeley, as lampshaded [3] in this song [4]! Even without names, though, there are the tricky cases and the ones where different localities pronounce the same word very differently; Unicode shouldn't have to deal with that by changing whether something's a single character or two. Considering that phonograms aren't even ligatures (though there is overlap, eg "Th"), it's much cleaner to leave them as multiple characters. ChrisA [1] https://en.wikipedia.org/wiki/River_Thames [2] Though it's better known as the Isis. https://en.wikipedia.org/wiki/The_Isis [3] http://tvtropes.org/pmwiki/pmwiki.php/Main/LampshadeHanging [4] http://www.stagebeauty.net/plays/th-arca2.html - "Mosh-banks", "Mow", and "Chumley" are the pronunciations used -- https://mail.python.org/mailman/listinfo/python-list
Re: 'Straße' ('Strasse') and Python 2
On Jan 15, 2014, at 4:50 AM, Robin Becker wrote:
> On 15/01/2014 12:13, Ned Batchelder wrote:
>
>>> On my utf8 based system
>>>
>>>
robin@everest ~:
$ cat ooo.py
if __name__=='__main__':
import sys
s='A̅B'
print('version_info=%s\nlen(%s)=%d' % (sys.version_info,s,len(s)))
robin@everest ~:
$ python ooo.py
version_info=sys.version_info(major=3, minor=3, micro=3,
releaselevel='final', serial=0)
len(A̅B)=3
robin@everest ~:
$
>>>
>>>
>
>> You are right that more than one codepoint makes up a grapheme, and that
>> you'll
>> need code to deal with the correspondence between them. But let's not muddy
>> these already confusing waters by referring to that mapping as an encoding.
>>
>> In Unicode terms, an encoding is a mapping between codepoints and bytes.
>> Python
>> 3's str is a sequence of codepoints.
>>
> Semantics is everything. For me graphemes are the endpoint (or should be); to
> get a proper rendering of a sequence of graphemes I can use either a sequence
> of bytes or a sequence of codepoints. They are both encodings of the
> graphemes; what unicode says is an encoding doesn't define what encodings are
> ie mappings from some source alphabet to a target alphabet.
But you’re talking about two levels of encoding. One runs on top of the other.
So insisting that you be able to call them all encodings, makes the term
pointless, because now it’s ambiguous as to what you’re referring to. Are you
referring to encoding in the sense of representing code points with bytes? Or
are you referring to what the unicode guys call “forms”?
For example, the NFC form of ‘ñ’ is ’\u00F1’. ‘nThe NFD form represents the
exact same grapheme, but is ‘\u006e\u0303’. You can call them encodings if you
want, but I echo Ned’s sentiment that you keep that to yourself.
Conventionally, they’re different forms, not different encodings. You can
encode either form with an encoding, e.g.
'\u00F1'.encode('utf8’)
'\u00F1'.encode('utf16’)
'\u006e\u0303'.encode('utf8’)
'\u006e\u0303'.encode('utf16')
--
https://mail.python.org/mailman/listinfo/python-list
Re: 'Straße' ('Strasse') and Python 2
On 15/01/2014 12:13, Ned Batchelder wrote:
On my utf8 based system
robin@everest ~:
$ cat ooo.py
if __name__=='__main__':
import sys
s='A̅B'
print('version_info=%s\nlen(%s)=%d' % (sys.version_info,s,len(s)))
robin@everest ~:
$ python ooo.py
version_info=sys.version_info(major=3, minor=3, micro=3,
releaselevel='final', serial=0)
len(A̅B)=3
robin@everest ~:
$
You are right that more than one codepoint makes up a grapheme, and that you'll
need code to deal with the correspondence between them. But let's not muddy
these already confusing waters by referring to that mapping as an encoding.
In Unicode terms, an encoding is a mapping between codepoints and bytes. Python
3's str is a sequence of codepoints.
Semantics is everything. For me graphemes are the endpoint (or should be); to
get a proper rendering of a sequence of graphemes I can use either a sequence of
bytes or a sequence of codepoints. They are both encodings of the graphemes;
what unicode says is an encoding doesn't define what encodings are ie mappings
from some source alphabet to a target alphabet.
--
Robin Becker
--
https://mail.python.org/mailman/listinfo/python-list
Re: 'Straße' ('Strasse') and Python 2
On 1/15/14 7:00 AM, Robin Becker wrote:
On 12/01/2014 07:50, wxjmfa...@gmail.com wrote:
sys.version
2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)]
s = 'Straße'
assert len(s) == 6
assert s[5] == 'e'
jmf
On my utf8 based system
robin@everest ~:
$ cat ooo.py
if __name__=='__main__':
import sys
s='A̅B'
print('version_info=%s\nlen(%s)=%d' % (sys.version_info,s,len(s)))
robin@everest ~:
$ python ooo.py
version_info=sys.version_info(major=3, minor=3, micro=3,
releaselevel='final', serial=0)
len(A̅B)=3
robin@everest ~:
$
so two 'characters' are 3 (or 2 or more) codepoints. If I want to
isolate so called graphemes I need an algorithm even for python's
unicode ie when it really matters, python3 str is just another encoding.
You are right that more than one codepoint makes up a grapheme, and that
you'll need code to deal with the correspondence between them. But let's
not muddy these already confusing waters by referring to that mapping as
an encoding.
In Unicode terms, an encoding is a mapping between codepoints and bytes.
Python 3's str is a sequence of codepoints.
--
Ned Batchelder, http://nedbatchelder.com
--
https://mail.python.org/mailman/listinfo/python-list
Re: 'Straße' ('Strasse') and Python 2
On 12/01/2014 07:50, wxjmfa...@gmail.com wrote:
sys.version
2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)]
s = 'Straße'
assert len(s) == 6
assert s[5] == 'e'
jmf
On my utf8 based system
robin@everest ~:
$ cat ooo.py
if __name__=='__main__':
import sys
s='A̅B'
print('version_info=%s\nlen(%s)=%d' % (sys.version_info,s,len(s)))
robin@everest ~:
$ python ooo.py
version_info=sys.version_info(major=3, minor=3, micro=3, releaselevel='final',
serial=0)
len(A̅B)=3
robin@everest ~:
$
so two 'characters' are 3 (or 2 or more) codepoints. If I want to isolate so
called graphemes I need an algorithm even for python's unicode ie when it really
matters, python3 str is just another encoding.
--
Robin Becker
--
https://mail.python.org/mailman/listinfo/python-list
Mistake or Troll (was Re: 'Straße' ('Strasse') and Python 2)
On 1/13/2014 4:54 AM, wxjmfa...@gmail.com wrote: I'm afraid I'm understanding Python (on this aspect very well). Really? Do you belong to this group of people who are naively writing wrong Python code (usually not properly working) during more than a decade? To me, the important question is whether this and previous similar posts are intentional trolls designed to stir up the flurry of responses they get or 'innocently' misleading or even erroneous. If your claim of understanding Python and Unicode is true, then this must be a troll post. Either way, please desist, or your access to python-list from google-groups may be removed. 'ß' is the the fourth character in that text "Straße" (base index 0). As others have said, in the *unicode text "Straße", 'ß' is the fifth character, at character index 4, ... This assertions are correct (byte string and unicode). whereas, when the text is encoded into bytes, the byte index depends on the encoding and the assertion that it is always 4 is incorrect. Did you know this or were you truly ignorant? sys.version '2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)]' assert 'Straße'[4] == 'ß' Sometimes true, sometimes not. assert u'Straße'[4] == u'ß' PS Nothing to do with Py2/Py3. This issue has everything to do with Py2, where 'Straße' is encoded bytes, versus Py3, where 'Straße' is unicode text where each character of that word takes one code unit, whether each is 2 bytes or 4 bytes. If you replace 'ß' with any astral (non-BMP) character, this issue appears even for unicode text in 3.2-, where an astral character requires 2, not 1, code units on narrow builds, thereby screwing up indexing, just as can happen for encoded bytes. In 3.3+, all characters use 1 code unit and indexing (and slicing) always works properly. This is another unicode issue where you appear not to understand, but might just be trolling. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: 'Straße' ('Strasse') and Python 2
Am 13.01.2014 10:54 schrieb wxjmfa...@gmail.com: Not at all. I'm afraid I'm understanding Python (on this aspect very well). IBTD. Do you belong to this group of people who are naively writing wrong Python code (usually not properly working) during more than a decade? Why should I be? 'ß' is the the fourth character in that text "Straße" (base index 0). Character-wise, yes. But not byte-string-wise. In a byte string, this depends on the character set used. On CP 437, 850, 12xx (whatever Windows uses) or latin1, you are right, but not on the widely used UTF8. sys.version '2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)]' assert 'Straße'[4] == 'ß' assert u'Straße'[4] == u'ß' Linux box at home: Python 2.7.3 (default, Apr 14 2012, 08:58:41) [GCC] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> assert 'Straße'[4] == 'ß' Traceback (most recent call last): File "", line 1, in AssertionError >>> assert u'Straße'[4] == u'ß' Python 3.3.0 (default, Oct 01 2012, 09:13:30) [GCC] on linux Type "help", "copyright", "credits" or "license" for more information. >>> assert 'Straße'[4] == 'ß' >>> assert u'Straße'[4] == u'ß' Windows box at work: Python 2.7.5 (default, May 15 2013, 22:44:16) [MSC v.1500 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> assert 'Straße'[4] == 'ß' >>> assert u'Straße'[4] == u'ß' PS Nothing to do with Py2/Py3. As bytes and unicode and str stuff is heavily changed between them, of course it has to do. And I think you know that and try to confuse and FUD us all - with no avail. Thomas -- https://mail.python.org/mailman/listinfo/python-list
Re: 'Straße' ('Strasse') and Python 2
On 13/01/2014 16:24, wxjmfa...@gmail.com wrote: You are right. It's on Windows. It is only showing how Python can be a holy mess. Regarding unicode Python 2 was a holy mess, fixed in Python 3. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: 'Straße' ('Strasse') and Python 2
Le lundi 13 janvier 2014 11:57:28 UTC+1, Chris Angelico a écrit :
> On Mon, Jan 13, 2014 at 9:38 PM, Steven D'Aprano
>
> wrote:
>
> > I think you are using "from __future__ import unicode_literals".
>
> > Otherwise, that cannot happen in Python 2.x.
>
> >
>
>
>
> Alas, not true.
>
>
>
> >>> sys.version
>
> '2.7.4 (default, Apr 6 2013, 19:54:46) [MSC v.1500 32 bit (Intel)]'
>
> >>> sys.maxunicode
>
> 65535
>
> >>> assert 'Straße'[4] == 'ß'
>
> >>> list('Straße')
>
> ['S', 't', 'r', 'a', '\xdf', 'e']
>
>
>
> That's Windows XP. Presumably Latin-1 (or CP-1252, they both have that
>
> char at 0xDF). He happens to be correct, *as long as the source code
>
> encoding matches the output encoding and is one that uses 0xDF to mean
>
> U+00DF*. Otherwise, he's not.
>
>
You are right. It's on Windows. It is only showing how
Python can be a holy mess.
The funny aspect is when I'm reading " *YOUR* assertions
are false" when I'm presenting *PYTHON* assertions!
jmf
--
https://mail.python.org/mailman/listinfo/python-list
Re: 'Straße' ('Strasse') and Python 2
On 01/13/2014 02:54 AM, wxjmfa...@gmail.com wrote: > Not at all. I'm afraid I'm understanding Python (on this > aspect very well). Are you sure about that? Seems to me you're still confused as to the difference between unicode and encodings. > > Do you belong to this group of people who are naively > writing wrong Python code (usually not properly working) > during more than a decade? > > 'ß' is the the fourth character in that text "Straße" > (base index 0). > > This assertions are correct (byte string and unicode). How can they be? They only are true for the default encoding and character set you are using, which happens to have 'ß' as a single byte. Hence your little python 2.7 snippet is not using unicode at all, in any form. It's using a non-unicode character set. There are methods which can decode your character set to unicode and encode from unicode. But let's be clear. Your byte streams are not unicode! If the default byte encoding is UTF-8, which is a variable number of bytes per character, your assertions are completely wrong. Maybe it's time you stopped programming in Windows and use OS X or Linux which throw out the random single-byte character sets and instead provide a UTF-8 terminal environment to support non-latin characters. > sys.version > '2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)]' assert 'Straße'[4] == 'ß' assert u'Straße'[4] == u'ß' > > jmf > > PS Nothing to do with Py2/Py3. -- https://mail.python.org/mailman/listinfo/python-list
Re: 'Straße' ('Strasse') and Python 2
On Mon, Jan 13, 2014 at 9:38 PM, Steven D'Aprano
wrote:
> I think you are using "from __future__ import unicode_literals".
> Otherwise, that cannot happen in Python 2.x.
>
Alas, not true.
>>> sys.version
'2.7.4 (default, Apr 6 2013, 19:54:46) [MSC v.1500 32 bit (Intel)]'
>>> sys.maxunicode
65535
>>> assert 'Straße'[4] == 'ß'
>>> list('Straße')
['S', 't', 'r', 'a', '\xdf', 'e']
That's Windows XP. Presumably Latin-1 (or CP-1252, they both have that
char at 0xDF). He happens to be correct, *as long as the source code
encoding matches the output encoding and is one that uses 0xDF to mean
U+00DF*. Otherwise, he's not.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: 'Straße' ('Strasse') and Python 2
On Mon, 13 Jan 2014 01:54:21 -0800, wxjmfauth wrote:
sys.version
> '2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)]'
assert 'Straße'[4] == 'ß'
assert u'Straße'[4] == u'ß'
I think you are using "from __future__ import unicode_literals".
Otherwise, that cannot happen in Python 2.x. Using a narrow build:
# on my machine "ando"
py> sys.version
'2.7.2 (default, May 18 2012, 18:25:10) \n[GCC 4.1.2 20080704 (Red Hat
4.1.2-52)]'
py> sys.maxunicode
65535
py> assert 'Straße'[4] == 'ß'
Traceback (most recent call last):
File "", line 1, in
AssertionError
py> list('Straße')
['S', 't', 'r', 'a', '\xc3', '\x9f', 'e']
Using a wide build is the same:
# on my machine "orac"
>>> sys.maxunicode
1114111
>>> assert 'Straße'[4] == 'ß'
Traceback (most recent call last):
File "", line 1, in
AssertionError
But once you run the "from __future__" line, the behaviour changes to
what you show:
py> from __future__ import unicode_literals
py> list('Straße')
[u'S', u't', u'r', u'a', u'\xdf', u'e']
py> assert 'Straße'[4] == 'ß'
py>
But I still don't understand the point you are trying to make.
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
Re: 'Straße' ('Strasse') and Python 2
On Mon, Jan 13, 2014 at 8:54 PM, wrote: > This assertions are correct (byte string and unicode). > sys.version > '2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)]' assert 'Straße'[4] == 'ß' assert u'Straße'[4] == u'ß' > > jmf > > PS Nothing to do with Py2/Py3. This means that either your source encoding happens to include that character, or you have assertions disabled. It does NOT mean that you can rely on writing this string out to a file and having someone else read it in and understand it the same way. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: 'Straße' ('Strasse') and Python 2
Le lundi 13 janvier 2014 09:27:46 UTC+1, Thomas Rachel a écrit : > Am 12.01.2014 08:50 schrieb wxjmfa...@gmail.com: > > sys.version > > > 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)] > > s = 'Stra�e' > > assert len(s) == 6 > > assert s[5] == 'e' > > > > > > Wow. You just found one of the major differences between Python 2 and 3. > > > > Your assertins are just wrong, as s = 'Stra�e' leads - provided you use > > UTF8 - to a representation of 'Stra\xc3\x9fe', obviously leading to a > > length of 7. > > Not at all. I'm afraid I'm understanding Python (on this aspect very well). Do you belong to this group of people who are naively writing wrong Python code (usually not properly working) during more than a decade? 'ß' is the the fourth character in that text "Straße" (base index 0). This assertions are correct (byte string and unicode). >>> sys.version '2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)]' >>> assert 'Straße'[4] == 'ß' >>> assert u'Straße'[4] == u'ß' >>> jmf PS Nothing to do with Py2/Py3. -- https://mail.python.org/mailman/listinfo/python-list
Re: 'Straße' ('Strasse') and Python 2
Am 12.01.2014 08:50 schrieb wxjmfa...@gmail.com: sys.version 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)] s = 'Straße' assert len(s) == 6 assert s[5] == 'e' Wow. You just found one of the major differences between Python 2 and 3. Your assertins are just wrong, as s = 'Straße' leads - provided you use UTF8 - to a representation of 'Stra\xc3\x9fe', obviously leading to a length of 7. Thomas -- https://mail.python.org/mailman/listinfo/python-list
Re: 'Straße' ('Strasse') and Python 2
On 2014-01-12 08:31, Peter Otten wrote: wxjmfa...@gmail.com wrote: sys.version 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)] s = 'Straße' assert len(s) == 6 assert s[5] == 'e' jmf Signifying nothing. (Macbeth) Python 2.7.2+ (default, Jul 20 2012, 22:15:08) [GCC 4.6.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. s = "Straße" assert len(s) == 6 Traceback (most recent call last): File "", line 1, in AssertionError assert s[5] == "e" Traceback (most recent call last): File "", line 1, in AssertionError The point is that in Python 2 'Straße' is a bytestring and its length depends on the encoding of the source file. If the source file is UTF-8 then 'Straße' is a string literal with 7 bytes between the single quotes. -- https://mail.python.org/mailman/listinfo/python-list
Re: 'Straße' ('Strasse') and Python 2
On 12/01/2014 09:00, Stefan Behnel wrote: Peter Otten, 12.01.2014 09:31: wxjmfa...@gmail.com wrote: sys.version 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)] s = 'Straße' assert len(s) == 6 assert s[5] == 'e' jmf Signifying nothing. (Macbeth) Python 2.7.2+ (default, Jul 20 2012, 22:15:08) [GCC 4.6.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. s = "Straße" assert len(s) == 6 Traceback (most recent call last): File "", line 1, in AssertionError assert s[5] == "e" Traceback (most recent call last): File "", line 1, in AssertionError The point I think he was trying to make is that Linux is better than Windows, because the latter fails to fail on these assertions for some reason. Stefan :o) The point he's trying to make is that he also reads the pythondev mailing list, where Steven D'Aprano posted this very example, stating it is "Python 2 nonsense". Fixed in Python 3. Don't mention... :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: 'Straße' ('Strasse') and Python 2
On 1/12/14 2:50 AM, wxjmfa...@gmail.com wrote: sys.version 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)] s = 'Straße' assert len(s) == 6 assert s[5] == 'e' jmf Dumping random snippets of Python sessions here is useless. If you are trying to make a point, you have to put some English around it. You know what is in your head, but we do not. -- Ned Batchelder, http://nedbatchelder.com -- https://mail.python.org/mailman/listinfo/python-list
Re: 'Straße' ('Strasse') and Python 2
Peter Otten, 12.01.2014 09:31: > wxjmfa...@gmail.com wrote: > >> >>> sys.version >> 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)] >> >>> s = 'Straße' >> >>> assert len(s) == 6 >> >>> assert s[5] == 'e' >> >>> >> >> jmf > > Signifying nothing. (Macbeth) > > Python 2.7.2+ (default, Jul 20 2012, 22:15:08) > [GCC 4.6.1] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> s = "Straße" > >>> assert len(s) == 6 > Traceback (most recent call last): > File "", line 1, in > AssertionError > >>> assert s[5] == "e" > Traceback (most recent call last): > File "", line 1, in > AssertionError The point I think he was trying to make is that Linux is better than Windows, because the latter fails to fail on these assertions for some reason. Stefan :o) -- https://mail.python.org/mailman/listinfo/python-list
Re: 'Straße' ('Strasse') and Python 2
wxjmfa...@gmail.com wrote: sys.version > 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)] s = 'Straße' assert len(s) == 6 assert s[5] == 'e' > > jmf Signifying nothing. (Macbeth) Python 2.7.2+ (default, Jul 20 2012, 22:15:08) [GCC 4.6.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> s = "Straße" >>> assert len(s) == 6 Traceback (most recent call last): File "", line 1, in AssertionError >>> assert s[5] == "e" Traceback (most recent call last): File "", line 1, in AssertionError -- https://mail.python.org/mailman/listinfo/python-list
