Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: coreutils (Ubuntu)
Status: New => Confirmed
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to coreutils in Ubuntu.
https://bugs.launchpad.net/bugs/875713
Title:
cut fails to handle correctly utf-8
Status in coreutils package in Ubuntu:
Confirmed
Bug description:
1) I'm using Lucid :
$ lsb_release -rd
Description: Ubuntu 10.04.3 LTS
Release: 10.04
2) The version of coreutils (that contains the 'cut' utility)
$ apt-cache policy coreutils
coreutils:
Installé : 7.4-2ubuntu3
Candidat : 7.4-2ubuntu3
Table de version :
*** 7.4-2ubuntu3 0
500 http://fr.archive.ubuntu.com/ubuntu/ lucid-updates/main Packages
100 /var/lib/dpkg/status
7.4-2ubuntu2 0
500 http://fr.archive.ubuntu.com/ubuntu/ lucid/main Packages
3) What I expect to happen, as the man says:
$man cut
(...)
-b, --bytes=LIST
select only these bytes
-c, --characters=LIST
select only these characters
(...)
I expect a different behavior, when in 'not-1-byte' character sets
such as UTF-8, UTF-16, etc...
The same of different behavior we have with wc that says:
$man wc
(...)
-c, --bytes
print the byte counts
-m, --chars
print the character counts
(...)
So when I have on my environment :
$ env | grep 'LANG'
LANG=fr_FR.UTF-8
GDM_LANG=fr_FR.UTF-8
I can do:
$ printf '%s' 'déjà vu' | wc -c
9
$ printf '%s' 'déjà vu' | wc -m
7
That is CORRECT (with wc) because as the ENV variable says I'm using
UTF-8, the 'é' and 'à' count for 1 character but 2 bytes.
4) What happens instead: I get something wrong with 'cut'
$ printf '%s' 'déjà vu' | cut -b 1-4 | hd
00000000 64 c3 a9 6a 0a |d..j.|
$ printf '%s' 'déjà vu' | cut -c 1-4 | hd
00000000 64 c3 a9 6a 0a |d..j.|
(I piped it to 'hd', so that we have a better view of what's
happening)
It can even be worse and give an invalid UTF-8 output
$ printf '%s' 'déjà vu' | cut -c 1-5 | hd
00000000 64 c3 a9 6a c3 0a |d..j..|
That is because it 'cuts' in the middle of an UTF-8 sequence, and with the
appended \n, it gives an incoherent UTF-8 sequence as we can prove:
$ printf '%s' 'déjà vu' | cut -c 1-5 | iconv -f utf-8 -t iso8859-1
d�jiconv: séquence d'échappement non permise à la position 4
We can clearly see that -b or -c makes NO difference at all... but it should,
as 'wc' does, because we are in UTF-8
So either:
-Option a) the cut program is buggy, mixes the concept of 'byte' of
'character', and does things wrong when not in '1-byte' charset.
-Option b) the help/man is wrong, and there is no difference when handling
bytes and chars (but then why do we have two different options!)
Note that some other GNU utilities seam to mix the concept of 'byte' and
'char', but at least the misconception is clear in the man/help, for example
with 'head'
$man head
(...)
-c, --bytes=[-]N
print the first N bytes of each file; with the leading `-', print all
but the last N bytes of each file
The -b option is unused for 'head', and yet they chosed to use -c,
short of --bytes... Looks like they meant -c as --chars, but at the
last moment decided to handle only bytes!
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/coreutils/+bug/875713/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
