Pádraig Brady wrote:
> There have been some interesting "counting UTF-8 strings" threads
> over at reddit lately, all referenced from this article:
> http://www.daemonology.net/blog/2008-06-05-faster-utf8-strlen.html
But before these techniques can be used in practice in packages such as
coreutils, two problems would have to be solved satisfactorily:
1) "George Pollard makes the assumption that the input string is valid UTF-8".
This assumption cannot be upheld, as long as you use the same type
('char *') for UTF-8 encoded strings and normal C strings, or when
you occasionally convert between one and the other.
For example: Assume NAME is really a valid UTF-8 string.
A program then does
static char buf[20];
snprintf (buf, "%s", NAME);
utf8_strlen (buf);
Boing! You already have a buffer overrun: The snprintf can truncate
an UTF-8 character, and the utf8_strlen function then skips over the
terminating NUL byte and scans buf[21...infinity], and likely crashes.
2) We already have the problem that we want to keep good performance when
handling strings in the "C" locale or, more generally, in a unibyte locale.
So we get code duplication:
- code for unibyte locales,
- code for multibyte locales that uses mbrtowc().
If you want to optimize UTF-8 locales particularly, i.e. optimize away
the function calls inherent in mbrtowc(), then we get code triplication:
- code for unibyte locales,
- code for UTF-8 locales,
- code for multibyte locales other than UTF-8, that uses mbrtowc().
So, code size increases, and the testing requirements increase as well.
Bruno
_______________________________________________
Bug-coreutils mailing list
Bug-coreutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-coreutils
