On 01/17/2011 04:00 PM, Andrei Alexandrescu wrote:
On 1/17/11 6:44 AM, Steven Schveighoffer wrote:
We need to get some real numbers together. I'll see what I can create
for a type, but someone else needs to supply the input :) I'm on short
supply of unicode data, and any attempts I've made to create some result
in failure. I have one example of one composed character in this thread
that I can cling to, but in order to supply some real numbers, we need a
large amount of data.
Oh, one more thing. You don't need a lot of Unicode text containing
combining characters to write benchmarks. (You do need it for testing
purposes.) Most text won't contain combining characters anyway, so after
you implement graphemes, just benchmark them on regular text.
Correct. For this reason, we do not use the same source at all for
correctness and performance testing.
It is impossible to define typical or representative source (who
judges?) But at very minimum, source texts for perf measurement should
mix languages as diverse as possible, including some material of the
ones known to be problematic and/or atypical (english, korean, hebrew...)
The following (ripped and composed from ICU data sets) is just that:
https://bitbucket.org/denispir/denispir-d/src/c572ccaefa33/data/unicode.txt
Content:
12 natural languages
34767 bytes = utf8 code units
--> 20133 code points
--> 22033 normal codes (NFD decomposed)
--> 19205 piles = true characters
Denis
_________________
vita es estrany
spir.wikidot.com
