Re: Accented Characters and Counting Syllables
On Sunday, 7 December 2014 at 15:47:45 UTC, H. S. Teoh via Digitalmars-d-learn wrote: Ok, thanks. I just noticed that byGrapheme() lacks bidirectional access. Further it also lacks graphemeStrideBack() in complement to graphemeStride()? Similar to stride() and strideBack(). Is this difficult to implement? Not sure, but I wouldn't be surprised if it is. Unicode algorithms are generally non-trivial. T What's the best source of information for these algorithms? Is it certain that graphemes iteration is backwards iteratable by definition?
Re: Accented Characters and Counting Syllables
On Monday, 8 December 2014 at 14:57:06 UTC, Nordlöw wrote: What's the best source of information for these algorithms? Is it certain that graphemes iteration is backwards iteratable by definition? I guess https://en.wikipedia.org/wiki/Combining_character could be a good start.
Re: Accented Characters and Counting Syllables
On Saturday, 6 December 2014 at 22:37:19 UTC, Nordlöw wrote: Given the fact that static assert(é.length == 2); I was surprised that static assert(é.byCodeUnit.length == 2); static assert(é.byCodePoint.length == 2); string already iterates over code points. So byCodePoint doesn't have to do anything on it, and it just returns the same string again. string's .length is the number of code units. It's not compatible with the range primitives. That's why hasLength is false for string (and wstring). Don't use .length on ranges without checking hasLength. So, while é.byCodeUnit and é.byCodePoint have equal `.length`s, they have different range element counts.
Re: Accented Characters and Counting Syllables
On Saturday, 6 December 2014 at 22:37:19 UTC, Nordlöw wrote: static assert(é.byCodePoint.length == 2); Huh? Why is byCodePoint.length even defined?
Re: Accented Characters and Counting Syllables
On Sunday, 7 December 2014 at 13:24:28 UTC, Marc Schütz wrote: On Saturday, 6 December 2014 at 22:37:19 UTC, Nordlöw wrote: static assert(é.byCodePoint.length == 2); Huh? Why is byCodePoint.length even defined? because string has ElementType dchar (i.e. it already iterates by codepoint), which means that byCodePoint is just the identity function.
Re: Accented Characters and Counting Syllables
On Sunday, 7 December 2014 at 13:24:28 UTC, Marc Schütz wrote:
On Saturday, 6 December 2014 at 22:37:19 UTC, Nordlöw wrote:
static assert(é.byCodePoint.length == 2);
Huh? Why is byCodePoint.length even defined?
import std.uni;
pragma(msg, typeof(é.byCodePoint));
= string
Something's very broken...
It's this definition in std.uni:
Range byCodePoint(Range)(Range range)
if(isInputRange!Range is(Unqual!(ElementType!Range) ==
dchar))
{
return range;
}
`Unqual!(ElementType!string)` is indeed `dchar` because of
auto-decoding.
Filed as bug:
https://issues.dlang.org/show_bug.cgi?id=13829
Re: Accented Characters and Counting Syllables
On Saturday, 6 December 2014 at 23:11:49 UTC, H. S. Teoh via Digitalmars-d-learn wrote: This is a Unicode issue. What you want is neither byCodeUnit nor byCodePoint, but byGrapheme. A grapheme is the Unicode equivalent of what lay people would call a character. A Unicode character (or more precisely, a code point) is not necessarily a complete grapheme, as your example above shows; it's just a numerical value that uniquely identifies an entry in the Unicode character database. T Ok, thanks. I just noticed that byGrapheme() lacks bidirectional access. Further it also lacks graphemeStrideBack() in complement to graphemeStride()? Similar to stride() and strideBack(). Is this difficult to implement?
Re: Accented Characters and Counting Syllables
On Sun, Dec 07, 2014 at 02:30:13PM +, Nordlöw via Digitalmars-d-learn wrote: On Saturday, 6 December 2014 at 23:11:49 UTC, H. S. Teoh via Digitalmars-d-learn wrote: This is a Unicode issue. What you want is neither byCodeUnit nor byCodePoint, but byGrapheme. A grapheme is the Unicode equivalent of what lay people would call a character. A Unicode character (or more precisely, a code point) is not necessarily a complete grapheme, as your example above shows; it's just a numerical value that uniquely identifies an entry in the Unicode character database. T Ok, thanks. I just noticed that byGrapheme() lacks bidirectional access. Further it also lacks graphemeStrideBack() in complement to graphemeStride()? Similar to stride() and strideBack(). Is this difficult to implement? Not sure, but I wouldn't be surprised if it is. Unicode algorithms are generally non-trivial. T -- Who told you to swim in Crocodile Lake without life insurance??
Re: Accented Characters and Counting Syllables
On Sat, Dec 06, 2014 at 10:37:17PM +, Nordlöw via Digitalmars-d-learn wrote: Given the fact that static assert(é.length == 2); I was surprised that static assert(é.byCodeUnit.length == 2); static assert(é.byCodePoint.length == 2); Isn't there a way to iterate over accented characters (in my case UTF-8) in D? Or is this an inherent problem in Unicode? I need this in a syllable counting algorithm that needs to distinguish accented and non-accented variants of vowels. For example café (2 syllables) compared to babe (one syllable. This is a Unicode issue. What you want is neither byCodeUnit nor byCodePoint, but byGrapheme. A grapheme is the Unicode equivalent of what lay people would call a character. A Unicode character (or more precisely, a code point) is not necessarily a complete grapheme, as your example above shows; it's just a numerical value that uniquely identifies an entry in the Unicode character database. T -- There are 10 kinds of people in the world: those who can count in binary, and those who can't.
