On 7/4/23 07:56, Hairy Pixels via fpc-pascal wrote:

On Jul 4, 2023, at 11:50 AM, Hairy Pixels <generic...@gmail.com> wrote:

You know you're right, with properly enclosed patterns you can capture 
everything inside and it works. You won't know if you had unicode in your 
string or not though but that depends on what's being parsed and if you care or 
not (I'm doing a TOML parser).
Sorry I'm still curious even though it's not my current problem :)

How can I make this program output the expected results:

   w: widechar;
   a: array of widechar;
begin
    for w in 'abcđŸ»' do
      a += [w];
   // Outputs 7 instead of 4
   writeln(length(a));
end;

The user doesn't know about unicode they just want to get an array of 
characters and not worry about all these little details. What can FPC do to 
solve this problem?

Depends on what you need, but I suppose in this case you want to count the number of extended grapheme clusters (a.k.a. "user perceived characters" - how many character-like things are displayed on the screen). You might be tempted to count the number of Unicode code points, but that's not the same, due to the existence of combining characters:

https://en.wikipedia.org/wiki/Combining_character

For extended grapheme clusters, there's an iterator in the graphemebreakproperty unit. I implemented this for the Unicode KVM and FreeVision. There it's needed for figuring out how many character blocks in the console will be needed to display a certain string. For the console or other GUIs that use fixed width fonts, there's also the East Asian Width property as well - some characters (East Asian - Chinese, Japanese, Korean) take double the space. So, to figure out where to move the cursor, you need to take East Asian Width as well.

Nikolay

_______________________________________________
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Reply via email to