Re: [HarfBuzz] Order of combining diacriticals
> From: Khaled Hosny > Date: Thu, 20 Jun 2019 22:09:24 +0200 > Cc: Behdad Esfahbod , Harfbuzz > > > I mean whether you are using HarfBuzz with FreeType font functions, > internal ones or something custom does not matter for fallback Hebrew > shaping. > > If you want to additionally use HarfBuzz with bitmap or Type 1 fonts > on Windows, you would need to implement custom font functions for > thase that would use GDI API to access glyph metrics and kerning, but > this is orthogonal to fallback shaping. Ah, okay. I understand now, thanks. ___ HarfBuzz mailing list HarfBuzz@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/harfbuzz
Re: [HarfBuzz] Order of combining diacriticals
On Thu, Jun 20, 2019 at 6:35 PM Eli Zaretskii wrote: > > > From: Khaled Hosny > > Date: Thu, 20 Jun 2019 17:33:47 +0200 > > Cc: Behdad Esfahbod , Harfbuzz > > > > > > > >. For fonts that have no 'hebr' features, Emacs performs > > > > substitution of known precomposed characters before it invokes the > > > > shaping engine. In this case, it substituted U+FB31 for the > > > > sequence U+05D1,U+05BC, and passed the sequence U+FB31,U+05B0 to > > > > HarfBuzz. > > > > > > > > You should remove all such hacks. > > > > > > I understand that for HarfBuzz they are probably not needed, if the > > > necessary functions for accessing the glyphs are provided (something > > > that might not be true on Windows, where we don't use Freetype > > > directly). > > > > This functionality either depends on Unicode decompositions (or in > > case of Hebrew hard-coded tables in HarfBuzz), so the font functions > > used make no difference. > > I'm not sure I understand what font functions you are talking about > here. > > The simplest font backends in Emacs: Xfont on Unix and GDI on > MS-Windows, when working with fonts that don't have the necessary OTF > features, might be unable, to figure out that certain combinations of > base character and combining mark have precomposed glyphs in the font > being used. So Emacs feeds them the precomposed characters instead. I mean whether you are using HarfBuzz with FreeType font functions, internal ones or something custom does not matter for fallback Hebrew shaping. If you want to additionally use HarfBuzz with bitmap or Type 1 fonts on Windows, you would need to implement custom font functions for thase that would use GDI API to access glyph metrics and kerning, but this is orthogonal to fallback shaping. Regards, Khaled ___ HarfBuzz mailing list HarfBuzz@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/harfbuzz
Re: [HarfBuzz] Order of combining diacriticals
> From: Khaled Hosny > Date: Thu, 20 Jun 2019 17:33:47 +0200 > Cc: Behdad Esfahbod , Harfbuzz > > > > >. For fonts that have no 'hebr' features, Emacs performs > > > substitution of known precomposed characters before it invokes the > > > shaping engine. In this case, it substituted U+FB31 for the > > > sequence U+05D1,U+05BC, and passed the sequence U+FB31,U+05B0 to > > > HarfBuzz. > > > > > > You should remove all such hacks. > > > > I understand that for HarfBuzz they are probably not needed, if the > > necessary functions for accessing the glyphs are provided (something > > that might not be true on Windows, where we don't use Freetype > > directly). > > This functionality either depends on Unicode decompositions (or in > case of Hebrew hard-coded tables in HarfBuzz), so the font functions > used make no difference. I'm not sure I understand what font functions you are talking about here. The simplest font backends in Emacs: Xfont on Unix and GDI on MS-Windows, when working with fonts that don't have the necessary OTF features, might be unable, to figure out that certain combinations of base character and combining mark have precomposed glyphs in the font being used. So Emacs feeds them the precomposed characters instead. How are font functions related to this? Thanks. ___ HarfBuzz mailing list HarfBuzz@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/harfbuzz
Re: [HarfBuzz] Order of combining diacriticals
On Fri, Jun 14, 2019 at 9:06 PM Eli Zaretskii wrote: > > > From: Behdad Esfahbod > > Date: Fri, 14 Jun 2019 11:34:17 -0700 > > Cc: Khaled Hosny , > > "harfbuzz@lists.freedesktop.org" > > > > On Thu, Jun 13, 2019 at 2:18 AM Eli Zaretskii wrote: > > > >. For fonts that have no 'hebr' features, Emacs performs > > substitution of known precomposed characters before it invokes the > > shaping engine. In this case, it substituted U+FB31 for the > > sequence U+05D1,U+05BC, and passed the sequence U+FB31,U+05B0 to > > HarfBuzz. > > > > You should remove all such hacks. > > I understand that for HarfBuzz they are probably not needed, if the > necessary functions for accessing the glyphs are provided (something > that might not be true on Windows, where we don't use Freetype > directly). This functionality either depends on Unicode decompositions (or in case of Hebrew hard-coded tables in HarfBuzz), so the font functions used make no difference. Regards, Khaled ___ HarfBuzz mailing list HarfBuzz@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/harfbuzz
Re: [HarfBuzz] Order of combining diacriticals
> From: Behdad Esfahbod > Date: Fri, 14 Jun 2019 11:34:17 -0700 > Cc: Khaled Hosny , > "harfbuzz@lists.freedesktop.org" > > On Thu, Jun 13, 2019 at 2:18 AM Eli Zaretskii wrote: > >. For fonts that have no 'hebr' features, Emacs performs > substitution of known precomposed characters before it invokes the > shaping engine. In this case, it substituted U+FB31 for the > sequence U+05D1,U+05BC, and passed the sequence U+FB31,U+05B0 to > HarfBuzz. > > You should remove all such hacks. I understand that for HarfBuzz they are probably not needed, if the necessary functions for accessing the glyphs are provided (something that might not be true on Windows, where we don't use Freetype directly). But Emacs also has other font backends, which are not as capable. In any case, this particular situation uncovered a subtle bug in how Emacs uses the information provided by HarfBuzz, so it was a Good Thing we did have this particular hack. Thanks. ___ HarfBuzz mailing list HarfBuzz@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/harfbuzz
Re: [HarfBuzz] Order of combining diacriticals
On Thu, Jun 13, 2019 at 2:18 AM Eli Zaretskii wrote: > . For fonts that have no 'hebr' features, Emacs performs > substitution of known precomposed characters before it invokes the > shaping engine. In this case, it substituted U+FB31 for the > sequence U+05D1,U+05BC, and passed the sequence U+FB31,U+05B0 to > HarfBuzz. > You should remove all such hacks. -- behdad http://behdad.org/ ___ HarfBuzz mailing list HarfBuzz@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/harfbuzz
Re: [HarfBuzz] Order of combining diacriticals
> Date: Wed, 12 Jun 2019 22:24:12 +0200 > From: Khaled Hosny > Cc: harfbuzz@lists.freedesktop.org > > On Wed, Jun 12, 2019 at 10:22:48PM +0300, Eli Zaretskii wrote: > > In Emacs, we use HB_BUFFER_CLUSTER_LEVEL_MONOTONE_GRAPHEMES cluster > > level, because HB_BUFFER_CLUSTER_LEVEL_MONOTONE_CHARACTERS produced > > incorrect display. > > The cluster levels shouldn’t affect display, the glyph positions are > exactly the same for all the three: Thanks, I guess I was misremembering something I've read in the HarfBuzz docs. > > U+05D1 HEBREW LETTER BET > > U+05B0 HEBREW POINT SHEVA > > U+05BC HEBREW POINT DAGESH > > > > > I need to type them in the above order; if I type DAGESH before SHEVA, > > the produced display is incorrect. > > The glyph order and positions are the same regardless of the input order > (which is what I’d expect since HarfBuzz normalizes mark order), the > only difference is cluster values which is also expected AFICT: > > $ hb-shape NotoSerifHebrew-Regular.ttf --unicodes="U+05D1,U+05B0,U+05BC" > --cluster-level=1 > [uni05B0=1@178,0+0|uni05BC=1@153,0+0|uni05D1=0+539] > > $ hb-shape NotoSerifHebrew-Regular.ttf --unicodes="U+05D1,U+05BC,U+05B0" > --cluster-level=1 > [uni05B0=2@178,0+0|uni05BC=1@153,0+0|uni05D1=0+539] > > > Is this expected with level-0 clusters? Or should I look for a bug in > > how Emacs uses HarfBuzz? > > Might be a result of hb_buffer_reverse_clusters() used by Emacs. Since we work on cluster level 0, there's only one cluster in this case, no matter what is the order of the characters in the original text. So cluster reversal cannot (and does not) have any effect here. The problem was a different one. The puzzle had two parts: . I used the Courier New font, which evidently doesn't have the 'hebr' OTF features in its GSUB and GPOS tables. If I use a font that does have those features, e.g., Symbola, the problem doesn't happen. . For fonts that have no 'hebr' features, Emacs performs substitution of known precomposed characters before it invokes the shaping engine. In this case, it substituted U+FB31 for the sequence U+05D1,U+05BC, and passed the sequence U+FB31,U+05B0 to HarfBuzz. It turned out there was a subtle bug in the code which uses the information returned by HarfBuzz, which is triggered by this use case: the TO value of the LGLYPH object was computed in a way that confused the Emacs display engine. Fixing the logic in that case resolved the problem. Thanks. ___ HarfBuzz mailing list HarfBuzz@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/harfbuzz
Re: [HarfBuzz] Order of combining diacriticals
On Wed, Jun 12, 2019 at 10:22:48PM +0300, Eli Zaretskii wrote: > In Emacs, we use HB_BUFFER_CLUSTER_LEVEL_MONOTONE_GRAPHEMES cluster > level, because HB_BUFFER_CLUSTER_LEVEL_MONOTONE_CHARACTERS produced > incorrect display. The cluster levels shouldn’t affect display, the glyph positions are exactly the same for all the three: $ hb-shape NotoSerifHebrew-Regular.ttf --unicodes="U+05D1,U+05B0,U+05BC" --cluster-level=0 [uni05B0=0@178,0+0|uni05BC=0@153,0+0|uni05D1=0+539] $ hb-shape NotoSerifHebrew-Regular.ttf --unicodes="U+05D1,U+05B0,U+05BC" --cluster-level=1 [uni05B0=1@178,0+0|uni05BC=1@153,0+0|uni05D1=0+539] $ hb-shape NotoSerifHebrew-Regular.ttf --unicodes="U+05D1,U+05B0,U+05BC" --cluster-level=2 [uni05B0=1@178,0+0|uni05BC=2@153,0+0|uni05D1=0+539] This might indicate an issue with the Emacs display code. > With this level, whenever I type a Hebrew base > character with more than one diacritical, I need to type them in > certain order, otherwise the display is incorrect. > > For example, in this series of characters: > > U+05D1 HEBREW LETTER BET > U+05B0 HEBREW POINT SHEVA > U+05BC HEBREW POINT DAGESH > > I need to type them in the above order; if I type DAGESH before SHEVA, > the produced display is incorrect. The glyph order and positions are the same regardless of the input order (which is what I’d expect since HarfBuzz normalizes mark order), the only difference is cluster values which is also expected AFICT: $ hb-shape NotoSerifHebrew-Regular.ttf --unicodes="U+05D1,U+05B0,U+05BC" --cluster-level=1 [uni05B0=1@178,0+0|uni05BC=1@153,0+0|uni05D1=0+539] $ hb-shape NotoSerifHebrew-Regular.ttf --unicodes="U+05D1,U+05BC,U+05B0" --cluster-level=1 [uni05B0=2@178,0+0|uni05BC=1@153,0+0|uni05D1=0+539] > Is this expected with level-0 clusters? Or should I look for a bug in > how Emacs uses HarfBuzz? Might be a result of hb_buffer_reverse_clusters() used by Emacs. Regards, Khaled ___ HarfBuzz mailing list HarfBuzz@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/harfbuzz
[HarfBuzz] Order of combining diacriticals
In Emacs, we use HB_BUFFER_CLUSTER_LEVEL_MONOTONE_GRAPHEMES cluster level, because HB_BUFFER_CLUSTER_LEVEL_MONOTONE_CHARACTERS produced incorrect display. With this level, whenever I type a Hebrew base character with more than one diacritical, I need to type them in certain order, otherwise the display is incorrect. For example, in this series of characters: U+05D1 HEBREW LETTER BET U+05B0 HEBREW POINT SHEVA U+05BC HEBREW POINT DAGESH I need to type them in the above order; if I type DAGESH before SHEVA, the produced display is incorrect. Is this expected with level-0 clusters? Or should I look for a bug in how Emacs uses HarfBuzz? Thanks. ___ HarfBuzz mailing list HarfBuzz@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/harfbuzz
