As we all know, Unicode is a "logical" encoding, in the sense that it
assigns codes to "abstract characters", rather than to the actual signs
("glyphs") which are visible on a printed page. This design principle has
been chosen because it makes all non-visual text processing much easier.
Recently, Arjun Aggarwal this principle has been criticized for Devanagari,
on the ground that the elements of an Unicode Devanagari string do not
correspond to the graphic elements of Devanagari text.
Several people have explained in detail how this is not an acceptable
criticism, because Unicode code points are NOT supposed to be displayed with
a direct one-to-one mapping to glyphs.
I think that this criticism was addressed adequately, for what concerns the
ENCODING part, and that it is now Mr. Aggarwal turn to make an effort to
understand better what he is criticizing.
However, I think that only considering the encoding point of view does not
catch the real reasons behind the discontent are periodically expressed by
Indian users and engineers.
It has always been my impression that, for a native user of Indic scripts,
it is much more natural to work with visual glyphs.
Why shouldn't it be so? When you write "Arjun" with a pencil, you trace:
<a>, <j->, <danda>, <-u>, <repha>, <n->, <danda>, exactly in this order.
Who cares if, by the lexicographic point of view, <j-> plus <danda>
constitutes a unit? Who cares if, by the phonological point of view, <repha>
is pronounced before <j->? Who cares if, by a logical point of view, <repha>
is a <ra> plus a virtual "virama"?
Yet, by the graphical point of view, that name is spelled using that
sequence of *glyphs*.
Similarly, what the users see on a computer screen are *glyphs*, not
abstract characters. Consequently, they should be enabled to interact
(enter, modify, delete) the *glyphs*.
How can users be asked to enter, modify or delete objects (such as "virama",
"ZW(N)J") which are not visible and tangible on the screen? Or how can they
be asked to interact with an entity which is in a certain position,
pretending that it was somewhere else (repha, short i matra)? And why
should it be forbidden to edit visible and tangible objects (such as the
"danda" at the right side of many letters) on the basis that "logically they
do not exist"?
See the difference between the name "Arjun" as coded (©) in terms of Unicode
characters, and as rendered (®) in terms of glyphs (for a visual
representation of this example, see the attached file ARJUN.GIF.):
© a ra virama ja -u na
® a j- danda -u repha n- danda
Unicode requires that © form is converted to ® form before being displayed.
This process is called "rendering" and, for Devanagari, it could be
summarized in four logical steps:
1: Convert character codes into "glyph codes";
2: Join some glyphs (e.g.: turn ra + virama into repha);
3: Reorder some glyphs (e.g.: move repha to its visual position);
4: Split some glyphs (e.g.: turn full C's into half C's + danda)
(Notice that this is a very schematic algorithm, and that actual
implementations can vary considerably; especially point 1 and 4 may be
dropped.)
In the case of "Arjun", the four steps perform the following changes (see
again ARJUN.GIF):
1: a ra virama ja -u na
2: a repha ja -u na
3: a ja -u repha na
4: a j- danda -u repha n- danda
So far so good: I see "Arjun" on the screen.
But what if now I want to change "Arjun" into, say, "Aljun"? By the
"logical" point of view, I should simply delete the ra and enter a la in the
same position.
But, on my screen, there is no ra at all! Moreover, there is no consonant at
all before the ja, because the group ra+virama is displayed as a combining
repha AFTER the j+danda+u group.
Looking at the screen, the natural thing to do is to move to the repha and
delete it, then move between the a and the ja and insert a half la.
In order to accomplish a WYSIWYG editing of this kind, Unicode text should
be preventively converted to a TEMPORARY INTERMEDIATE FORM, less "logic" and
more "visual".
In the case of Devanagari, a glyphic representation quite similar to the old
"font encodings" should be used. With such an intermediate code, the user
should be enabled to select and delete the danda of a letter to form a half
letter, to enter or delete a matra i or a repha by placing the cursor in
their visual position, and so on.
The algorithm to convert Unicode to this intermediate glyphic representation
already exists, and it is the four steps that I described above, which are
now part of rendering engines and smart fonts.
The difference is that this algorithm should be run BEFORE going into the
visualization phase.
The big difference is that editing actions should be executed on this
intermediate code and, therefore, there is the need of a "DErendering"
algorithm, which converts a portion of visual text back to real Unicode.
A very similar thing has been discussed months ago on this list about
bidirectional editing. I find that the process of reversing Indic rendering
is even easier than a "reverse bidi" algorithm.
It is possible to DErender Devanagari text by running the same rendering
algorithm listed before backwards and with reversed meanings:
4: Join some glyphs (e.g.: turn half C's + danda into full C's);
3: Reorder some glyphs (e.g.: more repha to its logical position);
2: Split some glyphs (e.g.: turn repha into ra + virama);
1: Convert glyph codes into character codes.
In the case of "Arjun", the four steps perform the following changes (see
again ARJUN.GIF, reading the four points from bottom to top):
4: a j- danda -u repha n- danda
3: a ja -u repha na
2: a repha ja -u na
1: a ra virama ja -u na
Notice that such an intermediate code can however be slightly MORE abstract
than a mere list of glyph variants: tiny and insignificant variations (such
as the different heights or sizes of combining glyphs, or the choice of
ligatures that are not strictly mandatory) may still be left to smart fonts
to handle.
Just my 0.02 euros.
_ Marco
arjun.gif
Description: GIF image
