Indic editing (was: RE: The real solution)
As we all know, Unicode is a "logical" encoding, in the sense that it assigns codes to "abstract characters", rather than to the actual signs ("glyphs") which are visible on a printed page. This design principle has been chosen because it makes all non-visual text processing much easier. Recently, Arjun Aggarwal this principle has been criticized for Devanagari, on the ground that the elements of an Unicode Devanagari string do not correspond to the graphic elements of Devanagari text. Several people have explained in detail how this is not an acceptable criticism, because Unicode code points are NOT supposed to be displayed with a direct one-to-one mapping to glyphs. I think that this criticism was addressed adequately, for what concerns the ENCODING part, and that it is now Mr. Aggarwal turn to make an effort to understand better what he is criticizing. However, I think that only considering the encoding point of view does not catch the real reasons behind the discontent are periodically expressed by Indian users and engineers. It has always been my impression that, for a native user of Indic scripts, it is much more natural to work with visual glyphs. Why shouldn't it be so? When you write "Arjun" with a pencil, you trace: , , , <-u>, , , , exactly in this order. Who cares if, by the lexicographic point of view, plus constitutes a unit? Who cares if, by the phonological point of view, is pronounced before ? Who cares if, by a logical point of view, is a plus a virtual "virama"? Yet, by the graphical point of view, that name is spelled using that sequence of *glyphs*. Similarly, what the users see on a computer screen are *glyphs*, not abstract characters. Consequently, they should be enabled to interact (enter, modify, delete) the *glyphs*. How can users be asked to enter, modify or delete objects (such as "virama", "ZW(N)J") which are not visible and tangible on the screen? Or how can they be asked to interact with an entity which is in a certain position, pretending that it was somewhere else (repha, short i matra)? And why should it be forbidden to edit visible and tangible objects (such as the "danda" at the right side of many letters) on the basis that "logically they do not exist"? See the difference between the name "Arjun" as coded (©) in terms of Unicode characters, and as rendered (®) in terms of glyphs (for a visual representation of this example, see the attached file ARJUN.GIF.): © a ra virama ja -u na ® a j- danda -u repha n- danda Unicode requires that © form is converted to ® form before being displayed. This process is called "rendering" and, for Devanagari, it could be summarized in four logical steps: 1: Convert character codes into "glyph codes"; 2: Join some glyphs (e.g.: turn ra + virama into repha); 3: Reorder some glyphs (e.g.: move repha to its visual position); 4: Split some glyphs (e.g.: turn full C's into half C's + danda) (Notice that this is a very schematic algorithm, and that actual implementations can vary considerably; especially point 1 and 4 may be dropped.) In the case of "Arjun", the four steps perform the following changes (see again ARJUN.GIF): 1: a ra virama ja -u na 2: a repha ja -u na 3: a ja -u repha na 4: a j- danda -u repha n- danda So far so good: I see "Arjun" on the screen. But what if now I want to change "Arjun" into, say, "Aljun"? By the "logical" point of view, I should simply delete the ra and enter a la in the same position. But, on my screen, there is no ra at all! Moreover, there is no consonant at all before the ja, because the group ra+virama is displayed as a combining repha AFTER the j+danda+u group. Looking at the screen, the natural thing to do is to move to the repha and delete it, then move between the a and the ja and insert a half la. In order to accomplish a WYSIWYG editing of this kind, Unicode text should be preventively converted to a TEMPORARY INTERMEDIATE FORM, less "logic" and more "visual". In the case of Devanagari, a glyphic representation quite similar to the old "font encodings" should be used. With such an intermediate code, the user should be enabled to select and delete the danda of a letter to form a half letter, to enter or delete a matra i or a repha by placing the cursor in their visual position, and so on. The algorithm to convert Unicode to this intermediate glyphic representation already exists, and it is the four steps that I described above, which are now part of rendering engines and smart fonts. The difference is that this algorithm should be run BEFORE going into the visualization phase. The big difference is that editing actions should be executed on this intermediate code and, therefore, there is the need of a "DErendering" algorithm, which converts a portion of visual text back to real Unicode. A very similar thing
Re: Indic editing (was: RE: The real solution)
Marco wrote: > > In the case of "Arjun", the four steps perform the following changes (see > again ARJUN.GIF): > > 1: a ra virama ja -u na > 2: a repha ja -u na > 3: a ja -u repha na > 4: a j- danda -u repha n- danda > > > So far so good: I see "Arjun" on the screen. > > But what if now I want to change "Arjun" into, say, "Aljun"? By the > "logical" point of view, I should simply delete the ra and enter a la in the > same position. > > But, on my screen, there is no ra at all! Moreover, there is no consonant at > all before the ja, because the group ra+virama is displayed as a combining > repha AFTER the j+danda+u group. > > Looking at the screen, the natural thing to do is to move to the repha and > delete it, then move between the a and the ja and insert a half la. Actually, I would disagree with this. Trying to select and edit a repha, or any other mark above or below another letter is a pain, both to implement and from the point of view of a user trying to work with selection. My answer to this is that the natural thing to do is to cursor down before the na to get an insertion point. Then: backspace backspace backspace backspace la virama ja u Or, in terms of backing store: a ra virama ja -u | na a ra virama ja | na a ra virama | na a ra | na a | na a la | na a la virama | na a la virama ja | na a la virama ja -u | na And I'm done. 8 keystrokes after the cursor down, but more efficient than trying to mess with selecting the repha. Consider how often people will correct spelling errors, for example, by backspacing and retyping, rather than trying to select to a specific spot to correct and then having to reselect back to the original spot to continue. It is simply more efficient to do it this way. And the above example could be even more efficient if the editing system implemented the backspace/erase function to clobber syllable parts (or grapheme clusters) instead of character at a time. But you also have to consider ergonomic issues there. It may introduce inefficiencies and mistakes if a backspace/erase deletes more characters than one keystroke's worth of entry. One principle of low-level editing (without IME input/select/commit operations) ought generally to be: key key key erase erase erase should leave you with no change to text. > > In order to accomplish a WYSIWYG editing of this kind, Unicode text should > be preventively converted to a TEMPORARY INTERMEDIATE FORM, less "logic" and > more "visual". I'm not suggesting that this isn't also a possible approach to implementing Devanagari editing -- just that the issue of what a user does to deal with editing existing text, under the current Unicode model, isn't that big a deal for repha and its ilk. On the other hand, the reordrant vowels might well lend themselves to editor extensions that work in a visual mode as well as a logical mode. --Ken
Re: Indic editing (was: RE: The real solution)
Monday, November 26, 2001 It seems to me that we have three separate domains to deal with: 1. What should be keyed as input of Indic scripts, mainly Devanagari? 2. How shall Indic scripts data be stored and exchanged? 3. How should Indic scripts be displayed on screens and in print? ISCII and Unicode are not concerned with the first. They are very concerned with the second. There may be general agreement on the third, but a variety of output devices are involved. Unless ISCII changed from a phonetic based approach to a graphic based one I doubt that Unicode and ISO 10646 would even consider doing so. Having attended a meeting in 1982 of those who drafted ISCII I doubt that this will happen. Might it be possible to key data in user oriented glyph/graphic fashion and then convert it to a phonetic encoding for storage, processing and sharing? And then, for rendering, convert it from phoneitc encoding to whatever the local display needed (OS, fonts, etc.) for human consumption? I do not know if agreement could be achieved on keyboard layouts for Indian scripts; though desirable to facilitate mobility of those with keying skills, such standardization may not be necessry--both qwerty and Dvorak keyboards can result in ASCII data. Regards, Jim Agenbroad (disclaimer and addresses at bottom) On Mon, 26 Nov 2001, Marco Cimarosti wrote: > As we all know, Unicode is a "logical" encoding, in the sense that it > assigns codes to "abstract characters", rather than to the actual signs > ("glyphs") which are visible on a printed page. This design principle has > been chosen because it makes all non-visual text processing much easier. > > Recently, Arjun Aggarwal this principle has been criticized for Devanagari, > on the ground that the elements of an Unicode Devanagari string do not > correspond to the graphic elements of Devanagari text. > > Several people have explained in detail how this is not an acceptable > criticism, because Unicode code points are NOT supposed to be displayed with > a direct one-to-one mapping to glyphs. > > I think that this criticism was addressed adequately, for what concerns the > ENCODING part, and that it is now Mr. Aggarwal turn to make an effort to > understand better what he is criticizing. > > > However, I think that only considering the encoding point of view does not > catch the real reasons behind the discontent are periodically expressed by > Indian users and engineers. > > It has always been my impression that, for a native user of Indic scripts, > it is much more natural to work with visual glyphs. > > Why shouldn't it be so? When you write "Arjun" with a pencil, you trace: > , , , <-u>, , , , exactly in this order. > > Who cares if, by the lexicographic point of view, plus > constitutes a unit? Who cares if, by the phonological point of view, > is pronounced before ? Who cares if, by a logical point of view, > is a plus a virtual "virama"? > > Yet, by the graphical point of view, that name is spelled using that > sequence of *glyphs*. > > Similarly, what the users see on a computer screen are *glyphs*, not > abstract characters. Consequently, they should be enabled to interact > (enter, modify, delete) the *glyphs*. > > How can users be asked to enter, modify or delete objects (such as "virama", > "ZW(N)J") which are not visible and tangible on the screen? Or how can they > be asked to interact with an entity which is in a certain position, > pretending that it was somewhere else (repha, short i matra)? And why > should it be forbidden to edit visible and tangible objects (such as the > "danda" at the right side of many letters) on the basis that "logically they > do not exist"? > > > See the difference between the name "Arjun" as coded (©) in terms of Unicode > characters, and as rendered (®) in terms of glyphs (for a visual > representation of this example, see the attached file ARJUN.GIF.): > > © a ra virama ja -u na > ® a j- danda -u repha n- danda > > Unicode requires that © form is converted to ® form before being displayed. > This process is called "rendering" and, for Devanagari, it could be > summarized in four logical steps: > > 1: Convert character codes into "glyph codes"; > 2: Join some glyphs (e.g.: turn ra + virama into repha); > 3: Reorder some glyphs (e.g.: move repha to its visual position); > 4: Split some glyphs (e.g.: turn full C's into half C's + danda) > > (Notice that this is a very schematic algorithm, and that actual > implementations can vary considerably; especially point 1 and 4 may be > dropped.) > > In the case of "Arjun", the four steps perform the following changes (see > again ARJUN.GIF): > > 1: a ra virama ja -u na > 2: a repha ja -u na > 3: a ja -u repha na > 4: a j- danda -u r
Re: Indic editing (was: RE: The real solution)
The debate about Indic does highlight important issues for the end user. (I am not a Hindi reader/writer.) Hindi text inputted under Notepad in XP exhibits the following behaviour. Backspacing deletes characters as they appear to be typed. Pressing delete before characters deletes blocks of characters. Charlie Jolly - Original Message - From: "Kenneth Whistler" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]> Sent: Monday, November 26, 2001 4:14 PM Subject: Re: Indic editing (was: RE: The real solution) > Marco wrote: > > > > > In the case of "Arjun", the four steps perform the following changes (see > > again ARJUN.GIF): > > > > 1: a ra virama ja -u na > > 2: a repha ja -u na > > 3: a ja -u repha na > > 4: a j- danda -u repha n- danda > > > > > > So far so good: I see "Arjun" on the screen. > > > > But what if now I want to change "Arjun" into, say, "Aljun"? By the > > "logical" point of view, I should simply delete the ra and enter a la in the > > same position. > > > > But, on my screen, there is no ra at all! Moreover, there is no consonant at > > all before the ja, because the group ra+virama is displayed as a combining > > repha AFTER the j+danda+u group. > > > > Looking at the screen, the natural thing to do is to move to the repha and > > delete it, then move between the a and the ja and insert a half la. > > Actually, I would disagree with this. Trying to select and edit a > repha, or any other mark above or below another letter is a pain, > both to implement and from the point of view of a user trying to > work with selection. > > My answer to this is that the natural thing to do is to cursor down > before the na to get an insertion point. Then: > >backspace backspace backspace backspace la virama ja u > > Or, in terms of backing store: > >a ra virama ja -u | na >a ra virama ja | na >a ra virama | na >a ra | na >a | na >a la | na >a la virama | na >a la virama ja | na >a la virama ja -u | na > > And I'm done. 8 keystrokes after the cursor down, but more efficient > than trying to mess with selecting the repha. > > Consider how often people will correct spelling errors, for example, > by backspacing and retyping, rather than trying to select to a > specific spot to correct and then having to reselect back to the > original spot to continue. It is simply more efficient to do it > this way. > > And the above example could be even more efficient if the editing > system implemented the backspace/erase function to clobber syllable > parts (or grapheme clusters) instead of character at a time. But > you also have to consider ergonomic issues there. It may introduce > inefficiencies and mistakes if a backspace/erase deletes more > characters than one keystroke's worth of entry. One principle of > low-level editing (without IME input/select/commit operations) > ought generally to be: key key key erase erase erase > should leave you with no change to text. > > > > > In order to accomplish a WYSIWYG editing of this kind, Unicode text should > > be preventively converted to a TEMPORARY INTERMEDIATE FORM, less "logic" and > > more "visual". > > I'm not suggesting that this isn't also a possible approach to implementing > Devanagari editing -- just that the issue of what a user does to > deal with editing existing text, under the current Unicode model, > isn't that big a deal for repha and its ilk. On the other hand, the > reordrant vowels might well lend themselves to editor extensions that > work in a visual mode as well as a logical mode. > > --Ken > > > >
What is meant by '\u0000'?
> As for cut & paste, it might work among Microsoft Apps > but if one wants to interface an app with a disclosed > clipboard format he will realize that he can not paste > unicode text that contains '\u' characters. Impossible. Does he mean specifically the character U+, or rather any character referenced by hex codepoint? Hex codepoints (or something similar) are sometimes needed to keep ASCII-only systems from trashing your data. I have used them so much I took my copy of the Unicode standard and on the hiragana page, wrote some decimal equivalents of hex numbers there. Hex codepoints are an excellent idea when making a display that is to be shown on browsers set to who knows what codepage. Tex, how did you do the name page, anyway? It would be useful to have a utility where you type text and out come the '\u' type strings (or else HTML hash codes) for use in a Java program or Web page. -- ___ Get your free email from http://www.ranmamail.com Powered by Outblaze