Sorry for the slow response, I did not get notified about your comment (nor
do I know what I need to do for you to get notified about mine), Debian's
bugtracker has always been a legacy mystery to me.


The proof is not very direct, and surely isn't as nicely formatted as the
proofs you are used to.

"RTL and BiDi Introduction" -> "Why terminals are a truly special story" is
a key point. It elaborates what happens if a non-BiDi-aware app (let's say
Vim) wishes to display (and scroll horizontally) a 100-char wide text file
in an 80-char wide terminal. If the app knows nothing about BiDi and the
terminal shuffles the letters, combined with a heavily bi-directional line,
then the result is catastrophic. The editor sends out the wrong set of
letters, there's no way the terminal could rearrange them properly because
it's not just the arrangement, but the set of characters seen by the
terminal is wrong to begin with.

The editor needs to perform BiDi to be able to perform visual cropping. If
the terminal also does BiDi then it would be a "double BiDi" which is
obviously totally wrong. A way to remedy would be for the application to
perform a "reverse BiDi" to cancel that, resulting in a pretty bad
architecture; but more importantly, as I show in the previous section "RTL
and BiDi text handling in general", such a "reverse BiDi algorithm" doesn't
exist (if BiDi controls are not allowed; if they are allowed then a big
hammer is to place everything between LRO...PDF but then a separate mode
explicitly designed for this kind of operation is cleaner).

Create a text file that contains two 100-char lines. One is fully English,
the other is a complex mixture of the two directionalities. Open it in a
text editor of your choice that does _not_ perform BiDi. Scroll
horizontally in your choice of terminal emulator, one that does BiDi
according to your preference, e.g. mlterm. Notice that what you see is a
total craziness, not resembling at all how you'd slide a larger piece of
paper under a smaller viewing window. You can't fix it in the terminal. You
need to fix it in the app.


Re punctuation marks:

Note that whatever I say here below is utterly irrelevant because it's all
within the "implicit" mode (the terminal does BiDi), whereas, as per the
above, this mode of operation is not suitable for text editors, they need
to switch to "explicit" mode (the terminal does _not_ do BiDi, the app is
expected to).

Open a new LibreOffice text document. Type some RTL text followed by a
question mark, notice that the question mark appears on the "wrong" (i.e.
unexpected for you; on the right) side.

Open a html page as simple as this in your favorite browser (I trired
Firefox and Chromium):

<html><body><textarea></textarea></body></html>

and type the same thing, notice the same behavior again.

Open the UBA test page at https://util.unicode.org/UnicodeJsps/bidi.jsp ,
type the same thing. At the text area you'll see the same behavior again.

Choose "LTR Paragraph Direction", then click "Show Bidi". Notice that under
"Reordered" you get the question mark on the right; again the behavior
you're unhappy with.

VTE implement this exact behavior, specified by the UBA, as its default
behavior.

Now, you might choose "Auto Paragraph Direction" or "RTL Paragraph
direction" on the BiDi test page. You might apply a dir=auto or dir=rtl
attribute to the textarea node in the tiny html example. I'm sure
LibreOffice also has something equivalent. These modify the behavior to
your liking.

In VTE you can also do the same, we have escape sequences for that. First
you need to understand what role "Paragraph (a.k.a. base) direction" plays
in UBA, then pick your preferred behavior (in VTE: via escape sequences; in
mlterm: you're probably out of luck, it only offers one of the behaviors).

One thing we could perhaps argue about is: which one of the possible
behaviors (autodetection or not) should be the terminal's default. One is
better in certain situtations, the other is better in others, it's hard to
pick one. Autodetection (your preferred behavior) would mean that a file
listing (e.g. output of simple "ls") aligns English filenames to the left
edge of the terminal and Arabic/Hebrew filenames to the right edge of the
terminals; for lists I believe it's more desireable to align everything to
the same side. Especially for those who don't really care about BiDi, just
happen to have a single Arabic or Hebrew filename among all the English
ones, this behavior would be very confusing. I had no strong data but a
slight feeling that no autodetection is better suited as the default value
(whenever the terminal is used correctly; running Vim and letting the
terminal rearrange the letters does not count).

But, again, it's all irrelevant for Vim. The autodetected (or hardwired
RTL) paragraph direction within the terminal's implicit BiDi mode cannot
fix its behavior. Only the explicit BiDi mode can fix it. Only Vim knows
the entire string to know where that question mark needs to go, the
terminal potentially only sees a small substring thereof (even a totally
wrong substring if Vim didn't do BiDi), it doesn't have the knowledge to
correctly place it.


VTE implements exactly what the BiDi algorithm says. In fact VTE doesn't
reimplement it, it just calls the corresponding Fribidi methods. The BiDi
algorithm is more complicated than you think. It has multiple possible
behaviors depending on the paragraph direction which is to be passed to it
as an external parameter. One of the possible behaviors places the question
marks at the place you find undesireable, but again, this is what the UBA
tells to do, VTE did not make it up or did not falsely implement UBA. The
paragraph direction (including whether to autodetect it or not) can be
modified according to your liking via escape sequences, as per the
documentation, go and find the mode that suits your use cases the best. But
all this is within "implicit mode" suitable for simple utilities' outputs
where the terminal sees the entire string. For complex utilities where the
terminal potentially only sees a cropped string (e.g. text editors) this
theoretically cannot work at all, the only thing that can work is for the
app (e.g. Vim) to shuffle and the terminal to leave the order unchanged
("explicit mode").


e.

Reply via email to