Re: Numeric group separators and Bidi

2019-07-15 Thread Doug Ewell via Unicode
Philippe Verdy wrote: > [... U+202F ...] and not even accessible in most input tools... > including the Windows "Charmap" where it is not even listed with other > spaces or punctuations, except if we display the FULL list of > characters supported by a selected font that maps it (many fonts

Re: Numeric group separators and Bidi

2019-07-10 Thread Egmont Koblinger via Unicode
and find a workaround, i.e. a Unicode representation of numbers in RTL context with narrow spaces which is immune to those bugs? Not sure if any of us here are eager to help with that, I'm not, sorry. Not sure if it's possible at all (if there are really such bugs), probably not, given your furth

Fwd: Numeric group separators and Bidi

2019-07-09 Thread Philippe Verdy via Unicode
> Well my first feeling was that U+202F should work all the time, but I > found cases where this is not always the case. So this must be bugs in > those renderers. > I think we can attribute these bugs to the fact that this character is insufficiently known, and not even accessible in most input

Re: Numeric group separators and Bidi

2019-07-09 Thread Egmont Koblinger via Unicode
On Tue, Jul 9, 2019 at 10:43 PM Philippe Verdy wrote: > > Well my first feeling was that U+202F should work all the time, but I found > cases where this is not always the case. So this must be bugs in those > renderers. Could you share some concrete examples?

Re: Numeric group separators and Bidi

2019-07-09 Thread Philippe Verdy via Unicode
Well my first feeling was that U+202F should work all the time, but I found cases where this is not always the case. So this must be bugs in those renderers. And using Bidi controls (LRI/BDI) is absolutely not an option. These controls are only intended to be used in pure plain-text files

Re: Numeric group separators and Bidi

2019-07-09 Thread Egmont Koblinger via Unicode
k you need to pick a character whose BiDi class is "Common Number Separator", see e.g. https://www.compart.com/en/unicode/bidiclass/CS for a list of such characters including U+00A0 no-break space and U+202F narrow no-break space. This suggests to me that U+202F is a correct choice if you n

Re: Numeric group separators and Bidi

2019-07-09 Thread Eli Zaretskii via Unicode
> Date: Tue, 9 Jul 2019 20:59:15 +0200 > From: Philippe Verdy via Unicode > > I can't find a way to use narrow spaces instead of punctuation signs (dot or > comma) for example in > Arabic/Hebrew, for example to present tabular numeric data in a really > language-neutral way. In Arabic/Hebrew >

Numeric group separators and Bidi

2019-07-09 Thread Philippe Verdy via Unicode
Is there a narrow space usable as a numeric group separator, and that also has the same bidi property as digits (i.e. neutral outside the span of digits and separators, but inheriting the implied directionality of the previous digit) ? I can't find a way to use narrow spaces instead

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-18 Thread Egmont Koblinger via Unicode
On Sun, Feb 17, 2019 at 1:59 PM Philippe Verdy wrote: > Resist this idea, I've not been impolite. I didn't say a word about you being impolite. I said I might be impolite for not wishing to continue this discussion in that direction. > I just want to show you that terminals are legacy

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-17 Thread Philippe Verdy via Unicode
Le ven. 8 févr. 2019 à 13:56, Egmont Koblinger a écrit : > Philippe, I hate do say it, but at the risk of being impolite, I just > have to. > Resist this idea, I've not been impolite. I just want to show you that terminals are legacy environments that are far behind what is needed for proper

Re: Bidi paragraph direction in terminal emulators

2019-02-14 Thread Philippe Verdy via Unicode
Le mar. 12 févr. 2019 à 14:16, Egmont Koblinger via Unicode < unicode@unicode.org> a écrit : > > There is nothing magic about the grid of cells, and once you introduce > new escape sequences, you might as well truly modernise the terminal. > > The magic about the grid of cells is all the software

Re: Bidi paragraph direction in terminal emulators

2019-02-13 Thread Egmont Koblinger via Unicode
On Tue, Feb 12, 2019 at 9:35 PM Richard Wordingham via Unicode wrote: > Bash already seems to handle proportional fonts quite well when run > under Emacs 'M-x shell', Having never used bash inside Emacs's shell, here's my experience after about a minute of trying it: Cursor keys allow you to

Re: Bidi paragraph direction in terminal emulators

2019-02-12 Thread Richard Wordingham via Unicode
On Tue, 12 Feb 2019 13:50:00 +0100 Egmont Koblinger via Unicode wrote: > For > starter, I'd love to see a shell with interactive line editing (like > bash, zsh),... Bash already seems to handle proportional fonts quite well when run under Emacs 'M-x shell', which is more than can be said for

Re: Bidi paragraph direction in terminal emulators

2019-02-12 Thread Egmont Koblinger via Unicode
Hi Elias, > For all the willingness to come up with ways to modernise the terminal, > you've only spoken about trying to showhorn rtl text in to the vt102 basic > terminal. Yes, addressing BiDi was the exact thing that I did now. What's wrong with that? I can't address all the imperf

Re: Bidi paragraph direction in terminal emulators

2019-02-12 Thread Egmont Koblinger via Unicode
Hi Philippe, > The monospace restriction is a strong limitator: but then I don't see why a > "terminal" could not handle fonts with variable metrics, and why it must be > modeled only as a regular grid of rectangular cells (all of equal size) > containing only one "character" (or cluster?).

Re: Bidi paragraph direction in terminal emulators

2019-02-10 Thread Elias Mårtenson via Unicode
On Sun, 10 Feb 2019, 18:39 Egmont Koblinger via Unicode On Sun, Feb 10, 2019 at 2:57 AM Richard Wordingham via Unicode > wrote: > > > Which side do you align RTL cells on? > > It's out of the scope of my docs. > > In the current work-in-progress implementation I align them to the > left, but

Re: Bidi paragraph direction in terminal emulators

2019-02-10 Thread Richard Wordingham via Unicode
On Sun, 10 Feb 2019 14:54:39 +0100 Philippe Verdy via Unicode wrote: > Le sam. 9 févr. 2019 à 20:55, Egmont Koblinger via Unicode < > unicode@unicode.org> a écrit : > > > Hi Asmus, > > > > > On quick reading this appears to be a strong argument why such > > > emulators > > will > > >

Re: Bidi paragraph direction in terminal emulators

2019-02-10 Thread Philippe Verdy via Unicode
erminal protocosl will remain broken and it is more viable to work with the W3C to define a basic HTML profile suitable for terminals, but that will benefit of all the improvements made in HTRML to support i18n, including required ones (BiDi, variable-width fonts needed for complex scripts, acce

Re: Bidi paragraph direction in terminal emulators

2019-02-10 Thread Egmont Koblinger via Unicode
On Sun, Feb 10, 2019 at 2:57 AM Richard Wordingham via Unicode wrote: > Which side do you align RTL cells on? It's out of the scope of my docs. In the current work-in-progress implementation I align them to the left, but there's a TODO entry to align them to the right instead (or maybe center

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Richard Wordingham via Unicode
On Sun, 10 Feb 2019 00:59:46 +0100 Egmont Koblinger via Unicode wrote: > Is there such a monospace font obeying wcwidth (that is: double wide > character for when a spacing mark is combined) for Devanagari? For CV, that would correspond to a Hindi typewriter, so the odds look good. The

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Richard Wordingham via Unicode
On Sat, 9 Feb 2019 18:42:52 +0100 Egmont Koblinger via Unicode wrote: > The > problem that I don't know how to address is: What if harfbuzz tells us > that the overall width for rendering a particular grapheme cluster is > significantly different from its designated area (the number of >

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Richard Wordingham via Unicode
On Sat, 9 Feb 2019 22:29:31 +0100 Adam Borowski via Unicode wrote: > On Sat, Feb 09, 2019 at 10:01:21PM +0200, Eli Zaretskii via Unicode > wrote: > > I don't know. Maybe it keeps a database of character combinations > > that need shaping, each one with the maximum width on display the > >

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Egmont Koblinger via Unicode
Hi, On Sun, Feb 10, 2019 at 12:52 AM Richard Wordingham via Unicode wrote: > This is an example of where one needs a font designed for terminal > emulators. Definitely, this is another approach I forgot to mention in my mail, rather than VTE switching to harfbuzz and figuring out all the

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Richard Wordingham via Unicode
On Sat, 9 Feb 2019 22:31:37 +0100 Egmont Koblinger via Unicode wrote: > Let's take the Devanagari improvement of the other day. Until now, > there were plenty of dotted circles shown, and combining spacing marks > that should've been placed before the letter were placed after the > letter,

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Richard Wordingham via Unicode
On Sat, 9 Feb 2019 13:02:55 -0800 "Asmus Freytag \(c\) via Unicode" wrote: > To force Hindi crosswords mode you need to segment the string into > syllables, > each having a variable number of characters, and then assign a single > display > position to them. Now some syllables are wider than

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Asmus Freytag (c) via Unicode
useless effort, but it is a far cry from Unicode's universal support for ALL writing systems. A./ PS: also we have been seriously hijacking a thread related to bidi e.

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Egmont Koblinger via Unicode
On Sat, Feb 9, 2019 at 10:10 PM Asmus Freytag via Unicode wrote: > > I hope though that all the scripts can be supported with more or less > > compromises, e.g. like it would appear in a crossword. But maybe not. > > See other messages: not. For the crossword analogy, I can see why it's not

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Egmont Koblinger via Unicode
ith, > just a few more scripts than European + CJK? I don't have a clearly defined goal. I find fun in developing VTE (and slightly improving other terminal emulators too by spreading ideas, knowledge, comments etc.), addressing various kinds of goals, whatever happens to come next. At

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Adam Borowski via Unicode
On Sat, Feb 09, 2019 at 10:01:21PM +0200, Eli Zaretskii via Unicode wrote: > > From: Egmont Koblinger > > Date: Sat, 9 Feb 2019 20:36:50 +0100 > > Cc: Richard Wordingham , > > unicode Unicode Discussion > > > > On Sat, Feb 9, 2019 at 8:13 PM Eli Zaretskii wrote: > > > > > That's the

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Asmus Freytag (c) via Unicode
On 2/9/2019 11:48 AM, Egmont Koblinger wrote: Hi Asmus, On quick reading this appears to be a strong argument why such emulators will never be able to be used for certain scripts. Effectively, the model described works well with any scripts where characters are laid out (or can be laid out)

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Asmus Freytag via Unicode
On 2/9/2019 12:07 PM, Egmont Koblinger via Unicode wrote: On Sat, Feb 9, 2019 at 9:01 PM Eli Zaretskii wrote: then what you say is that some scripts can never be supported by text terminals. I'm not familiar at all with all the

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Ken Whistler via Unicode
x script issues. At any rate, this is once again straying over into the issue of whether terminals can  be adapted for the requirements of shaping rules for complex scripts -- rather than the nominal subject of the thread, which has to do with bidi text layout in terminals. --Ken

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Egmont Koblinger via Unicode
Hi Ken, > There are crossword puzzles for Hindi (in the Devanagari script). Just > do an image search for "Hindi crossword puzzle". It's easy to confirm the existence by an image search, it's hard to confirm the non-existence ;) > The existence proof of techniques to cut up text into syllables

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Egmont Koblinger via Unicode
On Sat, Feb 9, 2019 at 9:01 PM Eli Zaretskii wrote: > then what you say is that some scripts > can never be supported by text terminals. I'm not familiar at all with all the scripts and their requirements, but yes, basically this is what I'm saying. I'm afraid some scripts can never be

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Sat, 9 Feb 2019 20:36:50 +0100 > Cc: Richard Wordingham , > unicode Unicode Discussion > > On Sat, Feb 9, 2019 at 8:13 PM Eli Zaretskii wrote: > > > That's the application's problem, not the terminal's. An application > > that wants its column to line

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Egmont Koblinger via Unicode
Hi Asmus, > On quick reading this appears to be a strong argument why such emulators will > never be able to be used for certain scripts. Effectively, the model > described works > well with any scripts where characters are laid out (or can be laid out) in > fixed > width cells that are

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Egmont Koblinger via Unicode
On Sat, Feb 9, 2019 at 8:13 PM Eli Zaretskii wrote: > That's the application's problem, not the terminal's. An application > that wants its column to line up _and_ wants to support complex text > scripts will need to move cursor to certain coordinates, not to assume > that 7 codepoints always

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Sat, 9 Feb 2019 20:03:21 +0100 > Cc: Richard Wordingham , > unicode Unicode Discussion > > Let's suppose a utility outputs these two lines of text: > abcdefg| > complex| > > whereas "abcdefg" are these English letters themselves, but "complex" > is a

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Asmus Freytag via Unicode
On quick reading this appears to be a strong argument why such emulators will never be able to be used for certain scripts. Effectively, the model described works well with any scripts where characters are laid out (or can be laid out) in fixed width cells

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Egmont Koblinger via Unicode
On Sat, Feb 9, 2019 at 7:56 PM Eli Zaretskii wrote: > I'm probably missing something, because I don't see the grave problems > you hint at. Any width provided back by a shaper can be rounded to > the nearest integral character cell, so your canvas can still remain > rectangular. Let's suppose

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Sat, 9 Feb 2019 19:25:08 +0100 > Cc: Richard Wordingham , > unicode Unicode Discussion > > > You need to use what HarfBuzz tells you _instead_ of wcswidth. It is > > in general wrong to use wcswidth or anything similar when you use a > > shaping engine

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Egmont Koblinger via Unicode
On Sat, Feb 9, 2019 at 7:07 PM Eli Zaretskii wrote: > You need to use what HarfBuzz tells you _instead_ of wcswidth. It is > in general wrong to use wcswidth or anything similar when you use a > shaping engine and support complex script shaping. This approach is not viable at all. Terminal

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Eli Zaretskii via Unicode
> Date: Sat, 9 Feb 2019 18:42:52 +0100 > Cc: unicode Unicode Discussion > From: Egmont Koblinger via Unicode > > What if harfbuzz tells us that the overall width for rendering a > particular grapheme cluster is significantly different from its > designated area (the number of character cells

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Egmont Koblinger via Unicode
ngle step, that is, the spacing combining mark is applied around its base letter by Pango as expected. (Previously the spacing combining mark was rendered on its own, around a dotted circle, which was obviously pretty bad.) What I'm working on currently, as you all know by now, is BiDi-shuffling

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Richard Wordingham via Unicode
On Sat, 09 Feb 2019 09:42:09 +0200 Eli Zaretskii via Unicode wrote: > > Date: Sat, 9 Feb 2019 00:18:14 + > > From: Richard Wordingham via Unicode > > > > > For character composition, you must have a shaping engine to talk > > > to, and the shaper should tell you the width of each

Re: Bidi paragraph direction in terminal emulators

2019-02-09 Thread Eli Zaretskii via Unicode
> From: Elias Mårtenson > Date: Sat, 9 Feb 2019 13:33:49 +0800 > Cc: Egmont Koblinger , unicode > > Moreover, emitting the control sequences that set the mode is in > itself a complication, because if the terminal doesn't support them, > the result could be corrupted display. You will need

Re: Bidi paragraph direction in terminal emulators

2019-02-08 Thread Eli Zaretskii via Unicode
> Date: Sat, 9 Feb 2019 00:18:14 + > From: Richard Wordingham via Unicode > > > For character composition, you must have a shaping engine to talk to, > > and the shaper should tell you the width of each grapheme cluster it > > returns. > > (a) What defines the grapheme clusters? The

Re: Bidi paragraph direction in terminal emulators

2019-02-08 Thread Elias Mårtenson via Unicode
On Wed, 6 Feb 2019, 00:09 Eli Zaretskii via Unicode > Moreover, emitting the control sequences that set the mode is in > itself a complication, because if the terminal doesn't support them, > the result could be corrupted display. You will need methods of > detecting the support, and those

Re: Bidi paragraph direction in terminal emulators

2019-02-08 Thread Richard Wordingham via Unicode
On Sat, 09 Feb 2019 00:16:30 +0200 Eli Zaretskii via Unicode wrote: > > Date: Fri, 8 Feb 2019 21:55:58 + > > From: Richard Wordingham via Unicode > > I will give a concrete application. If I want to make a font that > > is interpretable for Tai Tham and maximally usable with VTE, what > >

Re: Bidi paragraph direction in terminal emulators

2019-02-08 Thread Eli Zaretskii via Unicode
> Date: Fri, 8 Feb 2019 21:55:58 + > From: Richard Wordingham via Unicode > > > > What's the sledgehammer for Windows? > > > Not sure what you meant. "M-x term" doesn't work on Windows. > > So my question is, 'What do I use on Windows?' The application may be > disproportionate to the

Re: Bidi paragraph direction in terminal emulators

2019-02-08 Thread Richard Wordingham via Unicode
e why. There are terminal emulators out there which support > proportional fonts. Emacs is perhaps the only one whose terminal > emulator currently supports bidi more or less in full, but is that > related to proportional fonts? Emacs is the one I know that can be made to support

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-08 Thread Egmont Koblinger via Unicode
On Fri, Feb 8, 2019 at 10:36 PM Eli Zaretskii wrote: > No one in their right minds will run Emacs inside the Emacs terminal > emulator. And even for other applications, disabling bidi will almost > always needed only for full-screen programs, which use curses-like > librarie

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-08 Thread Eli Zaretskii via Unicode
s > the explicit mode). No one in their right minds will run Emacs inside the Emacs terminal emulator. And even for other applications, disabling bidi will almost always needed only for full-screen programs, which use curses-like libraries to address the entire screen. So you'd switch off reorde

Columns in Terminal Emulators (was: Bidi paragraph direction in terminal emulators)

2019-02-08 Thread Richard Wordingham via Unicode
On Fri, 08 Feb 2019 15:45:15 +0200 Eli Zaretskii via Unicode wrote: > > From: Egmont Koblinger > > Date: Fri, 8 Feb 2019 13:30:42 +0100 > > Cc: Richard Wordingham , > > unicode Unicode Discussion > > > > Hi Eli, > > > > > Not sure why. There are terminal emulators out there which > >

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-08 Thread Egmont Koblinger via Unicode
oin their efforts wrt. new extensions, rather than ad-hoc collaborations or each going their own separate ways. We'd like its work to be widely accepted as a basis for the desired behavior. My BiDi work is one of the works hosted there. It'll probably never be an "authority" like ECMA,

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-08 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Fri, 8 Feb 2019 15:42:51 +0100 > Cc: Richard Wordingham , > unicode Unicode Discussion > > On Fri, Feb 8, 2019 at 3:28 PM Eli Zaretskii wrote: > > > You can have what you call the "explicit mode" if you set the

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-08 Thread Egmont Koblinger via Unicode
On Fri, Feb 8, 2019 at 3:28 PM Eli Zaretskii wrote: > You can have what you call the "explicit mode" if you set the variable > bidi-display-reordering to nil. So, if someone is running a mixture of applications requiring implicit vs. explicit modes, they'll have to con

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-08 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Fri, 8 Feb 2019 14:57:56 +0100 > Cc: Richard Wordingham , > unicode Unicode Discussion > > According to the description you give, Emacs's terminal always applies > the BiDi algorithm, therefore by its design only implements what I

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-08 Thread Egmont Koblinger via Unicode
that there's much more to BiDi in terminal emulators than running the UBA. If one takes a step backwards to look at the big picture, it becomes clear that in some cases the UBA needs to be run, while in other cases it mustn't. And then of course there needs to be some means of switching, a

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-08 Thread Eli Zaretskii via Unicode
6.x, I don't know. I told you what changed: Emacs 25 forces LTR paragraph direction, whereas Emacs 26 and later does not. You can get dynamic paragraph direction in your Emacs 25 as well if you set bidi-paragraph-direction to nil in the *term* buffer. > And now you suddenly tell that Emacs's

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-08 Thread Egmont Koblinger via Unicode
delays and without risks of > overwriting surrounding contents. At this point you're already touching much more the core of terminal emulator behavior than e.g. my BiDi work does, it's a way more essential, way more complex change – with much less clear goal to me, like, why should emulators implement

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-08 Thread Egmont Koblinger via Unicode
terminal > emulator currently supports bidi more or less in full Let's not get started from here, please. In Emacs-25.2's terminal emulator I executed "cat TUTORIAL.he". For the entire contents, LTR paragraph direction was used and was aligned to the left. Maybe something has changed

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-08 Thread Eli Zaretskii via Unicode
gt; > The message I take from that and this thread in general is that Emacs > and 'M-x term' are the route to take if one only has proportional fonts. Not sure why. There are terminal emulators out there which support proportional fonts. Emacs is perhaps the only one whose terminal emulator currentl

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-07 Thread Eli Zaretskii via Unicode
> Date: Thu, 7 Feb 2019 22:35:23 + > From: Richard Wordingham via Unicode > > > > Do you mean you aim to maintain a regex that matches everyone's > > > prompt in the world, without a significant amount of false positive > > > matches on non-prompt lines? > > > Yes. > > Wow! You'll do

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-07 Thread Richard Wordingham via Unicode
On Fri, 8 Feb 2019 00:38:24 +0100 Egmont Koblinger via Unicode wrote: > I, for one, am not to the slightest bit interested in abandoning the > character grid and allowing for proportional fonts. This would just > break a gazillion of things. The message I take from that and this thread in

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-07 Thread Philippe Verdy via Unicode
39, Egmont Koblinger a écrit : > Hi Philippe, > > > I have never said anything about your work because I don't know where > you spoke about it or where you made some proposals. I must have missed one > of your messages (did it reach this list?). > > This entire conversat

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-07 Thread Egmont Koblinger via Unicode
to bring usable BiDi to terminal emulators. > Terminals are not displaying plain text, they create their own upper layer > protocol which requires and enforces the 2D layout [...] Bidi does not > specify the 2D layout completely, it is purely 1D and speaks about left and > right

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-07 Thread Richard Wordingham via Unicode
On Thu, 07 Feb 2019 22:00:20 +0200 Eli Zaretskii via Unicode wrote: > > From: Egmont Koblinger > > Date: Thu, 7 Feb 2019 19:01:33 +0100 > > On Thu, Feb 7, 2019 at 6:53 PM Eli Zaretskii wrote: > > > No, it needs no interaction. Unless the regexp doesn't work for > > > you, which you should

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-07 Thread Philippe Verdy via Unicode
lay width, which cannot be monospaced in all scripts and are definitely not encoded in logical order: try adding characters at end of a logical line, with a Bidi text you do not just replace the content of one cell, you have to scroll the content of surrounding cells and your input curet position

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-07 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Thu, 7 Feb 2019 19:01:33 +0100 > Cc: Richard Wordingham , > unicode Unicode Discussion > > On Thu, Feb 7, 2019 at 6:53 PM Eli Zaretskii wrote: > > > No, it needs no interaction. Unless the regexp doesn't work for you, > > which you should then report

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-07 Thread Egmont Koblinger via Unicode
Hi Philippe, On Thu, Feb 7, 2019 at 3:21 PM Philippe Verdy wrote: > "Rules" are not formally written, they are just a sense of best practices. When it comes to BiDi in terminals, I haven't seen anything that I consider reasonably okay, let alone "best practice". It

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-07 Thread Egmont Koblinger via Unicode
On Thu, Feb 7, 2019 at 6:53 PM Eli Zaretskii wrote: > No, it needs no interaction. Unless the regexp doesn't work for you, > which you should then report as a bug in Emacs. Do you mean you aim to maintain a regex that matches everyone's prompt in the world, without a significant amount of

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-07 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Thu, 7 Feb 2019 18:20:02 +0100 > Cc: Richard Wordingham , > unicode Unicode Discussion > > > It uses a regular expression, see term-prompt-regexp. > > So, it's not automatic, needs user interaction No, it needs no interaction. Unless the regexp doesn't

Re: Bidi paragraph direction in terminal emulators

2019-02-07 Thread Egmont Koblinger via Unicode
On Thu, Feb 7, 2019 at 6:33 PM Eli Zaretskii wrote: > Well, let's just say that Emacs uses the HL1 rule, and determines the > base direction for the entire chunk of text between empty lines. Exactly! Now it's my turn to figure out how to add this behavior to terminals, preferably stopping

Re: Bidi paragraph direction in terminal emulators

2019-02-07 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Thu, 7 Feb 2019 18:12:37 +0100 > Cc: Richard Wordingham , > unicode Unicode Discussion > > I believe it's not my mental model that's weird, but your use of > terminology that doesn't match UBA's that confused me. Well, let's just say that Emacs uses the

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-07 Thread Egmont Koblinger via Unicode
Hi, On Thu, Feb 7, 2019 at 3:27 PM Eli Zaretskii wrote: > It uses a regular expression, see term-prompt-regexp. So, it's not automatic, needs user interaction, and for that reason, may not have worked for me. (I have other weird things in my prompt, like 256-color sequences that Emacs didn't

Re: Bidi paragraph direction in terminal emulators

2019-02-07 Thread Egmont Koblinger via Unicode
On Thu, Feb 7, 2019 at 3:14 PM Eli Zaretskii wrote: > Not a bug, a feature. Emacs doesn't remove the bidi controls from > display (that's another deviation allowed by the UBA, see section > 5.2). On GUI displays, these controls are displayed as thin 1-pixel > spaces, but on text-mo

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-07 Thread Eli Zaretskii via Unicode
at > TUTORIAL.he". All the paragraphs are rendered as LTR ones, > left-aligned. Not the way the file is opened in Emacs. In what version of Emacs is that? In the latest version 26 I have here, the tutorial displays with most paragraphs in RTL direction. > If you claim Emacs's

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-07 Thread Eli Zaretskii via Unicode
> Date: Wed, 6 Feb 2019 23:32:43 + > From: Richard Wordingham via Unicode > > > You define paragraphs as emptyline-separated blocks on which you > > perform autodetection of the paragraph direction. This is great! As > > I've mentioned, I'd love to have such a mode in terminals, but it's > >

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-07 Thread Philippe Verdy via Unicode
Le jeu. 7 févr. 2019 à 13:29, Egmont Koblinger a écrit : > Hi Philippe, > > > There's some rules for correct display including with Bidi: > > In what sense are these "rules"? Where are these written, in what kind > of specification or existing practice? >

Re: Bidi paragraph direction in terminal emulators

2019-02-07 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Wed, 6 Feb 2019 22:01:59 +0100 > Cc: Richard Wordingham , unicode@unicode.org > > - Emacs running in a terminal shows an underscore wherever there's a > BiDi control in the source file – while the graphical one doesn't. > This looks like a si

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-07 Thread Egmont Koblinger via Unicode
Hi Philippe, > There's some rules for correct display including with Bidi: In what sense are these "rules"? Where are these written, in what kind of specification or existing practice? > - Separate paragraphs that need a different default Bidi by double newlines > (to

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-07 Thread Richard Wordingham via Unicode
gt; > does. > > I tried it. Executed my default shell, and inside that, a "cat > TUTORIAL.he". All the paragraphs are rendered as LTR ones, > left-aligned. Not the way the file is opened in Emacs. See above. I don't know how what your shell is. > If you claim Ema

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-06 Thread Philippe Verdy via Unicode
I read your email, you spoke for example about how a typical Unix/Linux tool shows its usage option (e.g. "anycommand --help") with a leading line then syntaxes and tabulated lists of options followed by translated help on the same line. There's some rules for correct display including

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-06 Thread Egmont Koblinger via Unicode
. Not the way the file is opened in Emacs. If you claim Emacs's built-in terminal emulator supports BiDi, I'm kindly asking you to present a documentation of its behavior, in similar spirit to my BiDi proposal. > Not necessarily. One might use cat to glue together files that had > split into

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-06 Thread Richard Wordingham via Unicode
did a "cat TUTORIAL.he" (the file taken from > 26.1), and compared to what I see in Emacs 25.2.2 – both the graphical > one, and the one running in a terminal of no BiDi. > > Apart from a few minor irrelevant differences, they look the same! > Hooray!!! > > (The dif

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-06 Thread Egmont Koblinger via Unicode
Hi, I was loose with my terminology once again, which is not a wise thing when you're trying to clarify misunderstandings :) > But once you have > decided on a direction, each _line_ within that data is passed > separately to the BiDi algorithm to get reshuffled; this is what Ema

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-06 Thread Egmont Koblinger via Unicode
Hi Philippe, Thanks a lot for your input! Another fundamental difficulty with terminal emulators is: These controls (CR, LF...) are control instructions that move the cursor in some ways, and then are forgotten. You cannot do BiDi on the instructions the terminal receives. You can only do BiDi

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-06 Thread Egmont Koblinger via Unicode
nd the one running in a terminal of no BiDi. Apart from a few minor irrelevant differences, they look the same! Hooray!!! (The differences are: - I had to slightly modify TUTORIAL.he to make sure none of the lines start with a BiDi control (I added a preceding character) because currently VTE doesn't su

Re: mildly OT from bidi - curious email

2019-02-06 Thread Arthur Reutenauer via Unicode
] > > where and are the bidi control characters. > > I don't think I've seen this before - I wonder why it happened? Maybe Stripe stores merchant names with surrounding bidi control characters, so that they’re always rendered in the appropriate direction, even by systems that don’t implement t

mildly OT from bidi - curious email

2019-02-06 Thread Julian Bradfield via Unicode
The current bidi discussion prompts me to post a curiosity I received today. I ordered something from a (UK) company, and the payment receipt came via Stripe. So far, so common. The curious thing is that the (entirely ASCII) company name was enclosed in a left-to-right direction, thus: Subject

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-05 Thread Philippe Verdy via Unicode
I think that before making any decision we must make some decision about what we mean by "newlines". There are in fact 3 different functions: - (1) soft line breaks (which are used to enforce a maximum display width between paragraph margins): these are equivalent to breakable and compressible

Re: Bidi paragraph direction in terminal emulators

2019-02-05 Thread Eli Zaretskii via Unicode
itor at its core, so every job programmers routinely do from the shell prompt has an equivalent feature in Emacs. You can even run shells inside Emacs, with Emacs serving as a terminal emulator (which then supports bidi ;-). > There are just sooo many use cases, it's impossible to perfectly

Re: Bidi paragraph direction in terminal emulators

2019-02-05 Thread Eli Zaretskii via Unicode
t most of them won't ever do that. > > No, this simple case must work reasonably well with the application > > _completely_ oblivious to the bidi aspects. If this can't work > > reasonably well, I submit that the entire concept of having a > > bidi-aware terminal emulator doesn't "hold water&quo

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-05 Thread Eli Zaretskii via Unicode
> Date: Tue, 5 Feb 2019 00:05:47 + > From: Richard Wordingham via Unicode > > > > Actually, UAX#9 defines "paragraph" as the chunk of text delimited > > > by paragraph separator characters. This means characters whose bidi > > > categor

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators)

2019-02-05 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Tue, 5 Feb 2019 00:08:10 +0100 > Cc: unicode@unicode.org > > every single newline character starts a new paragraph. The result of > printf "Hello\nWorld\n" > world.txt > is a text file consisting of two paragraphs, with 5 characters in each. > Correct? Yes. >

Re: Proposal for BiDi in terminal emulators

2019-02-04 Thread Eli Zaretskii via Unicode
n display, like > > control characters. Other than that, there should be no doubt what > > visual order means. > > To me, 'visual order' means in the dominant order of the script. That is not the correct definition, IMO. > Furthermore, let me quote from the Bidi Algorit

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-04 Thread Richard Wordingham via Unicode
t that the recommendations of ECMA TR/53 had been implemented in Issue 5 of ECMA-48. > As for the BiDi docs, I found that the current state of the art, > current best practices, exisiting BiDi algorithm differ so much from > ECMA's approach (which no one I'm aware of cared to implement for

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators)

2019-02-04 Thread Egmont Koblinger via Unicode
st likely to separate some even bigger semantical units. There are just sooo many use cases, it's impossible to perfectly address all of them at once. "cat TUTORIAL.he" is just one of them, not necessarily the most typical, not necessarily the one that should drive the BiDi design.

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-04 Thread Egmont Koblinger via Unicode
Hi Eli, > I think it's unreasonable and impractical to expect 'echo', 'cat', and > its ilk to emit bidi controls (or any other controls) to force > paragraph direction. For starters, they won't know what direction to > force, because they don't understand the text they are processi

  1   2   3   4   5   6   >