Bug#1122519: xterm: display corruption with double-width characters

[Vincent Lefevre]

Control: close -1
Control: fixed -1 405-1

On 2025-12-16 20:31:35 -0500, Thomas Dickey wrote:
> On Wed, Dec 17, 2025 at 01:34:03AM +0100, Vincent Lefevre wrote:
> > I've attached xterm-bug.mbox, and viewing the message with Mutt gives
> 
> How would I view that?  (mutt -f xterm-bug.mbox gives an error)

I don't get an error, and I don't see why there would be an error.

> This isn't a bug, but something that I see that I should make configurable,
> because different applications are going to behave differently.

At least there is a bug somewhere since the display gets broken.

> If you want to discuss that, you should close the bug that you reopened,
> (because I did fix it) and we'll proceed with that.

OK, but note that the original intent of this bug was to report
broken display in Mutt with xterm > 403, and this still occurs.

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / Pascaline project (LIP, ENS-Lyon)

Bug#1122519: xterm: display corruption with double-width characters

[Vincent Lefevre]

On 2025-12-17 01:34:03 +0100, Vincent Lefevre wrote:
> To reproduce the issue more easily, I've attached Xterm.log.xz
> (compressed xterm log file obtained with the simple test on
> xterm-bug.mbox). Just run
> 
>   xterm -geometry 80x60 -e 'unxz -c Xterm.log.xz ; sleep 999'
> 
> See the differences between xterm-403.png and xterm-405.png (attached).

In the log file, there is the following sequence of characters:

  🧝🏻‍♂️🌋 Candlelight

U+1F9DD ELF
U+1F3FB EMOJI MODIFIER FITZPATRICK TYPE-1-2
U+200D ZERO WIDTH JOINER
U+2642 MALE SIGN
U+FE0F VARIATION SELECTOR-16
U+1F30B VOLCANO
U+0020 SPACE
U+0043 LATIN CAPITAL LETTER C
etc.

With xterm 403, I get:

U+1F9DD ELF
U+1F3FB EMOJI MODIFIER FITZPATRICK TYPE-1-2
U+2642 MALE SIGN
U+FE0F VARIATION SELECTOR-16
U+1F30B VOLCANO
U+0020 SPACE
U+0043 LATIN CAPITAL LETTER C
etc.

but with xterm 405, I get:

U+1F9DD ELF
U+1F3FB EMOJI MODIFIER FITZPATRICK TYPE-1-2
U+2642 MALE SIGN
U+FE0F VARIATION SELECTOR-16
U+0020 SPACE
U+1F30B VOLCANO
U+0020 SPACE
U+0043 LATIN CAPITAL LETTER C
etc.

i.e. there is an additional space after U+FE0F VARIATION SELECTOR-16.

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / Pascaline project (LIP, ENS-Lyon)

Bug#1122519: xterm: display corruption with double-width characters

[Thomas Dickey]

On Wed, Dec 17, 2025 at 01:34:03AM +0100, Vincent Lefevre wrote:
> Control: reopen -1
> Control: found -1 405-1
> 
> On 2025-12-12 15:41:50 -0500, Thomas Dickey wrote:
> > On Fri, Dec 12, 2025 at 09:03:06PM +0100, Sven Joachim wrote:
> > > On 2025-12-11 20:10 -0500, Thomas Dickey wrote:
> > > > I see the cause, will make an update within the next day or so.
> > > 
> > > Thanks.  Do you intend to release xterm 405 this weekend, or should I
> > > cherry-pick the changes from xterm 404-c?  I would prefer the former,
> > > but can do the latter.
> > 
> > I'll release 405 this weekend (otherwise I'll be getting other reports).
> 
> Unfortunately, this does not fix the issue entirely.
> 
> I've attached xterm-bug.mbox, and viewing the message with Mutt gives

How would I view that?  (mutt -f xterm-bug.mbox gives an error)

> a line overflow, in particular (my Mutt configuration may have an
> influence, but see below to reproduce the issue more easily).
> 
> With Mutt in GNU Screen, I still get something like
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=1122519;filename=20251211-003112.png;msg=5
> 
> and even worse.
> 
> To reproduce the issue more easily, I've attached Xterm.log.xz
> (compressed xterm log file obtained with the simple test on
> xterm-bug.mbox). Just run
> 
>   xterm -geometry 80x60 -e 'unxz -c Xterm.log.xz ; sleep 999'
> 
> See the differences between xterm-403.png and xterm-405.png (attached).

The example has a VS16, which tells xterm to display the preceding
character with two cells if it happens to be an Emoji.

The preceding character is U+2642, and according to Unicode 17 is an Emoji.
Unicode.org's guideline here is to make Emoji's fullwidth:

https://unicode.org/reports/tr51/

Current practice is for emoji to have a square aspect ratio, deriving
from their origin in Japanese.  For interoperability, it is recommended
that this practice be continued with current and future emoji.  They
will typically have about the same vertical placement and advance width
as CJK ideographs.

"square aspect ratio" is the same thing as "fullwidth".

...I verified the change here:

https://www.jeffquast.com/post/ucs-detect-test-results/

This isn't a bug, but something that I see that I should make configurable,
because different applications are going to behave differently.

If you want to discuss that, you should close the bug that you reopened,
(because I did fix it) and we'll proceed with that.

> >From a@b.invalid Wed Dec 10 10:31:31 2025
> From: a@b.invalid
> Subject: =?UTF-8?B?8J+nnfCfj7vigI3imYLvuI/wn4yLIENhbmRsZWxpZ2h0wqA6IExl?=
>  =?UTF-8?B?IFNlaWduZXVyIGRlcyBBbm5lYXV4IHMnaW5zdGFsbGUgw6AgTHlvbg==?=
> Date: Wed, 10 Dec 2025 03:31:22 -0600
> MIME-Version: 1.0

-- 
Thomas E. Dickey 
https://invisible-island.net


signature.asc
Description: PGP signature

Bug#1122519: xterm: display corruption with double-width characters

[Thomas Dickey]

On Wed, Dec 17, 2025 at 02:29:05AM +0100, Vincent Lefevre wrote:
> On 2025-12-17 01:34:03 +0100, Vincent Lefevre wrote:
> > To reproduce the issue more easily, I've attached Xterm.log.xz
> > (compressed xterm log file obtained with the simple test on
> > xterm-bug.mbox). Just run
> > 
> >   xterm -geometry 80x60 -e 'unxz -c Xterm.log.xz ; sleep 999'
> > 
> > See the differences between xterm-403.png and xterm-405.png (attached).
> 
> In the log file, there is the following sequence of characters:

Your screenshot shows that U+2642 uses one cell in xterm-403, and xterm
is using two cells in xterm-405.  In my previous reply I pointed to where
that comes from.

The change in #404 was limited to cell-size and cursor-movement, because
a full implementation of VS16 probably will involve some work on Xft.

(as a reminder, this is no longer bug #1122519).
 
> i.e. there is an additional space after U+FE0F VARIATION SELECTOR-16.

-- 
Thomas E. Dickey 
https://invisible-island.net


signature.asc
Description: PGP signature

Bug#1122519: xterm: display corruption with double-width characters

[Vincent Lefevre]

Control: reopen -1
Control: found -1 405-1

On 2025-12-12 15:41:50 -0500, Thomas Dickey wrote:
> On Fri, Dec 12, 2025 at 09:03:06PM +0100, Sven Joachim wrote:
> > On 2025-12-11 20:10 -0500, Thomas Dickey wrote:
> > > I see the cause, will make an update within the next day or so.
> > 
> > Thanks.  Do you intend to release xterm 405 this weekend, or should I
> > cherry-pick the changes from xterm 404-c?  I would prefer the former,
> > but can do the latter.
> 
> I'll release 405 this weekend (otherwise I'll be getting other reports).

Unfortunately, this does not fix the issue entirely.

I've attached xterm-bug.mbox, and viewing the message with Mutt gives
a line overflow, in particular (my Mutt configuration may have an
influence, but see below to reproduce the issue more easily).

With Mutt in GNU Screen, I still get something like

https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=1122519;filename=20251211-003112.png;msg=5

and even worse.

To reproduce the issue more easily, I've attached Xterm.log.xz
(compressed xterm log file obtained with the simple test on
xterm-bug.mbox). Just run

  xterm -geometry 80x60 -e 'unxz -c Xterm.log.xz ; sleep 999'

See the differences between xterm-403.png and xterm-405.png (attached).

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / Pascaline project (LIP, ENS-Lyon)
>From a@b.invalid Wed Dec 10 10:31:31 2025
From: a@b.invalid
Subject: =?UTF-8?B?8J+nnfCfj7vigI3imYLvuI/wn4yLIENhbmRsZWxpZ2h0wqA6IExl?=
 =?UTF-8?B?IFNlaWduZXVyIGRlcyBBbm5lYXV4IHMnaW5zdGFsbGUgw6AgTHlvbg==?=
Date: Wed, 10 Dec 2025 03:31:22 -0600
MIME-Version: 1.0



Xterm.log.xz
Description: Binary data

Bug#1122519: xterm: display corruption with double-width characters

[Sven Joachim]

Control: severity -1 serious

On 2025-12-12 15:41 -0500, Thomas Dickey wrote:

> On Fri, Dec 12, 2025 at 09:03:06PM +0100, Sven Joachim wrote:
>> On 2025-12-11 20:10 -0500, Thomas Dickey wrote:
>> 
>> > On Thu, Dec 11, 2025 at 05:16:14PM -0500, Thomas Dickey wrote:
>> >> On Thu, Dec 11, 2025 at 01:53:38AM +0100, Vincent Lefevre wrote:
>> >> > Package: xterm
>> >> > Version: 404-1
>> >> > Severity: important
>> > ...
>> >> fwiw, all of the characters are U+1F9DD, which we're agreed should be two
>> >> cells, and the VS15/VS16 code shouldn't be a factor.  I'll have to bisect
>> >> my changes to see why this happens.
>> >
>> > I see the cause, will make an update within the next day or so.
>> 
>> Thanks.  Do you intend to release xterm 405 this weekend, or should I
>> cherry-pick the changes from xterm 404-c?  I would prefer the former,
>> but can do the latter.
>
> I'll release 405 this weekend (otherwise I'll be getting other reports).

Okay, I will wait for that then.  In the meantime, I am bumping the
severity to prevent this bug from migrating to testing.

Cheers,
   Sven

Bug#1122519: xterm: display corruption with double-width characters

[Thomas Dickey]

On Fri, Dec 12, 2025 at 09:03:06PM +0100, Sven Joachim wrote:
> On 2025-12-11 20:10 -0500, Thomas Dickey wrote:
> 
> > On Thu, Dec 11, 2025 at 05:16:14PM -0500, Thomas Dickey wrote:
> >> On Thu, Dec 11, 2025 at 01:53:38AM +0100, Vincent Lefevre wrote:
> >> > Package: xterm
> >> > Version: 404-1
> >> > Severity: important
> > ...
> >> fwiw, all of the characters are U+1F9DD, which we're agreed should be two
> >> cells, and the VS15/VS16 code shouldn't be a factor.  I'll have to bisect
> >> my changes to see why this happens.
> >
> > I see the cause, will make an update within the next day or so.
> 
> Thanks.  Do you intend to release xterm 405 this weekend, or should I
> cherry-pick the changes from xterm 404-c?  I would prefer the former,
> but can do the latter.

I'll release 405 this weekend (otherwise I'll be getting other reports).

-- 
Thomas E. Dickey 
https://invisible-island.net


signature.asc
Description: PGP signature

Bug#1122519: xterm: display corruption with double-width characters

[Sven Joachim]

On 2025-12-11 20:10 -0500, Thomas Dickey wrote:

> On Thu, Dec 11, 2025 at 05:16:14PM -0500, Thomas Dickey wrote:
>> On Thu, Dec 11, 2025 at 01:53:38AM +0100, Vincent Lefevre wrote:
>> > Package: xterm
>> > Version: 404-1
>> > Severity: important
> ...
>> fwiw, all of the characters are U+1F9DD, which we're agreed should be two
>> cells, and the VS15/VS16 code shouldn't be a factor.  I'll have to bisect
>> my changes to see why this happens.
>
> I see the cause, will make an update within the next day or so.

Thanks.  Do you intend to release xterm 405 this weekend, or should I
cherry-pick the changes from xterm 404-c?  I would prefer the former,
but can do the latter.

Cheers,
   Sven

Bug#1122519: xterm: display corruption with double-width characters

[Thomas Dickey]

On Thu, Dec 11, 2025 at 05:16:14PM -0500, Thomas Dickey wrote:
> On Thu, Dec 11, 2025 at 01:53:38AM +0100, Vincent Lefevre wrote:
> > Package: xterm
> > Version: 404-1
> > Severity: important
...
> fwiw, all of the characters are U+1F9DD, which we're agreed should be two
> cells, and the VS15/VS16 code shouldn't be a factor.  I'll have to bisect
> my changes to see why this happens.

I see the cause, will make an update within the next day or so.

-- 
Thomas E. Dickey 
https://invisible-island.net


signature.asc
Description: PGP signature

Bug#1122519: xterm: display corruption with double-width characters

[Thomas Dickey]

On Thu, Dec 11, 2025 at 01:53:38AM +0100, Vincent Lefevre wrote:
> Package: xterm
> Version: 404-1
> Severity: important
> 
> Display is corrupted when double-width characters are displayed.
> In particular, this affects Mutt (receiving mail with such characters
> is common), and one consequence is that wrong mail can be deleted
> because the cursor does not appear on the right message! I've
> attached a part of a screenshot (20251211-003112.png) to show
> the kind of output I get.
> 
> There is no such issue with rxvt and GNOME Terminal.
> 
> To reproduce a display issue, open a 80-column xterm, and generate
> output so that the shell prompt is at the very bottom. Then paste
> the following command:
> 
> printf "🧝🧝🧝\n"
> 
> The last 3 characters of the command (\n") do not appear (see attached
> screenshot 20251211-014531.png). They appear one character at a time
> when I type the left arrow. There is the same issue when I recall this
> command from the history (if the prompt is at the very bottom).

I can see some cursor-misplacement, using bash and zsh, not dash,
when pasting that text without a line-ending (so that the cursor
would be at the end of the line).

fwiw, all of the characters are U+1F9DD, which we're agreed should be two
cells, and the VS15/VS16 code shouldn't be a factor.  I'll have to bisect
my changes to see why this happens.

-- 
Thomas E. Dickey 
https://invisible-island.net


signature.asc
Description: PGP signature

Bug#1122519: xterm: display corruption with double-width characters

[Vincent Lefevre]

On 2025-12-11 01:53:38 +0100, Vincent Lefevre wrote:
> There is no such issue with rxvt and GNOME Terminal.

No such issue with Sakura either.

With mlterm, there are also display issues with Mutt, but the issue
with the test below is not reproducible.

For xterm, this is a regression: there are no issues at all with
xterm 403-1.

> To reproduce a display issue, open a 80-column xterm, and generate
> output so that the shell prompt is at the very bottom. Then paste
> the following command:
> 
> printf "🧝🧝🧝\n"
> 
> The last 3 characters of the command (\n") do not appear (see attached
> screenshot 20251211-014531.png). They appear one character at a time
> when I type the left arrow. There is the same issue when I recall this
> command from the history (if the prompt is at the very bottom).

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / Pascaline project (LIP, ENS-Lyon)

Bug#1122519: xterm: display corruption with double-width characters

[Vincent Lefevre]

Package: xterm
Version: 404-1
Severity: important

Display is corrupted when double-width characters are displayed.
In particular, this affects Mutt (receiving mail with such characters
is common), and one consequence is that wrong mail can be deleted
because the cursor does not appear on the right message! I've
attached a part of a screenshot (20251211-003112.png) to show
the kind of output I get.

There is no such issue with rxvt and GNOME Terminal.

To reproduce a display issue, open a 80-column xterm, and generate
output so that the shell prompt is at the very bottom. Then paste
the following command:

printf "🧝🧝🧝\n"

The last 3 characters of the command (\n") do not appear (see attached
screenshot 20251211-014531.png). They appear one character at a time
when I type the left arrow. There is the same issue when I recall this
command from the history (if the prompt is at the very bottom).

-- System Information:
Debian Release: forky/sid
  APT prefers unstable-debug
  APT policy: (500, 'unstable-debug'), (500, 'stable-updates'), (500, 
'stable-security'), (500, 'stable-debug'), (500, 'proposed-updates-debug'), 
(500, 'unstable'), (500, 'testing'), (500, 'stable'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 6.17.10+deb14-amd64 (SMP w/16 CPU threads; PREEMPT)
Locale: LANG=C.UTF-8, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages xterm depends on:
ii  libc6   2.42-5
ii  libfontconfig1  2.15.0-2.4
ii  libfreetype62.13.3+dfsg-1
ii  libice6 2:1.1.1-1
ii  libtinfo6   6.5+20251123-1
ii  libutempter01.2.1-4
ii  libx11-62:1.8.12-1
ii  libxaw7 2:1.0.16-1
ii  libxext62:1.3.4-1+b3
ii  libxft2 2.3.6-1+b4
ii  libxinerama12:1.1.4-3+b4
ii  libxmu6 2:1.1.3-3+b4
ii  libxpm4 1:3.5.17-1+b3
ii  libxt6t64   1:1.2.1-1.3
ii  xbitmaps1.1.1-2.2

Versions of packages xterm recommends:
ii  luit [luit]  2.0.20250912-1
ii  x11-utils7.7+7

Versions of packages xterm suggests:
pn  xfonts-cyrillic  

-- no debconf information

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / Pascaline project (LIP, ENS-Lyon)