>Synopsis: UTF-8 canonical erase splits characters and desynchronizes echo >Category: kernel >Environment: System: OpenBSD 7.9 Kernel: GENERIC.MP #16, Mon Oct 5 13:23:15 MDT 2026 Kernel build: [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP Architecture: amd64 Terminal: XTerm/OpenBSD(406), under X11 TERM=xterm LC_CTYPE=en_US.UTF-8 LC_ALL is unset stty: icanon echo echoe iexten, -istrip, -extproc, erase=^?
>Description: While entering text interactively in the base system ed, Backspace removes only one byte of a two-byte UTF-8 character. This leaves invalid UTF-8 after one Backspace. After a second Backspace, the editing echo no longer matches the actual input: the terminal displays "a", while the line received by ed is "ab". The ASCII b disappears from the display but remains in the data. This resembles a problem discussed in December 2017: https://www.mail-archive.com/[email protected]/msg157917.html Inspection of the current source suggests that this occurs in the TTY canonical ERASE handling, before ed receives the input. In sys/kern/tty.c, ttyinput(), the ERASE handler calls: ret = ttyrub(unputc(&tp->t_rawq), tp); unputc() removes one byte, while ttyrub() calls ttyrubo(tp, 1) for an ordinary byte. That erases one screen position. For a two-byte UTF-8 character occupying one screen position, this explains the observed mismatch. This source inspection was of -current, not a local build of my installed kernel. >How-To-Repeat: Run the tests interactively in xterm. For each test, start a fresh /bin/ed session. 1. Enter "a" followed by Enter to start input mode. 2. Type the text and press Backspace as described below. 3. Press Enter. 4. Enter "." followed by Enter to finish input mode. 5. Enter "l" followed by Enter to inspect the line. 6. Enter "Q" followed by Enter to exit without saving. The character č is U+010D, encoded in UTF-8 as C4 8D. The $ in the results below is the end-of-line marker printed by ed's l command, not part of the input. Test A: Type abč without pressing Backspace. Actual l output: ab\304\215$ Test B: Type abč and press Backspace once. Actual l output: ab\304$ The byte 8D has been removed, leaving the incomplete UTF-8 byte C4. Test C: Type abč and press Backspace twice. During input, only "a" remains visible. Actual l output after completing input: ab$ Expected behaviour for UTF-8-aware erase: One Backspace would remove the complete č character and leave ab. Two Backspaces would leave a. In both cases, the editing echo would match the actual input. >Fix: No patch proposed. Is this still a known limitation of canonical TTY input handling? Is there a supported workaround for interactive ed input in a UTF-8 terminal? Regards, Tom Terminal and system diagnostics follow. $ locale LANG= LC_COLLATE="C" LC_CTYPE=en_US.UTF-8 LC_MONETARY="C" LC_NUMERIC="C" LC_TIME="C" LC_MESSAGES="C" LC_ALL= $ stty -a speed 38400 baud; 24 rows; 80 columns; lflags: icanon isig iexten echo echoe echok echoke -echonl echoctl -echoprt -altwerase -noflsh -tostop -flusho pendin -nokerninfo -extproc -xcase iflags: -istrip icrnl -inlcr -igncr -iuclc ixon -ixoff -ixany -imaxbel -ignbrk -brkint -inpck -ignpar -parmrk oflags: opost onlcr -ocrnl -onocr -onlret -olcuc oxtabs -onoeot cflags: cread cs8 parenb -parodd hupcl -clocal -cstopb -crtscts -mdmbuf cchars: discard = ^O; dsusp = ^Y; eof = ^D; eol = <undef>; eol2 = ^@; erase = ^?; intr = ^C; kill = ^U; lnext = ^V; min = 1; quit = ^\; reprint = ^R; start = ^Q; status = <undef>; stop = ^S; susp = ^Z; time = 0; werase = ^W; $ printf '%s\n' "$TERM" xterm $ xterm -version XTerm/OpenBSD(406) dmesg: OpenBSD 7.9 (GENERIC.MP) #16: Mon Oct 5 13:23:15 MDT 2026 [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 16963280896 (16177MB) avail mem = 16421437440 (15660MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 3.1 @ 0xe0000 (83 entries) bios0: vendor Dell Inc. version "1.34.0" date 06/30/2025 bios0: Dell Inc. OptiPlex 3070 efi0 at bios0: UEFI 2.7 efi0: American Megatrends rev 0x5000d acpi0 at bios0: ACPI 6.1 acpi0: sleep states S0 S3 S4 S5 acpi0: tables DSDT FACP APIC FPDT FIDT MCFG SSDT BOOT SSDT HPET SSDT SSDT UEFI LPIT SSDT SSDT DBGP DBG2 MSDM SLIC DMAR SSDT BGRT TPM2 WSMT acpi0: wakeup devices RP01(S4) PXSX(S4) RP02(S4) PXSX(S4) RP03(S4) PXSX(S4) RP04(S4) PXSX(S4) RP05(S4) PXSX(S4) RP06(S4) PXSX(S4) RP07(S4) PXSX(S4) RP08(S4) PXSX(S4) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Core(TM) i5-9500T CPU @ 2.20GHz, 2095.12 MHz, 06-9e-0a, patch 000000fa cpu0: cpuid 1 edx=bfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> ecx=77fafbff<SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND> cpu0: cpuid 6 eax=27f7<SENSOR,ARAT,PTS> ecx=9<EFFFREQ> cpu0: cpuid 7.0 ebx=29c6fbf<FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT> ecx=40000000 edx=bc002e00<SRBDS_CTRL,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD> cpu0: cpuid a vers=4, gp=8, gpwidth=48, ff=3, ffwidth=48 cpu0: cpuid d.1 eax=f<XSAVEOPT,XSAVEC,XGETBV1,XSAVES> cpu0: cpuid 80000001 edx=2c100800<NXE,PAGE1GB,RDTSCP,LONG> ecx=121<LAHF,ABM,3DNOWP> cpu0: cpuid 80000007 edx=100<ITSC> cpu0: msr 10a=a000c04<RSBA,MISC_PKG_CT,ENERGY_FILT,GDS_CTRL,RFDS_NO> cpu0: MELTDOWN cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB 64b/line 4-way L2 cache, 9MB 64b/line 12-way L3 cache cpu0: smt 0, core 0, package 0, type P mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 24MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1.1.1, IBE cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Core(TM) i5-9500T CPU @ 2.20GHz, 2095.12 MHz, 06-9e-0a, patch 000000fa cpu1: smt 0, core 1, package 0, type P cpu2 at mainbus0: apid 4 (application processor) cpu2: Intel(R) Core(TM) i5-9500T CPU @ 2.20GHz, 2095.12 MHz, 06-9e-0a, patch 000000fa cpu2: smt 0, core 2, package 0, type P cpu3 at mainbus0: apid 6 (application processor) cpu3: Intel(R) Core(TM) i5-9500T CPU @ 2.20GHz, 2095.12 MHz, 06-9e-0a, patch 000000fa cpu3: smt 0, core 3, package 0, type P cpu4 at mainbus0: apid 8 (application processor) cpu4: Intel(R) Core(TM) i5-9500T CPU @ 2.20GHz, 2095.12 MHz, 06-9e-0a, patch 000000fa cpu4: smt 0, core 4, package 0, type P cpu5 at mainbus0: apid 10 (application processor) cpu5: Intel(R) Core(TM) i5-9500T CPU @ 2.20GHz, 2095.12 MHz, 06-9e-0a, patch 000000fa cpu5: smt 0, core 5, package 0, type P ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 120 pins acpimcfg0 at acpi0 acpimcfg0: addr 0xf8000000, bus 0-63 acpihpet0 at acpi0: 23999999 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus -1 (PEG1) acpiprt2 at acpi0: bus -1 (PEG2) acpiprt3 at acpi0: bus -1 (RP01) acpiprt4 at acpi0: bus -1 (RP02) acpiprt5 at acpi0: bus -1 (RP03) acpiprt6 at acpi0: bus -1 (RP04) acpiprt7 at acpi0: bus 1 (RP05) acpiprt8 at acpi0: bus -1 (RP06) acpiprt9 at acpi0: bus -1 (RP07) acpiprt10 at acpi0: bus -1 (RP08) acpiprt11 at acpi0: bus -1 (RP10) acpiprt12 at acpi0: bus -1 (RP11) acpiprt13 at acpi0: bus -1 (RP12) acpiprt14 at acpi0: bus -1 (RP13) acpiprt15 at acpi0: bus -1 (RP14) acpiprt16 at acpi0: bus -1 (RP15) acpiprt17 at acpi0: bus -1 (RP16) acpiprt18 at acpi0: bus -1 (RP17) acpiprt19 at acpi0: bus -1 (RP18) acpiprt20 at acpi0: bus -1 (RP19) acpiprt21 at acpi0: bus -1 (RP20) acpiprt22 at acpi0: bus -1 (RP21) acpiprt23 at acpi0: bus -1 (RP22) acpiprt24 at acpi0: bus -1 (RP23) acpiprt25 at acpi0: bus -1 (RP24) acpiprt26 at acpi0: bus -1 (PEG0) acpiprt27 at acpi0: bus 2 (RP09) acpipci0 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000 com0 at acpi0 UAR1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo acpicmos0 at acpi0 acpibtn0 at acpi0: SLPB "PNP0C14" at acpi0 not configured "PNP0C14" at acpi0 not configured intelpmc0 at acpi0: PEPD state 0: 0x7f:1:2:0x00:0x0000000000000060 counter: 0x7f:64:0:0x00:0x0000000000000632 frequency: 0 state 1: 0x7f:1:2:0x00:0x0000000000000060 counter: 0x00:32:0:0x03:0x00000000fe00193c frequency: 9580 acpibtn1 at acpi0: PWRB tpm0 at acpi0 TPM_ 2.0 (TIS) addr 0xfed40000/0x5000, device 0x00fc1050 rev 0x1 "PNP0C14" at acpi0 not configured "PNP0C14" at acpi0 not configured "PNP0C14" at acpi0 not configured "PNP0C0B" at acpi0 not configured "PNP0C0B" at acpi0 not configured "PNP0C0B" at acpi0 not configured "PNP0C0B" at acpi0 not configured "PNP0C0B" at acpi0 not configured acpipwrres0 at acpi0: USBC, resource for XDCI acpipwrres1 at acpi0: V0PR acpipwrres2 at acpi0: V1PR acpipwrres3 at acpi0: V2PR acpipwrres4 at acpi0: WRST acpicpu0 at acpi0: C3(200@1034 mwait.1@0x60), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS acpicpu1 at acpi0: C3(200@1034 mwait.1@0x60), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS acpicpu2 at acpi0: C3(200@1034 mwait.1@0x60), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS acpicpu3 at acpi0: C3(200@1034 mwait.1@0x60), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS acpicpu4 at acpi0: C3(200@1034 mwait.1@0x60), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS acpicpu5 at acpi0: C3(200@1034 mwait.1@0x60), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS acpipwrres5 at acpi0: FN00, resource for FAN0 acpipwrres6 at acpi0: FN01, resource for FAN1 acpipwrres7 at acpi0: FN02, resource for FAN2 acpipwrres8 at acpi0: FN03, resource for FAN3 acpipwrres9 at acpi0: FN04, resource for FAN4 acpitz0 at acpi0 acpitz0: critical temperature is 119 degC acpipwrres10 at acpi0: PIN_ acpivideo0 at acpi0: GFX0 acpivout0 at acpivideo0: DD1F cpu0: using VERW MDS workaround (except on vmm entry) cpu0: Enhanced SpeedStep 2095 MHz: speeds: 2201, 2200, 2100, 2000, 1900, 1800, 1700, 1600, 1500, 1400, 1300, 1200, 1100, 1000, 900, 800 MHz pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel Core 8G Host" rev 0x07 inteldrm0 at pci0 dev 2 function 0 "Intel UHD Graphics 630" rev 0x00 drm0 at inteldrm0 inteldrm0: msi, COFFEELAKE, gen 9 "Intel Core GMM" rev 0x00 at pci0 dev 8 function 0 not configured pchtemp0 at pci0 dev 18 function 0 "Intel 300 Series Thermal" rev 0x10 xhci0 at pci0 dev 20 function 0 "Intel 300 Series xHCI" rev 0x10: msi, xHCI 1.10 usb0 at xhci0: USB revision 3.0 uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00 addr 1 "Intel 300 Series Shared SRAM" rev 0x10 at pci0 dev 20 function 2 not configured "Intel 300 Series HECI" rev 0x10 at pci0 dev 22 function 0 not configured ahci0 at pci0 dev 23 function 0 "Intel 300 Series AHCI" rev 0x10: msi, AHCI 1.3.1 ahci0: PHY offline on port 0 scsibus1 at ahci0: 32 targets ppb0 at pci0 dev 28 function 0 "Intel 300 Series PCIE" rev 0xf0: msi pci1 at ppb0 bus 1 re0 at pci1 dev 0 function 0 "Realtek 8168" rev 0x15: RTL8168H/8111H (0x5400), msi, address 00:4e:01:c2:83:14 rgephy0 at re0 phy 7: RTL8251, rev. 0 ppb1 at pci0 dev 29 function 0 "Intel 300 Series PCIE" rev 0xf0: msi pci2 at ppb1 bus 2 nvme0 at pci2 dev 0 function 0 "Realtek RTS5765DL" rev 0x01: msix, NVMe 1.4 nvme0: Patriot M.2 P300 1024GB, firmware VC3S500Q, serial P300WCBA24090722572 scsibus2 at nvme0: 2 targets, initiator 0 sd0 at scsibus2 targ 1 lun 0: <NVMe, Patriot M.2 P300, VC3S> sd0: 976762MB, 512 bytes/sector, 2000409264 sectors pcib0 at pci0 dev 31 function 0 "Intel H370 LPC" rev 0x10 azalia0 at pci0 dev 31 function 3 "Intel 300 Series cAVS" rev 0x10: msi azalia0: codecs: Realtek ALC255, Intel/0x280b, using Realtek ALC255 audio0 at azalia0 ichiic0 at pci0 dev 31 function 4 "Intel 300 Series SMBus" rev 0x10: apic 2 int 21 iic0 at ichiic0 spdmem0 at iic0 addr 0x52: 16GB DDR4 SDRAM PC4-21300 SO-DIMM "Intel 300 Series SPI" rev 0x10 at pci0 dev 31 function 5 not configured isa0 at pcib0 isadma0 at isa0 pckbc0 at isa0 port 0x60/5 irq 1 irq 12 pckbd0 at pckbc0 (kbd slot) wskbd0 at pckbd0: console keyboard pcppi0 at isa0 port 0x61 spkr0 at pcppi0 vmm0 at mainbus0: VMX/EPT efifb at mainbus0 not configured uhidev0 at uhub0 port 1 configuration 1 interface 0 "Logitech USB Receiver" rev 2.00/5.03 addr 2 uhidev0: iclass 3/1 ukbd0 at uhidev0: 8 variable keys, 6 key codes wskbd1 at ukbd0 mux 1 uhidev1 at uhub0 port 1 configuration 1 interface 1 "Logitech USB Receiver" rev 2.00/5.03 addr 2 uhidev1: iclass 3/1, 11 report ids ums0 at uhidev1 reportid 2: 16 buttons, Z and W dir wsmouse0 at ums0 mux 0 ucc0 at uhidev1 reportid 3: 767 usages, 20 keys, array wskbd2 at ucc0 mux 1 uhid0 at uhidev1 reportid 4: input=1, output=0, feature=0 uhid1 at uhidev1 reportid 11: input=2, output=0, feature=0 uhidev2 at uhub0 port 1 configuration 1 interface 2 "Logitech USB Receiver" rev 2.00/5.03 addr 2 uhidev2: iclass 3/0, 17 report ids uhidpp0 at uhidev2 device 1 mouse "MX Ergo S", device 2 mouse "MX Ergo S", device 3 keyboard "MX KEYS S", device 4 keyboard "MX Keys Mini", device 5 keyboard "MX KEYS S" uhidev3 at uhub0 port 1 configuration 1 interface 3 "Logitech USB Receiver" rev 2.00/5.03 addr 2 uhidev3: iclass 3/0, 42 report ids uhid2 at uhidev3 reportid 40: input=28, output=0, feature=0 uhid3 at uhidev3 reportid 41: input=0, output=0, feature=1 uhid4 at uhidev3 reportid 42: input=0, output=0, feature=256 uhidev4 at uhub0 port 7 configuration 1 interface 0 "Logitech USB Receiver" rev 2.00/5.03 addr 3 uhidev4: iclass 3/1 ukbd1 at uhidev4: 8 variable keys, 6 key codes wskbd3 at ukbd1 mux 1 uhidev5 at uhub0 port 7 configuration 1 interface 1 "Logitech USB Receiver" rev 2.00/5.03 addr 3 uhidev5: iclass 3/1, 11 report ids ums1 at uhidev5 reportid 2: 16 buttons, Z and W dir wsmouse1 at ums1 mux 0 ucc1 at uhidev5 reportid 3: 767 usages, 20 keys, array wskbd4 at ucc1 mux 1 uhid5 at uhidev5 reportid 4: input=1, output=0, feature=0 uhid6 at uhidev5 reportid 11: input=2, output=0, feature=0 uhidev6 at uhub0 port 7 configuration 1 interface 2 "Logitech USB Receiver" rev 2.00/5.03 addr 3 uhidev6: iclass 3/0, 17 report ids uhidpp1 at uhidev6 device 1 keyboard "MX KEYS S" uhidev7 at uhub0 port 7 configuration 1 interface 3 "Logitech USB Receiver" rev 2.00/5.03 addr 3 uhidev7: iclass 3/0, 42 report ids uhid7 at uhidev7 reportid 40: input=28, output=0, feature=0 uhid8 at uhidev7 reportid 41: input=0, output=0, feature=1 uhid9 at uhidev7 reportid 42: input=0, output=0, feature=256 vscsi0 at root scsibus3 at vscsi0: 256 targets softraid0 at root scsibus4 at softraid0: 256 targets root on sd0a (aba37c37a4f98ea9.a) swap on sd0b dump on sd0b inteldrm0: 1920x2160, 32bpp wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation), using wskbd0 wskbd1: connecting to wsdisplay0 wskbd2: connecting to wsdisplay0 wskbd3: connecting to wsdisplay0 wskbd4: connecting to wsdisplay0 wsdisplay0: screen 1-5 added (std, vt100 emulation) usbdevs: Controller /dev/usb0: addr 01: 8086:0000 Intel, xHCI root hub super speed, self powered, config 1, rev 1.00 driver: uhub0 addr 02: 046d:c548 Logitech, USB Receiver full speed, power 98 mA, config 1, rev 5.03 driver: uhidev0 driver: uhidev1 driver: uhidev2 driver: uhidev3 addr 03: 046d:c548 Logitech, USB Receiver full speed, power 98 mA, config 1, rev 5.03 driver: uhidev4 driver: uhidev5 driver: uhidev6 driver: uhidev7
