Re: [PATCH v2 09/67] ui/console-vc: add UTF-8 input decoding with CP437 rendering
On Mon, Apr 20, 2026 at 11:54:40AM +0400, Marc-André Lureau wrote: > Hi > > On Wed, Apr 15, 2026 at 3:24 PM Daniel P. Berrangé > wrote: > > > > On Fri, Apr 10, 2026 at 11:18:31PM +0400, Marc-André Lureau wrote: > > > The text console receives bytes that may be UTF-8 encoded (e.g. from > > > a guest running a modern distro), but currently treats each byte as a > > > raw character index into the VGA/CP437 font, producing garbled output > > > for any multi-byte sequence. > > > > Presumably the key words here are "may be" as in, it > > also "may NOT be" UTF-8. > > > > IIUC, the current code is assuming that all data from the guest > > is in the CP437 encoding (8-bit Extended ASCII), and that encoding > > has valid characters for all 256 code points. > > > > By adding UTF-8 decoding for val > 0x80 this is breaking compat > > with any guest that is outputting data with the full range of > > CP437. > > > > IOW, this patch is moving the brokeness from guests which > > use UTF8, onto guests which use CP437. > > > > Only guests which strictly limit themselves to 7-bit ASCII > > are unaffected. > > > > I accept the UTF8 should probably be considered the common > > case for modern guests, but this hardcoding a different type > > of breakage feels undesirable to me. > > > > Surely we need an explicit config property here to select > > the between character sets we expect from the guest ? > > Probably, I don't know if many guest/apps rely on the serial encoding, > but we should probably be conservative. > > Adjusting the decoding at runtime may be possible, but it could be tricky. > > Instead, we could add a vc chardev option like charset=cp437/utf8. > > What should be the default? For compatibility reasons, use CP437 for > pc machine <=11.0 and default to utf8 for others? wdyt? Strictly speaking this is not guest ABI, just a change in defaults for the backend. So as long as we provide the config option, we could potentially just change the default to UTF8 unconditionally, on the basis that UTF8 has been the default charset in mainstream Linux for 20 years. Not sure what Windows uses by default, but use of the serial console with Linux guests is much more likely than Windows guests IMHO.
Re: [PATCH v2 09/67] ui/console-vc: add UTF-8 input decoding with CP437 rendering
Hi
On Wed, Apr 15, 2026 at 3:24 PM Daniel P. Berrangé wrote:
>
> On Fri, Apr 10, 2026 at 11:18:31PM +0400, Marc-André Lureau wrote:
> > The text console receives bytes that may be UTF-8 encoded (e.g. from
> > a guest running a modern distro), but currently treats each byte as a
> > raw character index into the VGA/CP437 font, producing garbled output
> > for any multi-byte sequence.
>
> Presumably the key words here are "may be" as in, it
> also "may NOT be" UTF-8.
>
> IIUC, the current code is assuming that all data from the guest
> is in the CP437 encoding (8-bit Extended ASCII), and that encoding
> has valid characters for all 256 code points.
>
> By adding UTF-8 decoding for val > 0x80 this is breaking compat
> with any guest that is outputting data with the full range of
> CP437.
>
> IOW, this patch is moving the brokeness from guests which
> use UTF8, onto guests which use CP437.
>
> Only guests which strictly limit themselves to 7-bit ASCII
> are unaffected.
>
> I accept the UTF8 should probably be considered the common
> case for modern guests, but this hardcoding a different type
> of breakage feels undesirable to me.
>
> Surely we need an explicit config property here to select
> the between character sets we expect from the guest ?
Probably, I don't know if many guest/apps rely on the serial encoding,
but we should probably be conservative.
Adjusting the decoding at runtime may be possible, but it could be tricky.
Instead, we could add a vc chardev option like charset=cp437/utf8.
What should be the default? For compatibility reasons, use CP437 for
pc machine <=11.0 and default to utf8 for others? wdyt?
>
> >
> > Add a UTF-8 decoder using Bjoern Hoehrmann's DFA. The DFA inherently
> > rejects overlong encodings, surrogates, and codepoints above U+10.
> > Completed codepoints are then mapped to CP437, unmappable characters are
> > displayed as '?'.
> >
> > Note that QEMU has a "buffered" utf8 decoder in util/unicode.c, but
> > it is not a good fit for byte-per-byte decoding.
> >
> > Signed-off-by: Marc-André Lureau
> > ---
> > ui/cp437.h | 13
> > ui/console-vc.c | 61 +
> > ui/cp437.c | 205
> >
> > ui/meson.build | 2 +-
> > 4 files changed, 280 insertions(+), 1 deletion(-)
> >
> > diff --git a/ui/cp437.h b/ui/cp437.h
> > new file mode 100644
> > index 000..81ace8317c7
> > --- /dev/null
> > +++ b/ui/cp437.h
> > @@ -0,0 +1,13 @@
> > +/*
> > + * SPDX-License-Identifier: GPL-2.0-or-later
> > + *
> > + * Copyright (c) QEMU contributors
> > + */
> > +#ifndef QEMU_CP437_H
> > +#define QEMU_CP437_H
> > +
> > +#include
> > +
> > +int unicode_to_cp437(uint32_t codepoint);
> > +
> > +#endif /* QEMU_CP437_H */
> > diff --git a/ui/console-vc.c b/ui/console-vc.c
> > index 8dee1f9bd01..d9d6966410a 100644
> > --- a/ui/console-vc.c
> > +++ b/ui/console-vc.c
> > @@ -9,6 +9,7 @@
> > #include "qemu/fifo8.h"
> > #include "qemu/option.h"
> > #include "ui/console.h"
> > +#include "ui/cp437.h"
> >
> > #include "trace.h"
> > #include "console-priv.h"
> > @@ -89,6 +90,8 @@ struct VCChardev {
> > enum TTYState state;
> > int esc_params[MAX_ESC_PARAMS];
> > int nb_esc_params;
> > +uint32_t utf8_state; /* UTF-8 DFA decoder state */
> > +uint32_t utf8_codepoint; /* accumulated UTF-8 code point */
> > TextAttributes t_attrib; /* currently active text attributes */
> > TextAttributes t_attrib_saved;
> > int x_saved, y_saved;
> > @@ -598,6 +601,46 @@ static void vc_clear_xy(VCChardev *vc, int x, int y)
> > vc_update_xy(vc, x, y);
> > }
> >
> > +/*
> > + * UTF-8 DFA decoder by Bjoern Hoehrmann.
> > + * Copyright (c) 2008-2010 Bjoern Hoehrmann
> > + * See https://github.com/polijan/utf8_decode for details.
> > + *
> > + * SPDX-License-Identifier: MIT
> > + */
> > +#define BH_UTF8_ACCEPT 0
> > +#define BH_UTF8_REJECT 12
> > +
> > +static uint32_t bh_utf8_decode(uint32_t *state, uint32_t *codep, uint32_t
> > byte)
> > +{
> > +static const uint8_t utf8d[] = {
> > +/* character class lookup */
> > +0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
> > +0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
> > +0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
> > +0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
> > +1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, 9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,
> > +7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7, 7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,
> > +8,8,2,2,2,2,2,2,2,2,2,2,2,2,2,2, 2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,
> > +10,3,3,3,3,3,3,3,3,3,3,3,3,4,3,3, 11,6,6,6,5,8,8,8,8,8,8,8,8,8,8,8,
> > +
> > +/* state transition lookup */
> > +0,12,24,36,60,96,84,12,12,12,48,72,
> > 12,12,12,12,12,12,12,12,12,12,12,12,
> > +12, 0,12,12,12,12,12, 0,12, 0,12,12,
> > 12,24,12,12,12,12,12,24,12,24,12,1
Re: [PATCH v2 09/67] ui/console-vc: add UTF-8 input decoding with CP437 rendering
On Fri, Apr 10, 2026 at 11:18:31PM +0400, Marc-André Lureau wrote:
> The text console receives bytes that may be UTF-8 encoded (e.g. from
> a guest running a modern distro), but currently treats each byte as a
> raw character index into the VGA/CP437 font, producing garbled output
> for any multi-byte sequence.
Presumably the key words here are "may be" as in, it
also "may NOT be" UTF-8.
IIUC, the current code is assuming that all data from the guest
is in the CP437 encoding (8-bit Extended ASCII), and that encoding
has valid characters for all 256 code points.
By adding UTF-8 decoding for val > 0x80 this is breaking compat
with any guest that is outputting data with the full range of
CP437.
IOW, this patch is moving the brokeness from guests which
use UTF8, onto guests which use CP437.
Only guests which strictly limit themselves to 7-bit ASCII
are unaffected.
I accept the UTF8 should probably be considered the common
case for modern guests, but this hardcoding a different type
of breakage feels undesirable to me.
Surely we need an explicit config property here to select
the between character sets we expect from the guest ?
>
> Add a UTF-8 decoder using Bjoern Hoehrmann's DFA. The DFA inherently
> rejects overlong encodings, surrogates, and codepoints above U+10.
> Completed codepoints are then mapped to CP437, unmappable characters are
> displayed as '?'.
>
> Note that QEMU has a "buffered" utf8 decoder in util/unicode.c, but
> it is not a good fit for byte-per-byte decoding.
>
> Signed-off-by: Marc-André Lureau
> ---
> ui/cp437.h | 13
> ui/console-vc.c | 61 +
> ui/cp437.c | 205
>
> ui/meson.build | 2 +-
> 4 files changed, 280 insertions(+), 1 deletion(-)
>
> diff --git a/ui/cp437.h b/ui/cp437.h
> new file mode 100644
> index 000..81ace8317c7
> --- /dev/null
> +++ b/ui/cp437.h
> @@ -0,0 +1,13 @@
> +/*
> + * SPDX-License-Identifier: GPL-2.0-or-later
> + *
> + * Copyright (c) QEMU contributors
> + */
> +#ifndef QEMU_CP437_H
> +#define QEMU_CP437_H
> +
> +#include
> +
> +int unicode_to_cp437(uint32_t codepoint);
> +
> +#endif /* QEMU_CP437_H */
> diff --git a/ui/console-vc.c b/ui/console-vc.c
> index 8dee1f9bd01..d9d6966410a 100644
> --- a/ui/console-vc.c
> +++ b/ui/console-vc.c
> @@ -9,6 +9,7 @@
> #include "qemu/fifo8.h"
> #include "qemu/option.h"
> #include "ui/console.h"
> +#include "ui/cp437.h"
>
> #include "trace.h"
> #include "console-priv.h"
> @@ -89,6 +90,8 @@ struct VCChardev {
> enum TTYState state;
> int esc_params[MAX_ESC_PARAMS];
> int nb_esc_params;
> +uint32_t utf8_state; /* UTF-8 DFA decoder state */
> +uint32_t utf8_codepoint; /* accumulated UTF-8 code point */
> TextAttributes t_attrib; /* currently active text attributes */
> TextAttributes t_attrib_saved;
> int x_saved, y_saved;
> @@ -598,6 +601,46 @@ static void vc_clear_xy(VCChardev *vc, int x, int y)
> vc_update_xy(vc, x, y);
> }
>
> +/*
> + * UTF-8 DFA decoder by Bjoern Hoehrmann.
> + * Copyright (c) 2008-2010 Bjoern Hoehrmann
> + * See https://github.com/polijan/utf8_decode for details.
> + *
> + * SPDX-License-Identifier: MIT
> + */
> +#define BH_UTF8_ACCEPT 0
> +#define BH_UTF8_REJECT 12
> +
> +static uint32_t bh_utf8_decode(uint32_t *state, uint32_t *codep, uint32_t
> byte)
> +{
> +static const uint8_t utf8d[] = {
> +/* character class lookup */
> +0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
> +0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
> +0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
> +0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
> +1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, 9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,
> +7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7, 7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,
> +8,8,2,2,2,2,2,2,2,2,2,2,2,2,2,2, 2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,
> +10,3,3,3,3,3,3,3,3,3,3,3,3,4,3,3, 11,6,6,6,5,8,8,8,8,8,8,8,8,8,8,8,
> +
> +/* state transition lookup */
> +0,12,24,36,60,96,84,12,12,12,48,72,
> 12,12,12,12,12,12,12,12,12,12,12,12,
> +12, 0,12,12,12,12,12, 0,12, 0,12,12,
> 12,24,12,12,12,12,12,24,12,24,12,12,
> +12,12,12,12,12,12,12,24,12,12,12,12,
> 12,24,12,12,12,12,12,12,12,24,12,12,
> +12,12,12,12,12,12,12,36,12,36,12,12,
> 12,36,12,12,12,12,12,36,12,36,12,12,
> +12,36,12,12,12,12,12,12,12,12,12,12,
> +};
> +uint32_t type = utf8d[byte];
> +
> +*codep = (*state != BH_UTF8_ACCEPT) ?
> +(byte & 0x3fu) | (*codep << 6) :
> +(0xffu >> type) & (byte);
> +
> +*state = utf8d[256 + *state + type];
> +return *state;
> +}
> +
> static void vc_put_one(VCChardev *vc, int ch)
> {
> QemuTextConsole *s = vc->console;
> @@ -761,6 +804,24 @@ static void vc_putchar(VCChardev *vc, int ch)
>
> switch(vc->state)
