Re: [luatex] UTF-8 byte sequence 0xEF, 0xBF, 0xBD causes invalid sequence error whereas ^^^^fffd does not

[Vítek Novotný]

On Sat, Jan 14, 2023 at 10:50:21AM +0100, luigi scarso wrote:
> On Sat, 14 Jan 2023 at 10:27, luigi scarso <luigi.sca...@gmail.com> wrote:
> 
> >
> >
> > On Fri, 13 Jan 2023 at 22:36, Vítek Novotný <wit...@mail.muni.cz> wrote:
> >
> >> Dear LuaTeX developers,
> >>
> >> assume the following plain TeX document `example.tex`:
> >>
> >>     \newwrite\outfile
> >>     \openout\outfile\jobname.out
> >>     \write\outfile{^^^^fffd}
> >>     \closeout\outfile
> >>     \bye
> >>
> >> Running `luatex example` will correctly produce file `example.out` with
> >> the
> >> UTF-8 encoding of U+FFFD: 0xEF, 0xBF, and 0xBD.
> >>
> >>     $ hexdump -C
> >>     00000000  ef bf bd 0a                                       |....|
> >>     00000004
> >>
> >> Now, let's change `example.tex` as follows:
> >>
> >>     \input\jobname.out
> >>     \bye
> >>
> >> Running `luatex example` produces the following error:
> >>
> >>     ! String contains an invalid utf-8 sequence.
> >>
> >> I would expect that LuaTeX would treat ^^^^fffd and the byte sequence
> >> 0xEF,
> >> 0xBF, and 0xBD the same. This issue was co-discovered by @lostenderman at
> >> <https://github.com/lostenderman/markdown/issues/34>.
> >>
> >>
> > hm, checking it now.
> >
> >
> hm I am not able to reproduce the error... My log says,
> Missing character: There is no � (U+FFFD) in font cmr10!
> but luatex  exits fine
> 
>  $ luatex --credits
> This is LuaTeX, Version 1.15.1 (TeX Live 2023/dev)
>
> The LuaTeX team is Hans Hagen, Hartmut Henkel, Taco Hoekwater, Luigi Scarso.
> 
> LuaTeX merges and builds upon (parts of) the code from these projects:
> 
> tex       : Donald Knuth
> etex      : Peter Breitenlohner, Phil Taylor and friends
> omega     : John Plaice and Yannis Haralambous
> aleph     : Giuseppe Bilotta
> pdftex    : Han The Thanh and friends
> kpathsea  : Karl Berry, Olaf Weber and others
> lua       : Roberto Ierusalimschy, Waldemar Celes and Luiz Henrique de
> Figueiredo
> metapost  : John Hobby, Taco Hoekwater, Luigi Scarso, Hans Hagen and friends
> pplib     : Paweł Jackowski
> fontforge : George Williams (partial)
> luajit    : Mike Pall (used in LuajitTeX)
> 
> Compiled with libpng 1.6.39; using 1.6.39
> Compiled with lua version 5.3.6
> Compiled with mplib version 2.02
> Compiled with zlib 1.2.13; using 1.2.13
> 
> Development id: 7554

Here is mine:

                # luatex --credits
                This is LuaTeX, Version 1.15.0 (TeX Live 2022)

                The LuaTeX team is Hans Hagen, Hartmut Henkel, Taco Hoekwater, 
Luigi Scarso.

                LuaTeX merges and builds upon (parts of) the code from these 
projects:

                tex       : Donald Knuth
                etex      : Peter Breitenlohner, Phil Taylor and friends
                omega     : John Plaice and Yannis Haralambous
                aleph     : Giuseppe Bilotta
                pdftex    : Han The Thanh and friends
                kpathsea  : Karl Berry, Olaf Weber and others
                lua       : Roberto Ierusalimschy, Waldemar Celes and Luiz 
Henrique de Figueiredo
                metapost  : John Hobby, Taco Hoekwater, Luigi Scarso, Hans 
Hagen and friends
                pplib     : Paweł Jackowski
                fontforge : George Williams (partial)
                luajit    : Mike Pall (used in LuajitTeX)

                Compiled with libpng 1.6.37; using 1.6.37
                Compiled with lua version 5.3.6
                Compiled with mplib version 2.02
                Compiled with zlib 1.2.11; using 1.2.11

                Development id: 7509

Here is the full sequence of commands to reproduce the issue:

    $ docker run --rm -it texlive/texlive

    # cat > example.tex << EOF
    > \newwrite\outfile
    > \openout\outfile\jobname.out
    > \write\outfile{^^^^fffd}
    > \closeout\outfile
    > \bye
    > EOF

    # luatex example
    This is LuaTeX, Version 1.15.0 (TeX Live 2022)
     restricted system commands enabled.
    (./example.tex 
[1{/usr/local/texlive/2022/texmf-var/fonts/map/pdftex/updmap/pdf
    
tex.map}])</usr/local/texlive/2022/texmf-dist/fonts/type1/public/amsfonts/cm/cm
    r10.pfb>
    Output written on example.pdf (1 page, 8143 bytes).
    Transcript written on example.log.

    # cat > example.tex << EOF
    > \input\jobname.out
    > \bye
    > EOF

    # luatex example
    This is LuaTeX, Version 1.15.0 (TeX Live 2022)
     restricted system commands enabled.
    (./example.tex (./example.out
    ! String contains an invalid utf-8 sequence.
    l.1
      ?
    ? q
    OK, entering \batchmode

I don't have the opportunity to test the latest LuaTeX but if I
understand you correctly, it seems that this issue has been fixed
sumetime between Development id 7509 and 7554 and will no longer be
present in TeX Live 2023.

Best,
Vit

signature.asc
Description: PGP signature