clone 834961 -2 -3 -4 retitle -2 perl: EU::MM outputs UTF8 strings on an ASCII filehandle in non-UTF8 locales reassign -2 perl 5.22.2-3 severity -2 normal submitter -2 ! forwarded -2 https://rt.cpan.org/Public/Bug/Display.html?id=106461 retitle -3 libencode-perl: panic: sv_setpvn called with negative strlen reassign -3 libencode-perl 2.84-2 forwarded -3 https://rt.cpan.org/Public/Bug/Display.html?id=65541 submitter -3 ! severity -3 important retitle -4 perl: (Encode) panic: sv_setpvn called with negative strlen reassign -4 perl 5.22.2-3 forwarded -4 https://rt.cpan.org/Public/Bug/Display.html?id=65541 block -4 with -3 severity -4 important submitter -4 ! thanks
My current understanding of this bug is: - ExtUtils::MakeMaker has a problem in non-UTF8 locales where it can write UTF8 metadata from for instance META.yml into Makefile through an ASCII-only file handle, causing the ""\x{00c2}" does not map to ascii" warnings. This is [rt.cpan.org #106461]; cloning a separate bug for it. - Encode has an easily reproducible problem when coercing multibyte characters into a single byte file handle (or something to that effect). The IO layer buffering can get flushed in the middle of a multibyte character, corrupting the output. A test case is perl -e 'binmode(STDOUT, ":encoding(ascii)"); print(("A"x shift) . "รค\n")' 1023 which outputs \x{fffd} when it shouldn't. - More or less related to the above, Encode can also end up using uninitialized memory in similar circumstances, sometimes ending up with 'panic: sv_setpvn called with negative strlen'. This happens occasionally on current sid when building libvitacilina-perl. I'm appending valgrind output and a gdb stack trace. This together with the previous issue is [rt.cpan.org #65541], so I'm keeping them in one report for now (but cloning a separate one against perl which will only be updated after the separate package.) The release critical part of this is probably easiest solved by either building everything with UTF-8 metadata in a UTF-8 locale, or fixing the ExtUtils::MakeMaker issue. I'm not sure how many packages are affected. The upstream tickets need to be updated; I'll do that tomorrow or so unless somebody wants to beat me to it. Valgrind output with Encode-2.86 without optimization follows, fully reproducible on current sid (perl 5.22.2-3), where it is sensitive to these exact command line arguments. % LC_ALL=C valgrind perl -I. Makefile.PL INSTALLDIRS=vendor create_packlist=0 ==18958== Memcheck, a memory error detector ==18958== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. ==18958== Using Valgrind-3.12.0.SVN and LibVEX; rerun with -h for copyright info ==18958== Command: perl -I. Makefile.PL INSTALLDIRS=vendor create_packlist=0 ==18958== 'CREATE_PACKLIST' is not a known MakeMaker parameter name. Generating a Unix-style Makefile Writing Makefile for Vitacilina "\x{00c2}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187. "\x{00a1}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187. "\x{00c3}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187. "\x{00a9}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187. "\x{00c2}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187. "\x{00a1}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187. ==18958== Conditional jump or move depends on uninitialised value(s) ==18958== at 0x52E2D2: Perl_utf8n_to_uvchr (utf8.c:588) ==18958== by 0x755237B: encode_method (Encode.xs:193) ==18958== by 0x7554C42: XS_Encode__XS_encode (Encode.xs:763) ==18958== by 0x4BDAF9: Perl_pp_entersub (pp_hot.c:3272) ==18958== by 0x4B69A5: Perl_runops_standard (run.c:41) ==18958== by 0x43C212: Perl_call_sv (perl.c:2764) ==18958== by 0x8F96E36: PerlIOEncode_flush (encoding.xs:425) ==18958== by 0x53B74E: Perl_PerlIO_flush (perlio.c:1630) ==18958== by 0x53C31B: PerlIOBuf_write (perlio.c:4165) ==18958== by 0x518364: Perl_do_print (doio.c:1386) ==18958== by 0x4B8815: Perl_pp_print (pp_hot.c:856) ==18958== by 0x4B69A5: Perl_runops_standard (run.c:41) ==18958== "\x{fffd}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187. "\x{fffd}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187. "\x{00a9}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187. Writing MYMETA.yml and MYMETA.json ==18958== ==18958== HEAP SUMMARY: ==18958== in use at exit: 14,378,715 bytes in 54,070 blocks ==18958== total heap usage: 179,891 allocs, 125,821 frees, 32,136,429 bytes allocated ==18958== ==18958== LEAK SUMMARY: ==18958== definitely lost: 69,536 bytes in 29 blocks ==18958== indirectly lost: 14,244,115 bytes in 53,915 blocks ==18958== possibly lost: 51,952 bytes in 89 blocks ==18958== still reachable: 13,112 bytes in 37 blocks ==18958== of which reachable via heuristic: ==18958== newarray : 280 bytes in 8 blocks ==18958== suppressed: 0 bytes in 0 blocks ==18958== Rerun with --leak-check=full to see details of leaked memory ==18958== ==18958== For counts of detected and suppressed errors, rerun with: -v ==18958== Use --track-origins=yes to see where uninitialised values come from ==18958== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0) GDB stack trace (LC_ALL=C again) with a conditional breakpoint in Perl_sv_setpvn(), triggered when len==-1 (so just before the panic). This is not fully reproducible and usually needs a few runs inside the same GDB session before triggering. As above, it is currently sensitive to these command arguments. Starting program: /usr/bin/perl -I. Makefile.PL INSTALLDIRS=vendor create_packlist=0 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 'CREATE_PACKLIST' is not a known MakeMaker parameter name. Generating a Unix-style Makefile Writing Makefile for Vitacilina "\x{00c2}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187. "\x{00a1}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187. "\x{00c3}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187. "\x{00a9}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187. "\x{00c2}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187. "\x{00a1}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187. "\x{00e0}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187. Breakpoint 1, Perl_sv_setpvn (my_perl=0x7d3010, sv=0x122d288, ptr=0x12d5731 "\247.\001", len=18446744073709551615) at sv.c:4905 4905 in sv.c (gdb) bt #0 Perl_sv_setpvn (my_perl=0x7d3010, sv=0x122d288, ptr=0x12d5731 "\247.\001", len=18446744073709551615) at sv.c:4905 #1 0x00007ffff68b8905 in encode_method (my_perl=0x7d3010, enc=0x7ffff6ac0aa0 <ascii_encoding>, dir=0x7ffff6ac02c0 <utf8_ascii>, src=0x122d288, check=2306, offset=0x0, term=0x0, retcode=0x0, fallback_cb=0x7d3140) at Encode.xs:279 #2 0x00007ffff68bac43 in XS_Encode__XS_encode (my_perl=0x7d3010, cv=0xb2c3b0) at Encode.xs:763 #3 0x00000000004bdafa in Perl_pp_entersub (my_perl=0x7d3010) at pp_hot.c:3272 #4 0x00000000004b69a6 in Perl_runops_standard (my_perl=0x7d3010) at run.c:41 #5 0x000000000043c213 in Perl_call_sv (my_perl=0x7d3010, sv=0x12241b8, flags=<optimized out>) at perl.c:2764 #6 0x00007ffff5e7ae37 in PerlIOEncode_flush () from /usr/lib/x86_64-linux-gnu/perl/5.22/auto/PerlIO/encoding/encoding.so #7 0x000000000053b74f in Perl_PerlIO_flush (my_perl=0x7d3010, f=<optimized out>) at perlio.c:1630 #8 0x000000000053c31c in PerlIOBuf_write (my_perl=0x7d3010, f=0x7ecc70, vbuf=<optimized out>, count=1272) at perlio.c:4165 #9 0x0000000000518365 in Perl_do_print (my_perl=my_perl@entry=0x7d3010, sv=<optimized out>, fp=fp@entry=0x7ecc70) at doio.c:1386 #10 0x00000000004b8816 in Perl_pp_print (my_perl=0x7d3010) at pp_hot.c:856 #11 0x00000000004b69a6 in Perl_runops_standard (my_perl=0x7d3010) at run.c:41 #12 0x000000000044409d in S_run_body (oldscope=<optimized out>, my_perl=<optimized out>) at perl.c:2453 #13 perl_run (my_perl=0x7d3010) at perl.c:2381 #14 0x000000000041cb6b in main (argc=5, argv=0x7fffffffe8d8, env=0x7fffffffe908) at perlmain.c:116 -- Niko Tyni nt...@debian.org