Multibyte string in Windows command line
Hi all, My perl6 runs an executable and prints its output. This output is printed as multibyte string. I assume the executable gives back a multibyte string and perl6 interpret its as one byte string for whatever reasons. I tried Run with different encodings like windows-1251, utf16 and so on. No luck. The string I get looks like that: S e t t i n g u p f o l d e r C : \ U s e r s \ w o l f Hex: 53 00 65 00 74 00 74 00 69 00 6E 00 67 00 20 00 75 00 70 00 20 00 66 00 6F 00 6C 00 64 00 65 00 72 00 20 00 43 00 3A 00 5C 00 55 00 73 00 65 00 Here my program: my $installer = run $install-file, :out; for $installer.out.lines -> $line { say $line; } Any idea whats wrong here. A WIndows issue or do I something wrong? Thanks for helping, Wolfgang
Re: Multibyte string in Windows command line
Hi Wolfgang, that is an Windows issue. You can see that if you pipe the output of your program in a textfile (from within the command interpreter: cmd.exe): c:\> install.exe > x.txt and the 'hex' your x.txt.. A. On 13.09.18 14:21, WFB wrote: > Hi all, > > My perl6 runs an executable and prints its output. This output is printed > as multibyte string. I assume the executable gives back a multibyte string > and perl6 interpret its as one byte string for whatever reasons. > I tried Run with different encodings like windows-1251, utf16 and so on. No > luck. > > The string I get looks like that: > S e t t i n g u p f o l d e r C : \ U s e r s \ w o l f > Hex: > 53 00 65 00 74 00 74 00 69 00 6E 00 67 00 20 00 75 00 70 00 20 00 66 00 6F > 00 6C 00 64 00 65 00 72 00 20 00 43 00 3A 00 5C 00 55 00 73 00 65 00 > > Here my program: > > my $installer = run $install-file, :out; > for $installer.out.lines -> $line { > say $line; > } > > Any idea whats wrong here. A WIndows issue or do I something wrong? > > Thanks for helping, > Wolfgang -- Andreas Müller - Raum: 35/114b - Tel: 2875
Re: Multibyte string in Windows command line
Thanks! Has nothing to do with perl6. Unfortunately, I have no idea how I can "fix" that. At least I can Windows blame again ^^ Best, Wolfgang On Thu, 13 Sep 2018 at 15:21, wrote: > Hi Wolfgang, > > that is an Windows issue. You can see that if you pipe the output of > your program in a textfile (from within the command interpreter: > cmd.exe): > > c:\> install.exe > x.txt > > and the 'hex' your x.txt.. > > A. > > > On 13.09.18 14:21, WFB wrote: > > Hi all, > > > > My perl6 runs an executable and prints its output. This output is printed > > as multibyte string. I assume the executable gives back a multibyte > string > > and perl6 interpret its as one byte string for whatever reasons. > > I tried Run with different encodings like windows-1251, utf16 and so on. > No > > luck. > > > > The string I get looks like that: > > S e t t i n g u p f o l d e r C : \ U s e r s \ w o l f > > Hex: > > 53 00 65 00 74 00 74 00 69 00 6E 00 67 00 20 00 75 00 70 00 20 00 66 00 > 6F > > 00 6C 00 64 00 65 00 72 00 20 00 43 00 3A 00 5C 00 55 00 73 00 65 00 > > > > Here my program: > > > > my $installer = run $install-file, :out; > > for $installer.out.lines -> $line { > > say $line; > > } > > > > Any idea whats wrong here. A WIndows issue or do I something wrong? > > > > Thanks for helping, > > Wolfgang > > -- > Andreas Müller - Raum: 35/114b - Tel: 2875 >
Re: Multibyte string in Windows command line
but you can solve that (quick and dirty) with perl6.. ..delete every secound char - it is most often \0 have fun A. On 13.09.18 16:03, WFB wrote: > Thanks! > > Has nothing to do with perl6. Unfortunately, I have no idea how I can "fix" > that. > At least I can Windows blame again ^^ > > Best, > Wolfgang > > On Thu, 13 Sep 2018 at 15:21, > wrote: > > > Hi Wolfgang, > > > > that is an Windows issue. You can see that if you pipe the output of > > your program in a textfile (from within the command interpreter: > > cmd.exe): > > > > c:\> install.exe > x.txt > > > > and the 'hex' your x.txt.. > > > > A. > > > > > > On 13.09.18 14:21, WFB wrote: > > > Hi all, > > > > > > My perl6 runs an executable and prints its output. This output is printed > > > as multibyte string. I assume the executable gives back a multibyte > > string > > > and perl6 interpret its as one byte string for whatever reasons. > > > I tried Run with different encodings like windows-1251, utf16 and so on. > > No > > > luck. > > > > > > The string I get looks like that: > > > S e t t i n g u p f o l d e r C : \ U s e r s \ w o l f > > > Hex: > > > 53 00 65 00 74 00 74 00 69 00 6E 00 67 00 20 00 75 00 70 00 20 00 66 00 > > 6F > > > 00 6C 00 64 00 65 00 72 00 20 00 43 00 3A 00 5C 00 55 00 73 00 65 00 > > > > > > Here my program: > > > > > > my $installer = run $install-file, :out; > > > for $installer.out.lines -> $line { > > > say $line; > > > } > > > > > > Any idea whats wrong here. A WIndows issue or do I something wrong? > > > > > > Thanks for helping, > > > Wolfgang > > > > -- > > Andreas Müller - Raum: 35/114b - Tel: 2875 > > -- Andreas Müller - Raum: 35/114b - Tel: 2875
Re: Multibyte string in Windows command line
On Thu, Sep 13, 2018 at 7:22 AM WFB wrote: > > Hi all, > > My perl6 runs an executable and prints its output. This output is printed as > multibyte string. I assume the executable gives back a multibyte string and > perl6 interpret its as one byte string for whatever reasons. > I tried Run with different encodings like windows-1251, utf16 and so on. No > luck. > > The string I get looks like that: > S e t t i n g u p f o l d e r C : \ U s e r s \ w o l f > Hex: > 53 00 65 00 74 00 74 00 69 00 6E 00 67 00 20 00 75 00 70 00 20 00 66 00 6F 00 > 6C 00 64 00 65 00 72 00 20 00 43 00 3A 00 5C 00 55 00 73 00 65 00 > > Here my program: > > my $installer = run $install-file, :out; > for $installer.out.lines -> $line { > say $line; > } > > Any idea whats wrong here. A WIndows issue or do I something wrong? > Windows uses UTF16 encoding by default (which you see here), while Perl 6 uses UTF8 by default. You should be able to fix it by setting the encoding (Note that it should probably be set with :encoding, but that doesn't work currently) run $install-file, :out, :enc Though I currently get an error from that. Streaming decode NYI for encoding 4 Also I'm sure I should be able to set it after that line, but it also produces an error > my $installer = run $install-file, :out; > $installer.out.encoding('utf16'); Cannot assign to an immutable value --- A work around is to use 'latin1' (almost binary) and use `.encode.decode` my $installer = run $install-file, :out, :enc; for $installer.out.lines.map(*.encode('latin1').decode('utf16')) -> $line { say $line; }
Re: Multibyte string in Windows command line
@Andreas Thanks, I was thinking to do that with regex by myself, but that would be a ugly workaround and only as a last resort. @Brad: Cool, thanks, I got the errors as well. But did not come up with your solution. One question, though: >> for $installer.out.lines.map(*.encode('latin1').decode('utf16')) -> $line { Why .encode? I would assume the IO::Handle is allready given back latin-1 because of the :enc<"latin-1"> in the Run command. On Thu, 13 Sep 2018 at 18:20, Brad Gilbert wrote: > On Thu, Sep 13, 2018 at 7:22 AM WFB wrote: > > > > Hi all, > > > > My perl6 runs an executable and prints its output. This output is > printed as multibyte string. I assume the executable gives back a multibyte > string and perl6 interpret its as one byte string for whatever reasons. > > I tried Run with different encodings like windows-1251, utf16 and so on. > No luck. > > > > The string I get looks like that: > > S e t t i n g u p f o l d e r C : \ U s e r s \ w o l f > > Hex: > > 53 00 65 00 74 00 74 00 69 00 6E 00 67 00 20 00 75 00 70 00 20 00 66 00 > 6F 00 6C 00 64 00 65 00 72 00 20 00 43 00 3A 00 5C 00 55 00 73 00 65 00 > > > > Here my program: > > > > my $installer = run $install-file, :out; > > for $installer.out.lines -> $line { > > say $line; > > } > > > > Any idea whats wrong here. A WIndows issue or do I something wrong? > > > > Windows uses UTF16 encoding by default (which you see here), while > Perl 6 uses UTF8 by default. > > You should be able to fix it by setting the encoding > (Note that it should probably be set with :encoding, but that > doesn't work currently) > > run $install-file, :out, :enc > > Though I currently get an error from that. > > Streaming decode NYI for encoding 4 > > Also I'm sure I should be able to set it after that line, but it also > produces an error > > > my $installer = run $install-file, :out; > > $installer.out.encoding('utf16'); > Cannot assign to an immutable value > > --- > > A work around is to use 'latin1' (almost binary) and use `.encode.decode` > > my $installer = run $install-file, :out, :enc; > for $installer.out.lines.map(*.encode('latin1').decode('utf16')) -> > $line { > say $line; > } >
Re: Multibyte string in Windows command line
Never mind. .lines gives back Str and thus .encode is needed... Thanks again On Fri, 14 Sep 2018 at 06:20, WFB wrote: > @Andreas Thanks, I was thinking to do that with regex by myself, but that > would be a ugly workaround and only as a last resort. > > @Brad: Cool, thanks, I got the errors as well. But did not come up with > your solution. One question, though: > >> for $installer.out.lines.map(*.encode('latin1').decode('utf16')) -> > $line { > Why .encode? I would assume the IO::Handle is allready given back latin-1 > because of the :enc<"latin-1"> in the Run command. > > On Thu, 13 Sep 2018 at 18:20, Brad Gilbert wrote: > >> On Thu, Sep 13, 2018 at 7:22 AM WFB wrote: >> > >> > Hi all, >> > >> > My perl6 runs an executable and prints its output. This output is >> printed as multibyte string. I assume the executable gives back a multibyte >> string and perl6 interpret its as one byte string for whatever reasons. >> > I tried Run with different encodings like windows-1251, utf16 and so >> on. No luck. >> > >> > The string I get looks like that: >> > S e t t i n g u p f o l d e r C : \ U s e r s \ w o l f >> > Hex: >> > 53 00 65 00 74 00 74 00 69 00 6E 00 67 00 20 00 75 00 70 00 20 00 66 00 >> 6F 00 6C 00 64 00 65 00 72 00 20 00 43 00 3A 00 5C 00 55 00 73 00 65 00 >> > >> > Here my program: >> > >> > my $installer = run $install-file, :out; >> > for $installer.out.lines -> $line { >> > say $line; >> > } >> > >> > Any idea whats wrong here. A WIndows issue or do I something wrong? >> > >> >> Windows uses UTF16 encoding by default (which you see here), while >> Perl 6 uses UTF8 by default. >> >> You should be able to fix it by setting the encoding >> (Note that it should probably be set with :encoding, but that >> doesn't work currently) >> >> run $install-file, :out, :enc >> >> Though I currently get an error from that. >> >> Streaming decode NYI for encoding 4 >> >> Also I'm sure I should be able to set it after that line, but it also >> produces an error >> >> > my $installer = run $install-file, :out; >> > $installer.out.encoding('utf16'); >> Cannot assign to an immutable value >> >> --- >> >> A work around is to use 'latin1' (almost binary) and use `.encode.decode` >> >> my $installer = run $install-file, :out, :enc; >> for $installer.out.lines.map(*.encode('latin1').decode('utf16')) -> >> $line { >> say $line; >> } >> >
Re: Multibyte string in Windows command line
I was just informed by samcv on #perl6-dev that she got the utf16 stream decoder implemented. http://colabti.org/irclogger/irclogger_log/perl6-dev?date=2018-09-15#l381 It should be in the next release, so after that there won't be a reason to do the dance I described earlier. The following should just work in the next release. run $install-file, :out, :enc On Thu, Sep 13, 2018 at 11:23 PM WFB wrote: > > Never mind. .lines gives back Str and thus .encode is needed... > Thanks again > > On Fri, 14 Sep 2018 at 06:20, WFB wrote: >> >> @Andreas Thanks, I was thinking to do that with regex by myself, but that >> would be a ugly workaround and only as a last resort. >> >> @Brad: Cool, thanks, I got the errors as well. But did not come up with your >> solution. One question, though: >> >> for $installer.out.lines.map(*.encode('latin1').decode('utf16')) -> $line >> >> { >> Why .encode? I would assume the IO::Handle is allready given back latin-1 >> because of the :enc<"latin-1"> in the Run command. >> >> On Thu, 13 Sep 2018 at 18:20, Brad Gilbert wrote: >>> >>> On Thu, Sep 13, 2018 at 7:22 AM WFB wrote: >>> > >>> > Hi all, >>> > >>> > My perl6 runs an executable and prints its output. This output is printed >>> > as multibyte string. I assume the executable gives back a multibyte >>> > string and perl6 interpret its as one byte string for whatever reasons. >>> > I tried Run with different encodings like windows-1251, utf16 and so on. >>> > No luck. >>> > >>> > The string I get looks like that: >>> > S e t t i n g u p f o l d e r C : \ U s e r s \ w o l f >>> > Hex: >>> > 53 00 65 00 74 00 74 00 69 00 6E 00 67 00 20 00 75 00 70 00 20 00 66 00 >>> > 6F 00 6C 00 64 00 65 00 72 00 20 00 43 00 3A 00 5C 00 55 00 73 00 65 00 >>> > >>> > Here my program: >>> > >>> > my $installer = run $install-file, :out; >>> > for $installer.out.lines -> $line { >>> > say $line; >>> > } >>> > >>> > Any idea whats wrong here. A WIndows issue or do I something wrong? >>> > >>> >>> Windows uses UTF16 encoding by default (which you see here), while >>> Perl 6 uses UTF8 by default. >>> >>> You should be able to fix it by setting the encoding >>> (Note that it should probably be set with :encoding, but that >>> doesn't work currently) >>> >>> run $install-file, :out, :enc >>> >>> Though I currently get an error from that. >>> >>> Streaming decode NYI for encoding 4 >>> >>> Also I'm sure I should be able to set it after that line, but it also >>> produces an error >>> >>> > my $installer = run $install-file, :out; >>> > $installer.out.encoding('utf16'); >>> Cannot assign to an immutable value >>> >>> --- >>> >>> A work around is to use 'latin1' (almost binary) and use `.encode.decode` >>> >>> my $installer = run $install-file, :out, :enc; >>> for $installer.out.lines.map(*.encode('latin1').decode('utf16')) -> >>> $line { >>> say $line; >>> }
Re: Multibyte string in Windows command line
Cool, thanks! On Sat, 15 Sep 2018 at 22:41, Brad Gilbert wrote: > I was just informed by samcv on #perl6-dev that she got the > utf16 stream decoder implemented. > http://colabti.org/irclogger/irclogger_log/perl6-dev?date=2018-09-15#l381 > > It should be in the next release, so after that there won't be a reason > to do the dance I described earlier. > > The following should just work in the next release. > > run $install-file, :out, :enc > > On Thu, Sep 13, 2018 at 11:23 PM WFB wrote: > > > > Never mind. .lines gives back Str and thus .encode is needed... > > Thanks again > > > > On Fri, 14 Sep 2018 at 06:20, WFB wrote: > >> > >> @Andreas Thanks, I was thinking to do that with regex by myself, but > that would be a ugly workaround and only as a last resort. > >> > >> @Brad: Cool, thanks, I got the errors as well. But did not come up with > your solution. One question, though: > >> >> for $installer.out.lines.map(*.encode('latin1').decode('utf16')) -> > $line { > >> Why .encode? I would assume the IO::Handle is allready given back > latin-1 because of the :enc<"latin-1"> in the Run command. > >> > >> On Thu, 13 Sep 2018 at 18:20, Brad Gilbert wrote: > >>> > >>> On Thu, Sep 13, 2018 at 7:22 AM WFB wrote: > >>> > > >>> > Hi all, > >>> > > >>> > My perl6 runs an executable and prints its output. This output is > printed as multibyte string. I assume the executable gives back a multibyte > string and perl6 interpret its as one byte string for whatever reasons. > >>> > I tried Run with different encodings like windows-1251, utf16 and so > on. No luck. > >>> > > >>> > The string I get looks like that: > >>> > S e t t i n g u p f o l d e r C : \ U s e r s \ w o l f > >>> > Hex: > >>> > 53 00 65 00 74 00 74 00 69 00 6E 00 67 00 20 00 75 00 70 00 20 00 66 > 00 6F 00 6C 00 64 00 65 00 72 00 20 00 43 00 3A 00 5C 00 55 00 73 00 65 00 > >>> > > >>> > Here my program: > >>> > > >>> > my $installer = run $install-file, :out; > >>> > for $installer.out.lines -> $line { > >>> > say $line; > >>> > } > >>> > > >>> > Any idea whats wrong here. A WIndows issue or do I something wrong? > >>> > > >>> > >>> Windows uses UTF16 encoding by default (which you see here), while > >>> Perl 6 uses UTF8 by default. > >>> > >>> You should be able to fix it by setting the encoding > >>> (Note that it should probably be set with :encoding, but that > >>> doesn't work currently) > >>> > >>> run $install-file, :out, :enc > >>> > >>> Though I currently get an error from that. > >>> > >>> Streaming decode NYI for encoding 4 > >>> > >>> Also I'm sure I should be able to set it after that line, but it also > >>> produces an error > >>> > >>> > my $installer = run $install-file, :out; > >>> > $installer.out.encoding('utf16'); > >>> Cannot assign to an immutable value > >>> > >>> --- > >>> > >>> A work around is to use 'latin1' (almost binary) and use > `.encode.decode` > >>> > >>> my $installer = run $install-file, :out, :enc; > >>> for $installer.out.lines.map(*.encode('latin1').decode('utf16')) > -> $line { > >>> say $line; > >>> } >