Multibyte string in Windows command line
Hi all,
My perl6 runs an executable and prints its output. This output is printed
as multibyte string. I assume the executable gives back a multibyte string
and perl6 interpret its as one byte string for whatever reasons.
I tried Run with different encodings like windows-1251, utf16 and so on. No
luck.
The string I get looks like that:
S e t t i n g u p f o l d e r C : \ U s e r s \ w o l f
Hex:
53 00 65 00 74 00 74 00 69 00 6E 00 67 00 20 00 75 00 70 00 20 00 66 00 6F
00 6C 00 64 00 65 00 72 00 20 00 43 00 3A 00 5C 00 55 00 73 00 65 00
Here my program:
my $installer = run $install-file, :out;
for $installer.out.lines -> $line {
say $line;
}
Any idea whats wrong here. A WIndows issue or do I something wrong?
Thanks for helping,
Wolfgang
Re: Multibyte string in Windows command line
Hi Wolfgang,
that is an Windows issue. You can see that if you pipe the output of
your program in a textfile (from within the command interpreter: cmd.exe):
c:\> install.exe > x.txt
and the 'hex' your x.txt..
A.
On 13.09.18 14:21, WFB wrote:
> Hi all,
>
> My perl6 runs an executable and prints its output. This output is printed
> as multibyte string. I assume the executable gives back a multibyte string
> and perl6 interpret its as one byte string for whatever reasons.
> I tried Run with different encodings like windows-1251, utf16 and so on. No
> luck.
>
> The string I get looks like that:
> S e t t i n g u p f o l d e r C : \ U s e r s \ w o l f
> Hex:
> 53 00 65 00 74 00 74 00 69 00 6E 00 67 00 20 00 75 00 70 00 20 00 66 00 6F
> 00 6C 00 64 00 65 00 72 00 20 00 43 00 3A 00 5C 00 55 00 73 00 65 00
>
> Here my program:
>
> my $installer = run $install-file, :out;
> for $installer.out.lines -> $line {
> say $line;
> }
>
> Any idea whats wrong here. A WIndows issue or do I something wrong?
>
> Thanks for helping,
> Wolfgang
--
Andreas Müller - Raum: 35/114b - Tel: 2875
Re: Multibyte string in Windows command line
Thanks!
Has nothing to do with perl6. Unfortunately, I have no idea how I can "fix"
that.
At least I can Windows blame again ^^
Best,
Wolfgang
On Thu, 13 Sep 2018 at 15:21,
wrote:
> Hi Wolfgang,
>
> that is an Windows issue. You can see that if you pipe the output of
> your program in a textfile (from within the command interpreter:
> cmd.exe):
>
> c:\> install.exe > x.txt
>
> and the 'hex' your x.txt..
>
> A.
>
>
> On 13.09.18 14:21, WFB wrote:
> > Hi all,
> >
> > My perl6 runs an executable and prints its output. This output is printed
> > as multibyte string. I assume the executable gives back a multibyte
> string
> > and perl6 interpret its as one byte string for whatever reasons.
> > I tried Run with different encodings like windows-1251, utf16 and so on.
> No
> > luck.
> >
> > The string I get looks like that:
> > S e t t i n g u p f o l d e r C : \ U s e r s \ w o l f
> > Hex:
> > 53 00 65 00 74 00 74 00 69 00 6E 00 67 00 20 00 75 00 70 00 20 00 66 00
> 6F
> > 00 6C 00 64 00 65 00 72 00 20 00 43 00 3A 00 5C 00 55 00 73 00 65 00
> >
> > Here my program:
> >
> > my $installer = run $install-file, :out;
> > for $installer.out.lines -> $line {
> > say $line;
> > }
> >
> > Any idea whats wrong here. A WIndows issue or do I something wrong?
> >
> > Thanks for helping,
> > Wolfgang
>
> --
> Andreas Müller - Raum: 35/114b - Tel: 2875
>
Re: Multibyte string in Windows command line
but you can solve that (quick and dirty) with perl6..
..delete every secound char - it is most often \0
have fun
A.
On 13.09.18 16:03, WFB wrote:
> Thanks!
>
> Has nothing to do with perl6. Unfortunately, I have no idea how I can "fix"
> that.
> At least I can Windows blame again ^^
>
> Best,
> Wolfgang
>
> On Thu, 13 Sep 2018 at 15:21,
> wrote:
>
> > Hi Wolfgang,
> >
> > that is an Windows issue. You can see that if you pipe the output of
> > your program in a textfile (from within the command interpreter:
> > cmd.exe):
> >
> > c:\> install.exe > x.txt
> >
> > and the 'hex' your x.txt..
> >
> > A.
> >
> >
> > On 13.09.18 14:21, WFB wrote:
> > > Hi all,
> > >
> > > My perl6 runs an executable and prints its output. This output is printed
> > > as multibyte string. I assume the executable gives back a multibyte
> > string
> > > and perl6 interpret its as one byte string for whatever reasons.
> > > I tried Run with different encodings like windows-1251, utf16 and so on.
> > No
> > > luck.
> > >
> > > The string I get looks like that:
> > > S e t t i n g u p f o l d e r C : \ U s e r s \ w o l f
> > > Hex:
> > > 53 00 65 00 74 00 74 00 69 00 6E 00 67 00 20 00 75 00 70 00 20 00 66 00
> > 6F
> > > 00 6C 00 64 00 65 00 72 00 20 00 43 00 3A 00 5C 00 55 00 73 00 65 00
> > >
> > > Here my program:
> > >
> > > my $installer = run $install-file, :out;
> > > for $installer.out.lines -> $line {
> > > say $line;
> > > }
> > >
> > > Any idea whats wrong here. A WIndows issue or do I something wrong?
> > >
> > > Thanks for helping,
> > > Wolfgang
> >
> > --
> > Andreas Müller - Raum: 35/114b - Tel: 2875
> >
--
Andreas Müller - Raum: 35/114b - Tel: 2875
Re: Multibyte string in Windows command line
On Thu, Sep 13, 2018 at 7:22 AM WFB wrote:
>
> Hi all,
>
> My perl6 runs an executable and prints its output. This output is printed as
> multibyte string. I assume the executable gives back a multibyte string and
> perl6 interpret its as one byte string for whatever reasons.
> I tried Run with different encodings like windows-1251, utf16 and so on. No
> luck.
>
> The string I get looks like that:
> S e t t i n g u p f o l d e r C : \ U s e r s \ w o l f
> Hex:
> 53 00 65 00 74 00 74 00 69 00 6E 00 67 00 20 00 75 00 70 00 20 00 66 00 6F 00
> 6C 00 64 00 65 00 72 00 20 00 43 00 3A 00 5C 00 55 00 73 00 65 00
>
> Here my program:
>
> my $installer = run $install-file, :out;
> for $installer.out.lines -> $line {
> say $line;
> }
>
> Any idea whats wrong here. A WIndows issue or do I something wrong?
>
Windows uses UTF16 encoding by default (which you see here), while
Perl 6 uses UTF8 by default.
You should be able to fix it by setting the encoding
(Note that it should probably be set with :encoding, but that
doesn't work currently)
run $install-file, :out, :enc
Though I currently get an error from that.
Streaming decode NYI for encoding 4
Also I'm sure I should be able to set it after that line, but it also
produces an error
> my $installer = run $install-file, :out;
> $installer.out.encoding('utf16');
Cannot assign to an immutable value
---
A work around is to use 'latin1' (almost binary) and use `.encode.decode`
my $installer = run $install-file, :out, :enc;
for $installer.out.lines.map(*.encode('latin1').decode('utf16')) -> $line {
say $line;
}
Re: Multibyte string in Windows command line
@Andreas Thanks, I was thinking to do that with regex by myself, but that
would be a ugly workaround and only as a last resort.
@Brad: Cool, thanks, I got the errors as well. But did not come up with
your solution. One question, though:
>> for $installer.out.lines.map(*.encode('latin1').decode('utf16')) ->
$line {
Why .encode? I would assume the IO::Handle is allready given back latin-1
because of the :enc<"latin-1"> in the Run command.
On Thu, 13 Sep 2018 at 18:20, Brad Gilbert wrote:
> On Thu, Sep 13, 2018 at 7:22 AM WFB wrote:
> >
> > Hi all,
> >
> > My perl6 runs an executable and prints its output. This output is
> printed as multibyte string. I assume the executable gives back a multibyte
> string and perl6 interpret its as one byte string for whatever reasons.
> > I tried Run with different encodings like windows-1251, utf16 and so on.
> No luck.
> >
> > The string I get looks like that:
> > S e t t i n g u p f o l d e r C : \ U s e r s \ w o l f
> > Hex:
> > 53 00 65 00 74 00 74 00 69 00 6E 00 67 00 20 00 75 00 70 00 20 00 66 00
> 6F 00 6C 00 64 00 65 00 72 00 20 00 43 00 3A 00 5C 00 55 00 73 00 65 00
> >
> > Here my program:
> >
> > my $installer = run $install-file, :out;
> > for $installer.out.lines -> $line {
> > say $line;
> > }
> >
> > Any idea whats wrong here. A WIndows issue or do I something wrong?
> >
>
> Windows uses UTF16 encoding by default (which you see here), while
> Perl 6 uses UTF8 by default.
>
> You should be able to fix it by setting the encoding
> (Note that it should probably be set with :encoding, but that
> doesn't work currently)
>
> run $install-file, :out, :enc
>
> Though I currently get an error from that.
>
> Streaming decode NYI for encoding 4
>
> Also I'm sure I should be able to set it after that line, but it also
> produces an error
>
> > my $installer = run $install-file, :out;
> > $installer.out.encoding('utf16');
> Cannot assign to an immutable value
>
> ---
>
> A work around is to use 'latin1' (almost binary) and use `.encode.decode`
>
> my $installer = run $install-file, :out, :enc;
> for $installer.out.lines.map(*.encode('latin1').decode('utf16')) ->
> $line {
> say $line;
> }
>
Re: Multibyte string in Windows command line
Never mind. .lines gives back Str and thus .encode is needed...
Thanks again
On Fri, 14 Sep 2018 at 06:20, WFB wrote:
> @Andreas Thanks, I was thinking to do that with regex by myself, but that
> would be a ugly workaround and only as a last resort.
>
> @Brad: Cool, thanks, I got the errors as well. But did not come up with
> your solution. One question, though:
> >> for $installer.out.lines.map(*.encode('latin1').decode('utf16')) ->
> $line {
> Why .encode? I would assume the IO::Handle is allready given back latin-1
> because of the :enc<"latin-1"> in the Run command.
>
> On Thu, 13 Sep 2018 at 18:20, Brad Gilbert wrote:
>
>> On Thu, Sep 13, 2018 at 7:22 AM WFB wrote:
>> >
>> > Hi all,
>> >
>> > My perl6 runs an executable and prints its output. This output is
>> printed as multibyte string. I assume the executable gives back a multibyte
>> string and perl6 interpret its as one byte string for whatever reasons.
>> > I tried Run with different encodings like windows-1251, utf16 and so
>> on. No luck.
>> >
>> > The string I get looks like that:
>> > S e t t i n g u p f o l d e r C : \ U s e r s \ w o l f
>> > Hex:
>> > 53 00 65 00 74 00 74 00 69 00 6E 00 67 00 20 00 75 00 70 00 20 00 66 00
>> 6F 00 6C 00 64 00 65 00 72 00 20 00 43 00 3A 00 5C 00 55 00 73 00 65 00
>> >
>> > Here my program:
>> >
>> > my $installer = run $install-file, :out;
>> > for $installer.out.lines -> $line {
>> > say $line;
>> > }
>> >
>> > Any idea whats wrong here. A WIndows issue or do I something wrong?
>> >
>>
>> Windows uses UTF16 encoding by default (which you see here), while
>> Perl 6 uses UTF8 by default.
>>
>> You should be able to fix it by setting the encoding
>> (Note that it should probably be set with :encoding, but that
>> doesn't work currently)
>>
>> run $install-file, :out, :enc
>>
>> Though I currently get an error from that.
>>
>> Streaming decode NYI for encoding 4
>>
>> Also I'm sure I should be able to set it after that line, but it also
>> produces an error
>>
>> > my $installer = run $install-file, :out;
>> > $installer.out.encoding('utf16');
>> Cannot assign to an immutable value
>>
>> ---
>>
>> A work around is to use 'latin1' (almost binary) and use `.encode.decode`
>>
>> my $installer = run $install-file, :out, :enc;
>> for $installer.out.lines.map(*.encode('latin1').decode('utf16')) ->
>> $line {
>> say $line;
>> }
>>
>
Re: Multibyte string in Windows command line
I was just informed by samcv on #perl6-dev that she got the
utf16 stream decoder implemented.
http://colabti.org/irclogger/irclogger_log/perl6-dev?date=2018-09-15#l381
It should be in the next release, so after that there won't be a reason
to do the dance I described earlier.
The following should just work in the next release.
run $install-file, :out, :enc
On Thu, Sep 13, 2018 at 11:23 PM WFB wrote:
>
> Never mind. .lines gives back Str and thus .encode is needed...
> Thanks again
>
> On Fri, 14 Sep 2018 at 06:20, WFB wrote:
>>
>> @Andreas Thanks, I was thinking to do that with regex by myself, but that
>> would be a ugly workaround and only as a last resort.
>>
>> @Brad: Cool, thanks, I got the errors as well. But did not come up with your
>> solution. One question, though:
>> >> for $installer.out.lines.map(*.encode('latin1').decode('utf16')) -> $line
>> >> {
>> Why .encode? I would assume the IO::Handle is allready given back latin-1
>> because of the :enc<"latin-1"> in the Run command.
>>
>> On Thu, 13 Sep 2018 at 18:20, Brad Gilbert wrote:
>>>
>>> On Thu, Sep 13, 2018 at 7:22 AM WFB wrote:
>>> >
>>> > Hi all,
>>> >
>>> > My perl6 runs an executable and prints its output. This output is printed
>>> > as multibyte string. I assume the executable gives back a multibyte
>>> > string and perl6 interpret its as one byte string for whatever reasons.
>>> > I tried Run with different encodings like windows-1251, utf16 and so on.
>>> > No luck.
>>> >
>>> > The string I get looks like that:
>>> > S e t t i n g u p f o l d e r C : \ U s e r s \ w o l f
>>> > Hex:
>>> > 53 00 65 00 74 00 74 00 69 00 6E 00 67 00 20 00 75 00 70 00 20 00 66 00
>>> > 6F 00 6C 00 64 00 65 00 72 00 20 00 43 00 3A 00 5C 00 55 00 73 00 65 00
>>> >
>>> > Here my program:
>>> >
>>> > my $installer = run $install-file, :out;
>>> > for $installer.out.lines -> $line {
>>> > say $line;
>>> > }
>>> >
>>> > Any idea whats wrong here. A WIndows issue or do I something wrong?
>>> >
>>>
>>> Windows uses UTF16 encoding by default (which you see here), while
>>> Perl 6 uses UTF8 by default.
>>>
>>> You should be able to fix it by setting the encoding
>>> (Note that it should probably be set with :encoding, but that
>>> doesn't work currently)
>>>
>>> run $install-file, :out, :enc
>>>
>>> Though I currently get an error from that.
>>>
>>> Streaming decode NYI for encoding 4
>>>
>>> Also I'm sure I should be able to set it after that line, but it also
>>> produces an error
>>>
>>> > my $installer = run $install-file, :out;
>>> > $installer.out.encoding('utf16');
>>> Cannot assign to an immutable value
>>>
>>> ---
>>>
>>> A work around is to use 'latin1' (almost binary) and use `.encode.decode`
>>>
>>> my $installer = run $install-file, :out, :enc;
>>> for $installer.out.lines.map(*.encode('latin1').decode('utf16')) ->
>>> $line {
>>> say $line;
>>> }
Re: Multibyte string in Windows command line
Cool, thanks!
On Sat, 15 Sep 2018 at 22:41, Brad Gilbert wrote:
> I was just informed by samcv on #perl6-dev that she got the
> utf16 stream decoder implemented.
> http://colabti.org/irclogger/irclogger_log/perl6-dev?date=2018-09-15#l381
>
> It should be in the next release, so after that there won't be a reason
> to do the dance I described earlier.
>
> The following should just work in the next release.
>
> run $install-file, :out, :enc
>
> On Thu, Sep 13, 2018 at 11:23 PM WFB wrote:
> >
> > Never mind. .lines gives back Str and thus .encode is needed...
> > Thanks again
> >
> > On Fri, 14 Sep 2018 at 06:20, WFB wrote:
> >>
> >> @Andreas Thanks, I was thinking to do that with regex by myself, but
> that would be a ugly workaround and only as a last resort.
> >>
> >> @Brad: Cool, thanks, I got the errors as well. But did not come up with
> your solution. One question, though:
> >> >> for $installer.out.lines.map(*.encode('latin1').decode('utf16')) ->
> $line {
> >> Why .encode? I would assume the IO::Handle is allready given back
> latin-1 because of the :enc<"latin-1"> in the Run command.
> >>
> >> On Thu, 13 Sep 2018 at 18:20, Brad Gilbert wrote:
> >>>
> >>> On Thu, Sep 13, 2018 at 7:22 AM WFB wrote:
> >>> >
> >>> > Hi all,
> >>> >
> >>> > My perl6 runs an executable and prints its output. This output is
> printed as multibyte string. I assume the executable gives back a multibyte
> string and perl6 interpret its as one byte string for whatever reasons.
> >>> > I tried Run with different encodings like windows-1251, utf16 and so
> on. No luck.
> >>> >
> >>> > The string I get looks like that:
> >>> > S e t t i n g u p f o l d e r C : \ U s e r s \ w o l f
> >>> > Hex:
> >>> > 53 00 65 00 74 00 74 00 69 00 6E 00 67 00 20 00 75 00 70 00 20 00 66
> 00 6F 00 6C 00 64 00 65 00 72 00 20 00 43 00 3A 00 5C 00 55 00 73 00 65 00
> >>> >
> >>> > Here my program:
> >>> >
> >>> > my $installer = run $install-file, :out;
> >>> > for $installer.out.lines -> $line {
> >>> > say $line;
> >>> > }
> >>> >
> >>> > Any idea whats wrong here. A WIndows issue or do I something wrong?
> >>> >
> >>>
> >>> Windows uses UTF16 encoding by default (which you see here), while
> >>> Perl 6 uses UTF8 by default.
> >>>
> >>> You should be able to fix it by setting the encoding
> >>> (Note that it should probably be set with :encoding, but that
> >>> doesn't work currently)
> >>>
> >>> run $install-file, :out, :enc
> >>>
> >>> Though I currently get an error from that.
> >>>
> >>> Streaming decode NYI for encoding 4
> >>>
> >>> Also I'm sure I should be able to set it after that line, but it also
> >>> produces an error
> >>>
> >>> > my $installer = run $install-file, :out;
> >>> > $installer.out.encoding('utf16');
> >>> Cannot assign to an immutable value
> >>>
> >>> ---
> >>>
> >>> A work around is to use 'latin1' (almost binary) and use
> `.encode.decode`
> >>>
> >>> my $installer = run $install-file, :out, :enc;
> >>> for $installer.out.lines.map(*.encode('latin1').decode('utf16'))
> -> $line {
> >>> say $line;
> >>> }
>
