[FYI] use encoding 'non-utf8-encoding'; use CGI;
I am currently writing yet another CGI book. That is for the Japanese
market and written in Japanese. So it is inevitable that you have to
face the labyrinth of character encoding.
Before perl 5.8.0, most book teaches how to handle Japanese in CGI goes
as follows;
* stick with EUC-JP. it does not poison perl like Shift_JIS.
* use jcode.pl or Jcode.pm when you have to convert encoding.
* you can use jcode::tr or Jcode->tr when you have to convert between
Hiragana and Katakana
fine, so far. But
* totally forget regex unless you are happy with a very
counter-intuitive measure illustrated in 6.18 of the Cookbook
* if you are desperate in Kanji regex, use jperl instead.
That has now changed with 'use encoding'. But when it comes to CGI,
'use encoding' alone will not cut it. But CGI.pm can handle
multipart/form-data . Together you can use regex safely and
intuitively without resorting to convert your CGI script to UTF-8.
The 120-line script right after my signature illustrates that. Sorry,
it contains some Japanese (or my point gets blurred).
As you see, tr/// is not subject to the magic of 'use encoding'. jhi,
have we made it so deliberately ? I am begging to think tr/// is
happier to enbrace the power thereof.
Still, it can be overcome by simple eval qq{} as illustrated. This
much idiom would not hurt much, at least not as much as the Cookbook
sample
Dan the Transcoded Man
#!/usr/local/bin/perl
#
# Save me in EUC-JP!
use 5.008;
use strict;
use CGI;
use CGI::Carp qw(fatalsToBrowser);
our $Method = 'POST';
#our $Method = 'GET';
our $Enctype = 'multipart/form-data';
#our $Enctype = 'application/x-www-form-urlencoded';
our $Charset = 'euc-jp';
use encoding 'euc-jp';
my $cgi = CGI->new();
my %Label =
(
name=> '名前',
kana=> 'フリガナ',
mailto => '電子メール',
mailto2 => '電子メール(確認)',
tel => '電話',
fax => 'ファックス',
zip => '〒',
address => '住所',
comment => 'ご意見',
);
unless ($cgi->param()){
print_input($cgi);
}else{
my $kana = $cgi->param('kana');
$kana =~ s/[¥s ]+//g; # beware of zenkaku space!
eval qq{ ¥$kana =~ tr/ぁ-ん/ァ-ン/ };
# $kana =~ tr/ぁ-ん/ァ-ン/; # will not work but do you
know why?
$cgi->param(kana => $kana);
print_output($cgi);
}
sub print_input{
my $c = shift;
print_html(
$c,
title =>"Form:入力",
name=> $c->textfield(-name => 'name'),
kana=> $c->textfield(-name => 'kana'),
mailto => $c->textfield(-name => 'mailto'),
mailto2 => $c->textfield(-name => 'mailto2'),
tel => $c->textfield(-name => 'tel'),
fax => $c->textfield(-name => 'fax'),
zip => $c->textfield(-name => 'zip'),
address => $c->textfield(-name => 'address'),
comment => $c->textarea(-name => 'comment'),
);
}
sub print_output{
my $c = shift;
print_html(
$c,
title => "Form:出力",
name=> $c->param('name'),
kana=> $c->param('kana'),
mailto => $c->param('mailto'),
mailto2 => $c->param('mailto2'),
tel => $c->param('tel'),
fax => $c->param('fax'),
zip => $c->param('zip'),
address => $c->param('address'),
comment => $c->param('comment'),
);
};
sub print_html{
my $c = shift;
my %arg = @_;
print
$c->header(-charset => $Charset),
$c->start_html(-title => $arg{title}),
$c->h1($arg{title});
$c->param() or print
$c->start_form(-method => $Method, -enctype => $Enctype);
print
$c->start_table({border => 1}),
$c->Tr([
$c->td([ $Label{name}=> $arg{name} ]),
$c->td([ $Label{kana}=> $arg{kana} ]),
$c->td([ $Label{mailto} => $arg{mailto} ]),
$c->td([ $Label{mailto2} => $arg{mailto2} ]),
$c->td([ $Label{tel} => $arg{tel} ]),
$c->td([ $Label{fax} => $arg{fax} ]),
$c->td([ $Label{zip} => $arg{zip} ]),
$c->td([ $Label{address} => $arg{address} ]),
$c->td([ $Label{comment} => $arg{comment} ]),
]);
if ($c->param()){
print
$c->td($c->a({href=>$ENV{SCRIPT_TEXT}}, "Retry"));
}else{
print
$c->td([$c->reset(), $c->submit()]),
};
print $c->end_form() unless $c->param();
print
$c->end_table(),
$c->end_html();
}
__END__
Re: [FYI] use encoding 'non-utf8-encoding'; use CGI;
> As you see, tr/// is not subject to the magic of 'use encoding'.
> jhi, have we made it so deliberately ? I am begging to think tr///
Not deliberately, no. I agree that making tr/// to understand
'use encoding' would be good.
> is happier to enbrace the power thereof.
>
> Still, it can be overcome by simple eval qq{} as illustrated. This
> much idiom would not hurt much, at least not as much as the Cookbook
> sample
--
Jarkko Hietaniemi <[EMAIL PROTECTED]> http://www.iki.fi/jhi/ "There is this special
biologist word we use for 'stable'. It is 'dead'." -- Jack Cohen
Re: [FYI] use encoding 'non-utf8-encoding'; use CGI;
However, I will need to stare at your example some more, since
for simpler cases I think tr/// *is* obeying the 'use encoding':
use encoding 'greek';
($a = "\x{3af}bc\x{3af}de") =~ tr/\xdf/a/;
print $a, "\n";
This does print "abcade\n", and it also works when I replace the \xdf
with the literal \xdf.
--
Jarkko Hietaniemi <[EMAIL PROTECTED]> http://www.iki.fi/jhi/ "There is this special
biologist word we use for 'stable'. It is 'dead'." -- Jack Cohen
Re: [FYI] use encoding 'non-utf8-encoding'; use CGI;
(Not that I understand any Japanese but) could you resend your script as an attachment? I'm afraid it might get mangled otherwise. In the headers I see the following: Content-Type: text/plain; charset=ISO-2022-JP; format=flowed ... Content-Transfer-Encoding: 7bit and when I save the message from mutt, I do not see any eight-bit characters in the saved file... -- Jarkko Hietaniemi <[EMAIL PROTECTED]> http://www.iki.fi/jhi/ "There is this special biologist word we use for 'stable'. It is 'dead'." -- Jack Cohen
Re: [FYI] use encoding 'non-utf8-encoding'; use CGI;
(Hi, it's me again...) Are you doing character ranges in the tr/// under 'use encoding'? (I'm asking because I see a "-" in the middle of what I assume is mangled EUC-JP) -- Jarkko Hietaniemi <[EMAIL PROTECTED]> http://www.iki.fi/jhi/ "There is this special biologist word we use for 'stable'. It is 'dead'." -- Jack Cohen
Re: [FYI] use encoding 'non-utf8-encoding'; use CGI;
On Wednesday, Oct 2, 2002, at 22:15 Asia/Tokyo, Jarkko Hietaniemi wrote: > (Hi, it's me again...) > > Are you doing character ranges in the tr/// under 'use encoding'? > (I'm asking because I see a "-" in the middle of what I assume is > mangled EUC-JP) Yes. that's where hiragana -> katakana conversion is attempted; English equivalent of tr/A-Z/a-z/. Dan
Re: [FYI] use encoding 'non-utf8-encoding'; use CGI;
On Wednesday, Oct 2, 2002, at 21:51 Asia/Tokyo, Jarkko Hietaniemi wrote:
> However, I will need to stare at your example some more, since
> for simpler cases I think tr/// *is* obeying the 'use encoding':
>
> use encoding 'greek';
> ($a = "\x{3af}bc\x{3af}de") =~ tr/\xdf/a/;
> print $a, "\n";
>
> This does print "abcade\n", and it also works when I replace the \xdf
> with the literal \xdf.
I can explain that. "\x{3af}bc\x{3af}de" is is a string literal so it
gets encoded. however, my example in escaped form is;
$kana =~ tr/\xA4\xA1-\xA4\xF3/\xA5\xA1-\xA5\xF3/
which does not get encoded. the intention was;
$kana =~ tr/\x{3041}-\x{3093}/\x{30a1}-\x{30f3}/
That's why
eval qq{ $kana =~ tr/\xA4\xA1-\xA4\xF3/\xA5\xA1-\xA5\xF3/ }
works because \xA4\xA1-\xA4\xF3 and \xA5\xA1-\xA5\xF3 are converted. to
\x{3041}-\x{3093} and \x{30a1}-\x{30f3}, respectively.
Dan
Re: [FYI] use encoding 'non-utf8-encoding'; use CGI;
> >Are you doing character ranges in the tr/// under 'use encoding'?
> >(I'm asking because I see a "-" in the middle of what I assume is
> >mangled EUC-JP)
>
> Yes. that's where hiragana -> katakana conversion is attempted;
> English equivalent of tr/A-Z/a-z/.
Okay... What are the {begin,end} codepoints of those ranges,
both LHS and RHS of tr, both in EUC-JP and in Unicode?
--
Jarkko Hietaniemi <[EMAIL PROTECTED]> http://www.iki.fi/jhi/ "There is this special
biologist word we use for 'stable'. It is 'dead'." -- Jack Cohen
Re[4]: Encode::compat 0.01 says "Unsupported conversion"
Hi,
On Tue, 1 Oct 2002, 10:07 GMT+08 (04:07 local time) Autrijus Tang
wrote:
> On Wed, Sep 25, 2002 at 07:44:47PM +0200, Robert Allerstorfer wrote:
>> my ($from, $to) = map { s/^utf8$/utf-8/i; lc($_) } ($_[1], $_[2]);
>> But this fails due to the attempt to change $_[1]. I fixed this by
>> replacing this line by
>>
>> my ($from, $to) = @_[1, 2];
>> ($from, $to) = map {
>> s/^utf8$/utf-8/i;
>> s/^shiftjis$/shift_jis/i;
>> lc;
>> } ($from, $to);
> Thanks for your input. I decide to bundle the Alias.pm with Encode::compat
> to fix this problem more cleanly.
cool, but this produces encoding names that may not be supported by
iconv that is called by the Iconv module on some platforms. For
example, if the Perl code is
Encode::from_to($text, "sjis", "utf8");
Encode::compat would send "sjis" to Encode::Alias that comes back
with "shiftjis". On Windows, this results in an error due to an
Unsupported conversion because Iconv.dll does not know "shiftjis", but
it does know "sjis". "sjis" did work fine on several platforms that I
have tested so far.
Similary, I have found some limitations on certain platforms:
s/^utf8$/utf-8/i unless $^O eq "hpux";
# at least Win32 requires UTF-8 to be called 'utf-8'
# On HP-UX, the encoding name for UTF-8 must be exactly 'utf8'
(lowercase)
s/^utf-8$/UTF-8/i if $^O eq "solaris";
# On SunOS, the encoding name for UTF-8 must be exactly 'UTF-8'
(UPPERcase)
s/^ucs-2$/ucs2/i if $^O eq "hpux";
# On HP-UX, the encoding name for UCS-2 must be exactly 'ucs2'
(lowercase)
s/^ucs-2$/UCS-2/i if $^O eq "solaris";
# On SunOS, the encoding name for UCS-2 must be exactly 'UCS-2'
(UPPERcase)
s/^shiftjis$/sjis/i;
s/^sjis$/SJIS/i if $^O eq "solaris";
# On SunOS, the encoding name for Shift_JIS must be exactly 'SJIS'
(UPPERcase)
lc unless $^O eq "solaris";
I used this code on Encode::compat 0.02 and will now have to fit it to
0.04. You have announced that you are planning to use Unicode::MapUTF8
instead of Text::Iconv in a future version. Will this add more
platform independency?
best,
rob.
Re: [FYI] use encoding 'non-utf8-encoding'; use CGI;
> I can explain that. "\x{3af}bc\x{3af}de" is is a string literal so
> it gets encoded. however, my example in escaped form is;
>
> $kana =~ tr/\xA4\xA1-\xA4\xF3/\xA5\xA1-\xA5\xF3/
>
> which does not get encoded. the intention was;
>
> $kana =~ tr/\x{3041}-\x{3093}/\x{30a1}-\x{30f3}/
>
> That's why
>
> eval qq{ $kana =~ tr/\xA4\xA1-\xA4\xF3/\xA5\xA1-\xA5\xF3/ }
>
> works because \xA4\xA1-\xA4\xF3 and \xA5\xA1-\xA5\xF3 are converted.
> to \x{3041}-\x{3093} and \x{30a1}-\x{30f3}, respectively.
I'm confused. Firstly, the tr/\xA4... converts bytes thusly:
A1 -> A1
A2 -> A2
A3 -> A3
A4 -> A5
A5 -> A5
F3 -> A5
So why isn't it just tr/\xA4\xF3/\xA5/?
Secondly, aren't you expecting tr/// to magically recognize that when
the EUC-JP codes \xA4, \xA1 to \xA4, and \xF3 are converted to their
Unicode counterparts they are supposed to spell out the Hiragana range?
The "range" concept of tr/// is very limited. I think you want s///e.
--
Jarkko Hietaniemi <[EMAIL PROTECTED]> http://www.iki.fi/jhi/ "There is this special
biologist word we use for 'stable'. It is 'dead'." -- Jack Cohen
Re: [FYI] use encoding 'non-utf8-encoding'; use CGI;
On Wednesday, Oct 2, 2002, at 22:34 Asia/Tokyo, Jarkko Hietaniemi wrote:
>> Yes. that's where hiragana -> katakana conversion is attempted;
>> English equivalent of tr/A-Z/a-z/.
>
> Okay... What are the {begin,end} codepoints of those ranges,
> both LHS and RHS of tr, both in EUC-JP and in Unicode?
Both. I think the operation needed is straight-forward. When you get
tr[LHS][RHS], decode'em then
feed it to the naked tr// .
Dan
Re: [FYI] use encoding 'non-utf8-encoding'; use CGI;
On Wed, Oct 02, 2002 at 10:44:06PM +0900, Dan Kogai wrote:
> On Wednesday, Oct 2, 2002, at 22:34 Asia/Tokyo, Jarkko Hietaniemi wrote:
> >>Yes. that's where hiragana -> katakana conversion is attempted;
> >>English equivalent of tr/A-Z/a-z/.
> >
> >Okay... What are the {begin,end} codepoints of those ranges,
> >both LHS and RHS of tr, both in EUC-JP and in Unicode?
>
> Both. I think the operation needed is straight-forward. When you get
> tr[LHS][RHS], decode'em then
> feed it to the naked tr// .
Urk... That means a dip into the toke.c, how the tr/// ranges are
implemented is... tricky. sv_recode_to_utf8() is needed somewhere...
but I'm a little bit pressed for time right now. I suggest you
perlbug this and move the process to perl5-porters. (Inaba Hiroto
also might have insight on this; he's the tr///-with-Unicode sensei,
really-- he practically implemented all of it. And he might read
*[gk]ana much better than me :-)
> Dan
>
>
--
Jarkko Hietaniemi <[EMAIL PROTECTED]> http://www.iki.fi/jhi/ "There is this special
biologist word we use for 'stable'. It is 'dead'." -- Jack Cohen
Re: Parsing JIS X 0208 & Shift JIS with 5.8.0 +++++Success
I'm cross posting this to the perl unicode list because the pods say they might be interested in my dopey luser feedback, well actually not with those words they don't :-).
The process by which I arrived at the solution might seem painful to some, but I'm listing it here in case anyone else is/will be facing the same problem:
which is
On Tuesday, October 1, 2002, at 11:50 PM, Robin wrote:
*parse a collection of ASCII docs mixed in with docs in iso-2022-jp, shiftjis and possibly 7bit-jis, (by which I mean each doc could be 1 of three encodings, not 1 doc a mixture of all three).
*parse for tokens (Kanji charcters - ie neither Hiragana or Katakana)
*do regex substitutions accordingly
On Wednesday, October 2, 2002, at 02:14 PM, Joel Rees wrote:
You probably want these:
toshi (year) is 0x472f (JIS) and 0x944e (shift).
tsuki (month) is 0x376e (JIS) and 0x8c8e (shift).
nichi (day) is 0x467c (JIS) and 0x93fa (shift).
Thanks Joel (for all your imput), that is exactly what I needed - the character codes of the kanji I'm testing for
===from the perluniintro pod===
See "Further Resources" for how to find all these numeric codes.
===from the perluniintro pod===
The "Further Resources" mentioned in the pod, lists the vast unicode website. Once there I can't find kanji related documents, which is logical due to their Chinese origins but doesn't seem to faciliate my search as I have no idea what Chinese characters belong together as a group - after wading through various pdfs the only listing I can find which features any (one actually) of the kanji I'm intending to use as a token is in U3200.pdf and has character codes 32C1 - 32CB (namely character =[NUMBER + tsuki] ). All I want (from this group) is the character code for tsuki . Of course it's on the site somewhere, but not where I expect it to be (ie in a section called Kanji, but that's my problem not the unicode consortium's).
Time for hubris and laziness, I know which kanji I want to test for, so why not get these codes programatically using ord(), the way I would with ASCII:
===from the perluniintro pod===
At run-time you can use "chr()":
my $hebrew_alef = chr(0x05d0);
Naturally, "ord()" will do the reverse: it turns a character into a code
point.
===from the perluniintro pod===
use Encode::jp;
print ord('月'); #tsuki
which outputs: 140 (aka \X8C)
Ok, obviously ord() is assuming it's testing ASCII and returning the value for the first bite of a multi byte character encoding, ergo I'm missing something vital about how encoding works.
A while back, when I was researching how to approach dealing with Japanese text, Sadahiro Tomoyuki (owner of http://homepage1.nifty.com/nomenclator/perl/indexE.htm) (arigatou gosai masu Sadahiro -san) kindly wrote and told me how it was effectively done in the past by Japanese perlers:
(1) conversion of input in Shift_JIS to EUC_JP
(2) processing (in EUC_JP)
(3) conversion of the result (in EUC_JP) to Shift_JIS
(4) output
the same method is used by perl 5.8.0 except unicode (UTF8) is used as the internal perl processing form instead of EUC_JP
Dan Kogai wrote:
use Encode qw/encode decode/;
#...
my $utf8 = decode('shift-jis', $string);
use strict;
use diagnostics-verbose;
use Encode qw/encode decode/;
my ($data)='月';
my $utf8 = decode('shift-jis', $data);
print ord($utf8);
print chr($utf8);
yielded:
26376
Argument "\x{6708}" isn't numeric in chr at /Users/robin/Desktop/test jp.pl line 13 (#1)
Ok so it was a partial success, but I grasp what I'm doing now. Thanks to everyone that took the time to reply.
