Re: [Koha-zebra] Re: Unimarc, marc21, Unicode, and MARC::File::XML
On 3/20/06, Pierrick LE GALL <[EMAIL PROTECTED]> wrote: > Hello Mike, > > I'll answer to the second question, since I worked with Paul on > Perl/MySQL and UTF-8... > > On Mon, 20 Mar 2006 09:59:32 -0500 > "Mike Rylander" <[EMAIL PROTECTED]> wrote: > > > Are you using decode_utf8($mysql_string) to let Perl know that the > > database is UTF8 encoded? IIRC, MySQL doesn't know how to tell Perl > > about that, and the DBD::MySQL maintainer haven't added that > > functionality to the module yet. > > We don't use decode_utf8. Just after the database handler creation, we > force communication to be UTF-8 with "set names 'UTF8'" SQL query. As > we know our data are UTF-8 stored and we want UTF-8, all works fine. > Except that Perl doesn't know that the data is already UTF8 ... which is the problem. Perl /does/ know that the MARC data is UTF8, and it has to convert one string or the other on output. If you explicitly use binmode() to set the PerlIO state to utf8, then the MARC::Record strings, which are known good UTF8, are not transformed, but the MySQL data, of which Perl has no encoding notions, gets "transformed", and thus broken. The only consistent and correct way to deal with UTF8 data in perl is to let PerlIO handle it by marking all sources as either providing UTF8 data or not. You can do that with binmode(), open() and several other ways, including this in modern Perls ( http://search.cpan.org/~nwclark/perl-5.8.8/lib/open.pm ). Because DBD::mysql doesn't give you a way to mark its socket as UTF8, you need to be a little underhanded and tell Perl as soon as possible using decode(), or by making utf8 the default mode for all PerlIO channels. There really isn't any way around this if you want to claim real UTF8 support and be able to use components that really do support UTF8 natively, like MARC::File::XML and MARC::Record. It's unfortunate that the DBD::mysql people won't fix their module, but there really is a right way to do this, even without their help. Is there a performance penalty with decode()? Yep. Would that go away with a fix to the DBD::mysql module? Mostly, so you really need to bug them. > Bye > > -- > Pierrick LE GALL > INEO media system > -- Mike Rylander [EMAIL PROTECTED] GPLS -- PINES Development Database Developer http://open-ils.org
Re: [Koha-zebra] Re: Unimarc, marc21, Unicode, and MARC::File::XML
Hello Mike, I'll answer to the second question, since I worked with Paul on Perl/MySQL and UTF-8... On Mon, 20 Mar 2006 09:59:32 -0500 "Mike Rylander" <[EMAIL PROTECTED]> wrote: > Are you using decode_utf8($mysql_string) to let Perl know that the > database is UTF8 encoded? IIRC, MySQL doesn't know how to tell Perl > about that, and the DBD::MySQL maintainer haven't added that > functionality to the module yet. We don't use decode_utf8. Just after the database handler creation, we force communication to be UTF-8 with "set names 'UTF8'" SQL query. As we know our data are UTF-8 stored and we want UTF-8, all works fine. Bye -- Pierrick LE GALL INEO media system
Re: Unimarc, marc21, Unicode, and MARC::File::XML
Mike Rylander a écrit : On 3/20/06, Paul POULAIN <[EMAIL PROTECTED]> wrote: Mike Rylander a écrit : I tested with the record you sent Ed and me, and everything seems to work for me ... As you can see, I tested several variants of the UNIMARC flag, and even tested not sending the encoding to new_from_xml() ... it all seems to work for me, and I'm not sure what problems you're seeing. Perhaps you just needed to set your binmode for the XML source? strange, strange... What does my script : * retrieve the MARC::Record from zebra * read some datas from mysql * build a page with HTML::Template * send the pages to the browser Are you getting XML or binary MARC from zebra? XML. The test.xml I sended to you on friday comes was the $raw = $rs->record(0)->raw(); record. Are you using decode_utf8($mysql_string) to let Perl know that the database is UTF8 encoded? IIRC, MySQL doesn't know how to tell Perl about that, and the DBD::MySQL maintainer haven't added that functionality to the module yet. I thought we had to decode_utf8($mysql_string), and began to investigate a lot. But after many hours of digging & getting problems, I now have a working mySQL in utf8 for all of Koha. without any binmode of decode_utf8 ... And it seems joshua & Tümer (Turkey) has the same conclusion : no more problems with mySQL & Perl. We all use a recent version of mySQL, even if DBD::mysql maintainer (from mysql.com : joshua dropped him a mail but got no answer) did nothing on the cpan package. -- Paul POULAIN et Henri Damien LAURENT Consultants indépendants en logiciels libres et bibliothéconomie (http://www.koha-fr.org)
Re: Unimarc, marc21, Unicode, and MARC::File::XML
On 3/20/06, Paul POULAIN <[EMAIL PROTECTED]> wrote: > Mike Rylander a écrit : > > I tested with the record you sent Ed and me, and everything seems to > > work for me ... > > As you can see, I tested several variants of the UNIMARC flag, and > > even tested not sending the encoding to new_from_xml() ... it all > > seems to work for me, and I'm not sure what problems you're seeing. > > Perhaps you just needed to set your binmode for the XML source? > > strange, strange... > > What does my script : > * retrieve the MARC::Record from zebra > * read some datas from mysql > * build a page with HTML::Template > * send the pages to the browser Are you getting XML or binary MARC from zebra? > > I added 3 lines to save the record in a file after reading from zebra. > Adding binmode(F,':utf8'); > before saving my record in F, give me correct UTF-8. > without binmode, it's NOK. > > But when I put the MARC::record in a page builded with HTML::Template, > it's wrong. > The HTML is utf-8 (html page encoding). > It also contains some strings from mySQL and all strings from mySQL > appear as correct utf8 while all strings coming from the MARC::record > coming from zebra are not ! > > I can add "binmode()" to the template output, but everything goes wrong > with strings from mySQL. > Are you using decode_utf8($mysql_string) to let Perl know that the database is UTF8 encoded? IIRC, MySQL doesn't know how to tell Perl about that, and the DBD::MySQL maintainer haven't added that functionality to the module yet. > Any suggestion welcomed ! > -- > Paul POULAIN et Henri Damien LAURENT > Consultants indépendants > en logiciels libres et bibliothéconomie (http://www.koha-fr.org) > -- Mike Rylander [EMAIL PROTECTED] GPLS -- PINES Development Database Developer http://open-ils.org
Re: Unimarc, marc21, Unicode, and MARC::File::XML
Mike Rylander a écrit : I tested with the record you sent Ed and me, and everything seems to work for me ... As you can see, I tested several variants of the UNIMARC flag, and even tested not sending the encoding to new_from_xml() ... it all seems to work for me, and I'm not sure what problems you're seeing. Perhaps you just needed to set your binmode for the XML source? strange, strange... What does my script : * retrieve the MARC::Record from zebra * read some datas from mysql * build a page with HTML::Template * send the pages to the browser I added 3 lines to save the record in a file after reading from zebra. Adding binmode(F,':utf8'); before saving my record in F, give me correct UTF-8. without binmode, it's NOK. But when I put the MARC::record in a page builded with HTML::Template, it's wrong. The HTML is utf-8 (html page encoding). It also contains some strings from mySQL and all strings from mySQL appear as correct utf8 while all strings coming from the MARC::record coming from zebra are not ! I can add "binmode()" to the template output, but everything goes wrong with strings from mySQL. Any suggestion welcomed ! -- Paul POULAIN et Henri Damien LAURENT Consultants indépendants en logiciels libres et bibliothéconomie (http://www.koha-fr.org)
