date:20130901

[Koha] Import MARC record error: utf8 \xE0

2013-09-01 Thread Pongtawat Chippimolchai

Hello all,

I got the following error while importing MARC records into Koha 3.12.3 (on
Ubuntu 12.04 amd64 via Debian package):

utf8 \xE0 does not map to Unicode at /usr/lib/perl/5.14/Encode.pm line
174.

The record staged fine but it couldn't be imported. When clicking to view
it in staging area, the same error appear.

I checked the imported file, and it seem to look fine to me. The MARCXML in
the database look ok too. But if I save the MARC BLOB and use marcdump on
it, I got the same error.

The imported file is here:
https://www.dropbox.com/s/u25mf4dqk69wtgx/error.mrc
The saved MARC BLOB is here:
https://www.dropbox.com/s/soawpzmnuy5j7zr/import_records-marc.bin

Running perl -d commit_file.pl show something like this:

-
utf8 \xE0 does not map to Unicode at /usr/lib/perl/5.14/Encode.pm line
174.
 at /usr/lib/perl/5.14/Encode.pm line 174
   Encode::decode('UTF-8', 'ha\x{1e}0
\x{1f}a\x{e0}\x{b9}\x{84}\x{e0}\x{b8}\x{9e}\x{e0}\x{b8}\x{a8}\x{e0}\x{b8}\x{b2}\x{e0}\x{b8}\x{a5}
\x{e0}\x{b8}\x{81}\x{e0}\x{b8}\x{a4}\x{e0}\x{b8}\x{a9}\x{e0}\x{b8}\x{8e}\x{e0}\x{b8}\x{b2}\x{e0}\x{b8}\x{98}\x{e0}\x{b8}\x{b4}\x{e0}\x{b8}\x{a7}\x{e0}\x{b8}\x{b8}\x{e0}\x{b8}\x{92}\x{e0}\x{b8}',
1) called at /usr/share/perl5/MARC/File/Encode.pm line 35
   MARC::File::Encode::marc_to_utf8('ha\x{1e}0
\x{1f}a\x{e0}\x{b9}\x{84}\x{e0}\x{b8}\x{9e}\x{e0}\x{b8}\x{a8}\x{e0}\x{b8}\x{b2}\x{e0}\x{b8}\x{a5}
\x{e0}\x{b8}\x{81}\x{e0}\x{b8}\x{a4}\x{e0}\x{b8}\x{a9}\x{e0}\x{b8}\x{8e}\x{e0}\x{b8}\x{b2}\x{e0}\x{b8}\x{98}\x{e0}\x{b8}\x{b4}\x{e0}\x{b8}\x{a7}\x{e0}\x{b8}\x{b8}\x{e0}\x{b8}\x{92}\x{e0}\x{b8}')
called at /usr/share/perl5/MARC/File/USMARC.pm line 172

 
MARC::File::USMARC::decode('00932nas0a2200217000455001700080043000170200014000600...')
called at /usr/share/perl5/MARC/Record.pm line 81
   MARC::Record::new_from_usmarc('MARC::Record',
'00932nas0a2200217000455001700080043000170200014000600...') called
at /usr/share/koha/lib/C4/ImportBatch.pm line 585
   C4::ImportBatch::BatchCommitRecords(13, '', 100, 'CODE(0x5227d58)')
called at /usr/share/koha/bin/commit_file.pl line 85
   main::process_batch(13) called at /usr/share/koha/bin/commit_file.plline 57
-

I really don't get the way Perl (and Koha) handle UTF8 at all. If I print
that escaped string out, some character will be readable but the others
appear as ?. For the records that could be imported, if I print out their
data from Perl I see the same pattern (readable char mixed with ?). But
they somehow end up correct in the database.

I have no idea what's wrong here. If you have any clue, please help.

Thank you very much,
Pongtawat
___
Koha mailing list  http://koha-community.org
Koha@lists.katipo.co.nz
http://lists.katipo.co.nz/mailman/listinfo/koha

Re: [Koha] Import MARC record error: utf8 \xE0

2013-09-01 Thread Paul


At 01:17 PM 8/31/2013 +0700, Pongtawat Chippimolchai wrote:

Hello all,

I got the following error while importing MARC records into Koha 3.12.3 (on
Ubuntu 12.04 amd64 via Debian package):

utf8 \xE0 does not map to Unicode at /usr/lib/perl/5.14/Encode.pm line
174.

The record staged fine but it couldn't be imported. When clicking to view
it in staging area, the same error appear.


Perhaps a long shot, but I remember having a similar problem which turned 
out to be (not Unicode) but simply including hyphens in ISBN numbers -- you 
have put an apparent ISSN with hyphen into 020$a, and my verification 
script shows: 020 - Invalid data found: 8057-7730.  Only 
[^(97(8|9))?\d{9}(\d|X)$] is valid within this field. (I realize that MARC 
purists will object that more - e.g. (pbk) - can be added, but bulk 
imports seem to be touchy on this subject, so I added the $ at the end of 
the regex.)


Best - Paul




I checked the imported file, and it seem to look fine to me. The MARCXML in
the database look ok too. But if I save the MARC BLOB and use marcdump on
it, I got the same error.

The imported file is here:
https://www.dropbox.com/s/u25mf4dqk69wtgx/error.mrc
The saved MARC BLOB is here:
https://www.dropbox.com/s/soawpzmnuy5j7zr/import_records-marc.bin

Running perl -d commit_file.pl show something like this:

-
utf8 \xE0 does not map to Unicode at /usr/lib/perl/5.14/Encode.pm line
174.
 at /usr/lib/perl/5.14/Encode.pm line 174
   Encode::decode('UTF-8', 'ha\x{1e}0
\x{1f}a\x{e0}\x{b9}\x{84}\x{e0}\x{b8}\x{9e}\x{e0}\x{b8}\x{a8}\x{e0}\x{b8}\x{b2}\x{e0}\x{b8}\x{a5}
\x{e0}\x{b8}\x{81}\x{e0}\x{b8}\x{a4}\x{e0}\x{b8}\x{a9}\x{e0}\x{b8}\x{8e}\x{e0}\x{b8}\x{b2}\x{e0}\x{b8}\x{98}\x{e0}\x{b8}\x{b4}\x{e0}\x{b8}\x{a7}\x{e0}\x{b8}\x{b8}\x{e0}\x{b8}\x{92}\x{e0}\x{b8}',
1) called at /usr/share/perl5/MARC/File/Encode.pm line 35
   MARC::File::Encode::marc_to_utf8('ha\x{1e}0
\x{1f}a\x{e0}\x{b9}\x{84}\x{e0}\x{b8}\x{9e}\x{e0}\x{b8}\x{a8}\x{e0}\x{b8}\x{b2}\x{e0}\x{b8}\x{a5}
\x{e0}\x{b8}\x{81}\x{e0}\x{b8}\x{a4}\x{e0}\x{b8}\x{a9}\x{e0}\x{b8}\x{8e}\x{e0}\x{b8}\x{b2}\x{e0}\x{b8}\x{98}\x{e0}\x{b8}\x{b4}\x{e0}\x{b8}\x{a7}\x{e0}\x{b8}\x{b8}\x{e0}\x{b8}\x{92}\x{e0}\x{b8}')
called at /usr/share/perl5/MARC/File/USMARC.pm line 172


MARC::File::USMARC::decode('00932nas0a2200217000455001700080043000170200014000600...')
called at /usr/share/perl5/MARC/Record.pm line 81
   MARC::Record::new_from_usmarc('MARC::Record',
'00932nas0a2200217000455001700080043000170200014000600...') called
at /usr/share/koha/lib/C4/ImportBatch.pm line 585
   C4::ImportBatch::BatchCommitRecords(13, '', 100, 'CODE(0x5227d58)')
called at /usr/share/koha/bin/commit_file.pl line 85
   main::process_batch(13) called at 
/usr/share/koha/bin/commit_file.plline 57

-

I really don't get the way Perl (and Koha) handle UTF8 at all. If I print
that escaped string out, some character will be readable but the others
appear as ?. For the records that could be imported, if I print out their
data from Perl I see the same pattern (readable char mixed with ?). But
they somehow end up correct in the database.

I have no idea what's wrong here. If you have any clue, please help.

Thank you very much,
Pongtawat
___
Koha mailing list  http://koha-community.org
Koha@lists.katipo.co.nz
http://lists.katipo.co.nz/mailman/listinfo/koha


---
Maritime heritage and history, preservation and conservation,
research and education through the written word and the arts.
http://NavalMarineArchive.com and http://UltraMarine.ca

___
Koha mailing list  http://koha-community.org
Koha@lists.katipo.co.nz
http://lists.katipo.co.nz/mailman/listinfo/koha

[Koha] Thank you - Circulation Rules Help needed - 1) Hard Time Proposal - 2) Query on Overnight/Weekend Loans

2013-09-01 Thread Matthew Charlesworth, S.J.

Dear all,

I used to use an earlier version of Koha in 2007/8 but then moved out of
library work (though I worked with Chrispin Sitali in Zambia who sometimes
posts here (wave)), and have always remained a fan of Koha - but recently
after moving to Nairobi, I had the opportunity to install a Koha
installation using the more updated version (what a lot of work has gone on
behind the scenes in the interim! Well done). In the last month or two we
transitioned approx. 90,000 records from InMagic and, beginning this
academic year (August), have started using Koha as our main OPAC and staff
client. So far, we are getting very positive results - thank you for
creating such a wonderful product.

I have two questions though and I don't know if I'm just not seeing it or
if I must submit a bug or feature request? I thought I'd ask here before I
do - perhaps someone could let me know how best to proceed.

I am using Koha 3.12.03.000 on Ubuntu 12.04.3 LTS (for some reason I can't
get apt-get update|upgrade to see the new 3.12.04 version - but perhaps
it's because I'm in Kenya and it'll take a while to filter to whatever
repositories are mirrored here). We have circulation rules that allow for
students to keep books for 2 weeks and staff for a month. The email they
receive correctly acknowledges check-out and check-ins with a date/time
stamp, but always puts the time to 23:59. I do not want to have the
Circulation Librarian manually enter the date and time (by sliding those
indicators) depending on the patron; I would prefer for the system to
automatically calculate, as it does, whether the check-out is for 2 or 4
weeks. But if our library closes at 17h30 I don't want the  due-time to be
at 23h59. Is there a way - or could there be a way created - to set a
closing time for the library, so that hourly loans (for our Reserve
Section) can still be calculated, but if it falls past closing-time, e.g.
17h30 (for instance 23h59 or 22h00) it is reduced to 17h30 (so the students
don't tell our library staff that they came at midnight but no-one was
around)? I hope you understand what I'm trying to say. Another way of
wording it might be to think about it in terms of the current 'Hard Date'
in the Circulation and Fine Rules section in Administration in the Staff
client. There is an option in the rules for a 'Hard Date' - but I would
like to propose a 'Hard Time', i.e. whatever day is arrived at by the rules
- the latest possible time would be the 'Hard Time'. This would work for us
as we have two branches, one whose books must be returned by 17h30 of that
day, and another whose books must be returned by 11h00. If there was a
hard-time preference in the circulation rule it would be possible to
specify it on a branch-by-branch basis.

My second question is not so much a feature request (or if it's already
there, a plea to have it pointed out to me) as it is to ask for some
guidance from someone who has more experience in setting the circulation
rules. Our library has a reserve section. Currently we allow books to be
borrowed for a short-term period - a few hours during the day, the book not
to be removed from the library (but we want the book checked-out and
checked-in from the Reserve section so we can track usage and which
students are actually accessing the material); But the reserved books may
leave the library if it is borrowed between 17h00-17h30 - then it may be
kept overnight, to be returned by 08h30. Currently I'm setting the rules up
so that books from the Reserve section (an itemtype) have a 16 hour loan
period so that if it's loaned at 17h00 the system calculates that it must
be due back by 08h00 (though they receive an email saying it's due back 16
hours later - which ideally I'd like it to be 3 or 4 hours). But I need it
so that if it's loaned on Friday it will be due on Monday 08h00. If I set
holiday dates in the Calendar of the Library - it will ignore weekends -
but that means for our 'normal loans (as explained in the previous problem)
the 14 or 28 day rules (which used to include weekends) becomes more of an
18 or 32 day rule - but if that's the only way to get the Reserve Section
check-outs working, then I will request our Librarian to change the rules
to exclude weekends from all circulation check-in calculations... (I have
individual rules for each branch, itemptype, and patron category - with the
only 'All' rule being for the Reserve section.) Another idea I had was that
fines would not accrue for Reserved books until 3 days - (i.e. Sat
08h30-Sun-08h30-Mon-08h30) - but that means a book could be taken out on
Monday and the patron only show up in our morning emailed fine report on
Wed, which given that we only have 1 copy of every book, would badly
inconvenience our students. I think the 'Hard Time' feature could be
extended to have an option that if it falls on 'end of week' then it is due
'beginning of following week' might solve it - but I don't know if that
would be of interest to the wider community - or

Re: [Koha] Import MARC record error: utf8 \xE0

2013-09-01 Thread Bernardo Gonzalez Kriegel

Hi,
I found the same error using the stage tool,
but I can import successfully using command line tool bulkmarcimport.pl [1]

Perhaps is a bug in the encoding/decoding of the stage tool.
As a workaround try the command line import tool.

Regards,
Bernardo

[1] http://snag.gy/T3X02.jpg




-- 
Bernardo Gonzalez Kriegel
bgkrie...@gmail.com


On Sat, Aug 31, 2013 at 3:17 AM, Pongtawat Chippimolchai 
pongtawa...@gmail.com wrote:

 Hello all,

 I got the following error while importing MARC records into Koha 3.12.3 (on
 Ubuntu 12.04 amd64 via Debian package):

 utf8 \xE0 does not map to Unicode at /usr/lib/perl/5.14/Encode.pm line
 174.

 The record staged fine but it couldn't be imported. When clicking to view
 it in staging area, the same error appear.

 I checked the imported file, and it seem to look fine to me. The MARCXML in
 the database look ok too. But if I save the MARC BLOB and use marcdump on
 it, I got the same error.

 The imported file is here:
 https://www.dropbox.com/s/u25mf4dqk69wtgx/error.mrc
 The saved MARC BLOB is here:
 https://www.dropbox.com/s/soawpzmnuy5j7zr/import_records-marc.bin

 Running perl -d commit_file.pl show something like this:

 -
 utf8 \xE0 does not map to Unicode at /usr/lib/perl/5.14/Encode.pm line
 174.
  at /usr/lib/perl/5.14/Encode.pm line 174
Encode::decode('UTF-8', 'ha\x{1e}0

 \x{1f}a\x{e0}\x{b9}\x{84}\x{e0}\x{b8}\x{9e}\x{e0}\x{b8}\x{a8}\x{e0}\x{b8}\x{b2}\x{e0}\x{b8}\x{a5}

 \x{e0}\x{b8}\x{81}\x{e0}\x{b8}\x{a4}\x{e0}\x{b8}\x{a9}\x{e0}\x{b8}\x{8e}\x{e0}\x{b8}\x{b2}\x{e0}\x{b8}\x{98}\x{e0}\x{b8}\x{b4}\x{e0}\x{b8}\x{a7}\x{e0}\x{b8}\x{b8}\x{e0}\x{b8}\x{92}\x{e0}\x{b8}',
 1) called at /usr/share/perl5/MARC/File/Encode.pm line 35
MARC::File::Encode::marc_to_utf8('ha\x{1e}0

 \x{1f}a\x{e0}\x{b9}\x{84}\x{e0}\x{b8}\x{9e}\x{e0}\x{b8}\x{a8}\x{e0}\x{b8}\x{b2}\x{e0}\x{b8}\x{a5}

 \x{e0}\x{b8}\x{81}\x{e0}\x{b8}\x{a4}\x{e0}\x{b8}\x{a9}\x{e0}\x{b8}\x{8e}\x{e0}\x{b8}\x{b2}\x{e0}\x{b8}\x{98}\x{e0}\x{b8}\x{b4}\x{e0}\x{b8}\x{a7}\x{e0}\x{b8}\x{b8}\x{e0}\x{b8}\x{92}\x{e0}\x{b8}')
 called at /usr/share/perl5/MARC/File/USMARC.pm line 172


  
 MARC::File::USMARC::decode('00932nas0a2200217000455001700080043000170200014000600...')
 called at /usr/share/perl5/MARC/Record.pm line 81
MARC::Record::new_from_usmarc('MARC::Record',
 '00932nas0a2200217000455001700080043000170200014000600...') called
 at /usr/share/koha/lib/C4/ImportBatch.pm line 585
C4::ImportBatch::BatchCommitRecords(13, '', 100, 'CODE(0x5227d58)')
 called at /usr/share/koha/bin/commit_file.pl line 85
main::process_batch(13) called at
 /usr/share/koha/bin/commit_file.plline 57
 -

 I really don't get the way Perl (and Koha) handle UTF8 at all. If I print
 that escaped string out, some character will be readable but the others
 appear as ?. For the records that could be imported, if I print out their
 data from Perl I see the same pattern (readable char mixed with ?). But
 they somehow end up correct in the database.

 I have no idea what's wrong here. If you have any clue, please help.

 Thank you very much,
 Pongtawat
 ___
 Koha mailing list  http://koha-community.org
 Koha@lists.katipo.co.nz
 http://lists.katipo.co.nz/mailman/listinfo/koha

___
Koha mailing list  http://koha-community.org
Koha@lists.katipo.co.nz
http://lists.katipo.co.nz/mailman/listinfo/koha

[Koha] Import MARC record error: utf8 \xE0

Re: [Koha] Import MARC record error: utf8 \xE0

[Koha] Thank you - Circulation Rules Help needed - 1) Hard Time Proposal - 2) Query on Overnight/Weekend Loans

Re: [Koha] Import MARC record error: utf8 \xE0

4 matches

Site Navigation

Mail list logo

Footer information