Hi Laurence,
> I'm trying to create MARC records from serials data exported
> from SFX, using MARC::Charset version 0.98 to convert UTF-8
> strings to MARC-8. It seems to be failing on extended latin
> characters like U+00C5 CAPITAL LETTER A WITH RING ABOVE
The encoding, U+00C5 (CAPITAL LETTER A WITH RING ABOVE), is a precomposed
character [1]. While U+00C5 is a perfectly good Unicode encoding, I believe
that it is still the recommended practice for Unicode-encoded MARC-21 records
to use base and combining characters, and U+00C5 doesn't have a direct
equivalent in the MARC-21 repertoire [2,3].
If the strings are first normalized using Unicode Normalization Form D, they
should convert okay [4,5].
> The records convert using Terry Reese's MarcEdit OK.
Perhaps MarcEdit incorporates the decomposition or has direct conversion of
precomposed Unicode to decomposed MARC-8.
-- Michael
[1] The decomposition (i.e. base and combining character) values for "CAPITAL
LETTER A WITH RING ABOVE" would be U+0041 (LATIN CAPITAL LETTER A) followed by
U+030A (COMBINING RING ABOVE).
[2] WORKING PRINCIPLES TO BE FOLLOWED IN MAPPING OF CHARACTERS FROM USMARC TO
UNICODE/UCS
* Accented letters ... will continue to be encoded as a base letter
and non-spacing marks. Use of precomposed accented letters is not
sanctioned at this stage.
From "USMARC Character Set Issues and Mapping to Unicode/UCS"
http://www.loc.gov/marc/marbi/1996/96-10.html
[3] MARC 21 Specifications > CHARACTER SETS > Code Tables
http://www.loc.gov/marc/specifications/specchartables.html
[4] Preprocessing Requirements
... preprocessing of the Unicode record before the conversion to
MARC-8 takes place. In all of the above techniques, the following
steps for decomposing diacritics were presumed.
Decompose the precomposed base character/character modifier combinations
using Unicode Normalization Form D (NFD) which produces exact equivalents,
and primarily applies decomposition to precomposed characters with
diacritics.
From "Technique for conversion of Unicode to MARC-8"
http://www.loc.gov/marc/marbi/2006/2006-04.html
[5] W3C > Charlint - A Character Normalization Tool
http://www.w3.org/International/charlint/
# Michael Doran, Systems Librarian
# University of Texas at Arlington
# 817-272-5326 office
# 817-688-1926 mobile
# [EMAIL PROTECTED]
# http://rocky.uta.edu/doran/
> -----Original Message-----
> From: Laurence Lockton [mailto:[EMAIL PROTECTED]
> Sent: Tuesday, September 18, 2007 5:21 AM
> To: perl4lib@perl.org
> Subject: MARC::Charset 'utf8_to_marc8'
>
> Hi,
>
> I'm trying to create MARC records from serials data exported
> from SFX, using MARC::Charset version 0.98 to convert UTF-8
> strings to MARC-8. It seems to be failing on extended latin
> characters like U+00C5 CAPITAL LETTER A WITH RING ABOVE,
> giving "no mapping found at position 176" for example.
> The records convert using Terry Reese's MarcEdit OK. Am I
> doing the wrong thing? Any advice gratefully received.
>
> Many thanks,
> Laurence Lockton
> University of Bath
> UK
>
