https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42948
Bug ID: 42948
Summary: Diacritics corrupted when saving authority records
(UTF-8 mangled into unrelated Unicode characters, e.g.
é→ř, ú/ñ→ž)
Initiative type: ---
Sponsorship ---
status:
Product: Koha
Version: 26.05
Hardware: PC
OS: Linux
Status: NEW
Severity: major
Priority: P5 - low
Component: MARC Authority data support
Assignee: [email protected]
Reporter: [email protected]
QA Contact: [email protected]
Target Milestone: ---
Created attachment 201329
-->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201329&action=edit
Attachments included: 1 Screenshot of authority before save (correct name)
2 Screenshot of authority after save (corrupted name) 3 Screenshot
showing the same corruption when typed
When creating or saving an authority record containing accented characters
(e.g. Spanish names with á, é, í, ó, ú, ñ), the diacritics are corrupted into
unrelated Unicode characters upon save. The corruption pattern is not simple
mojibake (e.g. "é") but rather diacritics merging with adjacent letters into
different Unicode characters, including letters from other alphabets (e.g. ř,
ž, ø, æ).
Example:
Correct name: "Pérez Zúñiga, Juan"
After saving the authority in Koha: "Přez Žiga, Juan"
Correct name: "García-Posada, Miguel"
After saving: "Garcá-Posada, Miguel" (character dropped entirely)
Steps to reproduce:
Go to Authorities module.
Create a new authority manually (New authority), entering a personal name with
accented characters directly in field 100 $a (e.g. "Hierro, José" or "Pérez
Zúñiga, Juan").
Save the record.
Reopen the saved authority record.
Observe that the accented characters have been replaced/corrupted.
This reproduces even when the authority is created entirely manually, without
using Z39.50/SRU import, ruling out any Z39.50 target encoding issue.
Troubleshooting already performed (ruling out other causes):
Confirmed the issue is NOT related to the Z39.50/BNE server encoding: tested
with UTF-8, MARC-8, ISO-5426, ISO-8859-1 and ISO_6937 encodings on the Z39.50
target — all produce different (but still wrong) corruption patterns, and the
corruption also occurs when manually typing the authority without using Z39.50
at all.
Confirmed the Leader position 09 (Character coding scheme) of the bibliographic
record is correctly set to "a" (UCS/Unicode), auto-filled.
Confirmed the same corruption occurs when importing the same correct UTF-8 data
from a different Z39.50 target (Library of Congress Names), which displays the
name correctly in the search results list, but the data is still corrupted once
saved as a Koha authority — pointing to the save process itself, not the data
source.
Confirmed database-level encoding is correctly configured throughout:
Database: utf8mb4 / utf8mb4_uca1400_ai_ci
Table auth_header: column marcxml is longtext with collation utf8mb4_unicode_ci
MySQL/MariaDB session variables (character_set_client,
character_set_connection, character_set_database, character_set_results,
character_set_server) all set to utf8mb4
collation_connection, collation_database, collation_server all
utf8mb4_uca1400_ai_ci
Given the above, the corruption appears to originate in the Koha application
layer (Perl/Plack) during the authority save process, rather than in any Z39.50
target configuration or in the database encoding configuration.
Environment:
Koha version: 26.05.00
Database: MariaDB (utf8mb4, collation utf8mb4_uca1400_ai_ci)
Authority framework: MARC21 PERSO_NAME
Attachments included:
• Screenshot of authority before save (correct name)
• Screenshot of authority after save (corrupted name)
• Screenshot showing the same corruption when typed manually (no Z39.50
involved)
--
You are receiving this mail because:
You are the assignee for the bug.
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[email protected]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
