Hi, This patch fixes Bug #19831, reported by chunling qin on pgsql-bugs (https://www.postgresql.org/message-id/[email protected]): convert() between single-byte Cyrillic encodings can return a wrong character, can silently succeed where the character has no representation in the target encoding, or can fail where it does. The same direct conversions are used for client encoding conversion and COPY ... ENCODING, so this is silent data corruption, not just a function-level oddity.
I reproduced every case from the report on 18.6. For example, KOI8R byte 0xa4 (U+2553) converts directly to WIN866 as 0xf3 (a different character), while going through UTF-8 with PostgreSQL's own generated maps yields 0xd6. Root cause: the hand-written per-byte tables in src/backend/utils/mb/conversion_procs/cyrillic_and_mic/cyrillic_and_mic.c have drifted from the Unicode-generated maps. To find every bad entry, I composed the generated maps (X_to_utf8 followed by utf8_to_Y) for all 12 direct pairs among KOI8R, WIN866, WIN1251 and ISO-8859-5, and compared against the tables. There were 234 disagreeing entries: 12 wrong characters, 26 entries that produced a byte where conversion must fail, and 196 entries that failed where conversion must succeed (the tables are mostly zero across the 0x80-0xBF pseudographics range). The patch corrects exactly those entries; no code changes. Regression tests: conversion.sql now checks, for each of the 12 pairs, converting every byte that has a representation in the target encoding, plus an individual must-fail case for each byte that previously slipped through. The new tests fail on the unpatched tree and pass with the fix. I also verified on a live server that, after the fix, the direct result agrees with the via-UTF-8 result for all 128 high bytes of all 12 pairs (before the fix, the per-pair mismatch counts were exactly the 234 above). One scope question: the same hand-written-table pattern exists in conversion_procs/latin2_and_win1250 (LATIN2 <-> WIN1250), and the report's LATIN2 case (byte 0x80 passing through unchanged) reproduces there too. I scoped this patch to the Cyrillic family covered by the bug report; I am happy to extend it or split it if reviewers prefer. This patch was prepared with AI assistance (Muse Spark, Meta AI); I built, reproduced, and tested everything locally myself. Regards, Koussay Mehdouani
fix-19831.patch
Description: Binary data
