Hi, Attached is v2 of the patch, along with a Python script I wrote to test PostgreSQL's SASLprep implementation against Python's Unicode 3.2 implementation.
The updated patch addresses two issues: 1. Prohibited output characters and bidirectional properties must be checked after NFKC normalization, using the normalized string rather than the original input. 2. Code points unassigned in Unicode 3.2 must be rejected before normalization. PostgreSQL's normalizer uses newer Unicode data, which can normalize some previously unassigned code points into characters that were already assigned in Unicode 3.2. Checking only after normalization would allow these characters to bypass the unassigned code point check. I've also added 13 regression test cases covering both issues. Nine of these fail with the original implementation, while all 13 pass with the patch. Additional testing To verify the changes more thoroughly, I wrote a Python script that uses unicodedata.ucd_3_2_0 and stringprep as a reference implementation of SASLprep. The script tests all Unicode scalar values (excluding U+0000 and surrogates), along with selected combinations intended to exercise normalization and bidirectional rules. It checks 1,112,210 test cases in total. The results were: - Original PostgreSQL implementation: 104 mismatches - Initial patch: 704 mismatches - Updated patch (v2): 5 mismatches The increase with the initial patch revealed the problem with checking unassigned code points only after normalization, which is now addressed in v2. The five remaining mismatches are all normalization differences involving the code points U+2F868, U+2F874, U+2F91F, U+2F95F, and U+2F9BF. These correspond to the five mappings corrected by Unicode Corrigendum #4: https://www.unicode.org/versions/corrigendum4.html In all five cases, both PostgreSQL and the reference implementation accept the input, but produce different normalized output. Test script I've attached saslprep_diff.py as a standalone testing tool. It requires Python 3 and psycopg, and runs against a PostgreSQL instance with the test_saslprep extension installed. I'm not sure how best to integrate this kind of exhaustive testing into PostgreSQL's existing test infrastructure, especially since the script currently depends on Python and psycopg. For now, I've included the representative test cases in the regular SQL regression tests and attached the Python script separately so the more extensive testing can be reproduced. I'd be happy to adapt the script or its generated test cases if there is a preferred way to include this kind of coverage in the existing test suite. I understand that the focus is currently on getting v19 ready, so there is no urgency for a review. I just wanted to make the updated patch and test results available. Thanks!
From 7db36e8d8a70f116f354281e511da18286a2cb95 Mon Sep 17 00:00:00 2001 From: Robert I Nilsson <[email protected]> Date: Tue, 10 Nov 2026 17:36:05 +0100 Subject: [PATCH v2] Fix SASLprep validation after NFKC normalization SASLprep requires prohibited output characters and bidirectional properties to be checked after Unicode normalization. However, pg_saslprep() performs these checks on the original input, which can cause valid strings to be rejected and invalid strings to be accepted. Additionally, SASLprep uses Unicode 3.2 to identify unassigned code points, while PostgreSQL's normalizer uses newer Unicode data. This can cause code points unassigned in Unicode 3.2 to normalize into assigned characters, bypassing the unassigned check. Reject unassigned Unicode 3.2 code points before normalization, then perform prohibited output and bidirectional checks on the normalized result. Add regression tests covering these cases. --- src/common/saslprep.c | 35 +++++++--- .../test_saslprep/expected/test_saslprep.out | 69 +++++++++++++++++++ .../test_saslprep/sql/test_saslprep.sql | 65 +++++++++++++++++ 3 files changed, 161 insertions(+), 8 deletions(-) diff --git a/src/common/saslprep.c b/src/common/saslprep.c index a2bb60ac0cb..4dc01fc70db 100644 --- a/src/common/saslprep.c +++ b/src/common/saslprep.c @@ -1049,6 +1049,7 @@ pg_saslprep(const char *input, char **output) char32_t *input_chars = NULL; char32_t *output_chars = NULL; int input_size; + int output_size; char *result; int result_size; int count; @@ -1113,6 +1114,17 @@ pg_saslprep(const char *input, char **output) if (input_size == 0) goto prohibited; /* don't allow empty password */ + /* + * Reject code points unassigned in Unicode 3.2 before normalization. + * The normalizer uses newer Unicode data and may map such code + * points to characters that were already assigned in Unicode 3.2. + */ + for (i = 0; i < input_size; i++) + { + if (IS_CODE_IN_TABLE(input_chars[i], unassigned_codepoint_ranges)) + goto prohibited; + } + /* * 2) Normalize -- Normalize the result of step 1 using Unicode * normalization. @@ -1121,13 +1133,20 @@ pg_saslprep(const char *input, char **output) if (!output_chars) goto oom; + output_size = 0; + for (wp = output_chars; *wp; wp++) + output_size++; + + if (output_size == 0) + goto prohibited; /* don't allow empty password, after NFKC */ + /* * 3) Prohibit -- Check for any characters that are not allowed in the * output. If any are found, return an error. */ - for (i = 0; i < input_size; i++) + for (i = 0; i < output_size; i++) { - char32_t code = input_chars[i]; + char32_t code = output_chars[i]; if (IS_CODE_IN_TABLE(code, prohibited_output_ranges)) goto prohibited; @@ -1157,9 +1176,9 @@ pg_saslprep(const char *input, char **output) * MUST be the last character of the string." */ contains_RandALCat = false; - for (i = 0; i < input_size; i++) + for (i = 0; i < output_size; i++) { - char32_t code = input_chars[i]; + char32_t code = output_chars[i]; if (IS_CODE_IN_TABLE(code, RandALCat_codepoint_ranges)) { @@ -1170,12 +1189,12 @@ pg_saslprep(const char *input, char **output) if (contains_RandALCat) { - char32_t first = input_chars[0]; - char32_t last = input_chars[input_size - 1]; + char32_t first = output_chars[0]; + char32_t last = output_chars[output_size - 1]; - for (i = 0; i < input_size; i++) + for (i = 0; i < output_size; i++) { - char32_t code = input_chars[i]; + char32_t code = output_chars[i]; if (IS_CODE_IN_TABLE(code, LCat_codepoint_ranges)) goto prohibited; diff --git a/src/test/modules/test_saslprep/expected/test_saslprep.out b/src/test/modules/test_saslprep/expected/test_saslprep.out index 92f93365343..1e66fc77fa0 100644 --- a/src/test/modules/test_saslprep/expected/test_saslprep.out +++ b/src/test/modules/test_saslprep/expected/test_saslprep.out @@ -149,4 +149,73 @@ SELECT <DEL> | \x7f | (,PROHIBITED) (128 rows) +-- Test SASLprep validation before and after normalization. +-- Prohibited output characters and bidi rules must be checked +-- after normalization, while code points unassigned in Unicode +-- 3.2 must be rejected before normalization. +WITH cases(name, src, expected_status, expected_res) AS ( + VALUES + -- Characters that become valid after normalization. + ('combining grave', 'cd80', + 'SUCCESS', 'cc80'), + ('combining acute', 'cd81', + 'SUCCESS', 'cc81'), + ('grave after a', '61cd80', + 'SUCCESS', 'c3a0'), + -- Normalization can increase the number of code points. + ('latin ligature ff', 'efac80', + 'SUCCESS', '6666'), + -- Normalization introduces prohibited bidi sequences. + ('hebrew presentation', 'efac9d', + 'PROHIBITED', NULL), + ('hebrew shin', 'efacaa', + 'PROHIBITED', NULL), + ('arabic ligature', 'efb19b', + 'PROHIBITED', NULL), + -- Normalization changes bidi properties (L to R). + ('alef after latin', '41e284b5', + 'PROHIBITED', NULL), + ('alef in RTL', 'd790e284b5d791', + 'SUCCESS', 'd790d790d791'), + -- Normalization introduces L characters into an RTL string. + ('rupee in RTL', 'd790e282a8d791', + 'PROHIBITED', NULL), + -- Code points unassigned in Unicode 3.2 that normalize + -- to assigned characters using newer Unicode data. + ('unassigned Greek', 'cfb9', + 'PROHIBITED', NULL), + ('unassigned modifier', 'e1b4ac', + 'PROHIBITED', NULL), + -- Control case: valid RTL string containing a digit. + ('valid RTL digits', 'd79031d791', + 'SUCCESS', 'd79031d791') +) +SELECT name, + actual.status, + encode(actual.res, 'hex') AS result, + actual.status = expected_status + AND encode(actual.res, 'hex') + IS NOT DISTINCT FROM expected_res AS passed +FROM cases +CROSS JOIN LATERAL test_saslprep( + decode(cases.src, 'hex') +) AS actual +ORDER BY name; + name | status | result | passed +---------------------+------------+--------------+-------- + alef after latin | PROHIBITED | | t + alef in RTL | SUCCESS | d790d790d791 | t + arabic ligature | PROHIBITED | | t + combining acute | SUCCESS | cc81 | t + combining grave | SUCCESS | cc80 | t + grave after a | SUCCESS | c3a0 | t + hebrew presentation | PROHIBITED | | t + hebrew shin | PROHIBITED | | t + latin ligature ff | SUCCESS | 6666 | t + rupee in RTL | PROHIBITED | | t + unassigned Greek | PROHIBITED | | t + unassigned modifier | PROHIBITED | | t + valid RTL digits | SUCCESS | d79031d791 | t +(13 rows) + DROP EXTENSION test_saslprep; diff --git a/src/test/modules/test_saslprep/sql/test_saslprep.sql b/src/test/modules/test_saslprep/sql/test_saslprep.sql index 00bad48eca7..07abc783b72 100644 --- a/src/test/modules/test_saslprep/sql/test_saslprep.sql +++ b/src/test/modules/test_saslprep/sql/test_saslprep.sql @@ -16,4 +16,69 @@ SELECT test_saslprep(set_byte('\x00'::bytea, 0, a)) AS saslprep FROM generate_series(0,127) AS a; +-- Test SASLprep validation before and after normalization. +-- Prohibited output characters and bidi rules must be checked +-- after normalization, while code points unassigned in Unicode +-- 3.2 must be rejected before normalization. +WITH cases(name, src, expected_status, expected_res) AS ( + VALUES + -- Characters that become valid after normalization. + ('combining grave', 'cd80', + 'SUCCESS', 'cc80'), + + ('combining acute', 'cd81', + 'SUCCESS', 'cc81'), + + ('grave after a', '61cd80', + 'SUCCESS', 'c3a0'), + + -- Normalization can increase the number of code points. + ('latin ligature ff', 'efac80', + 'SUCCESS', '6666'), + + -- Normalization introduces prohibited bidi sequences. + ('hebrew presentation', 'efac9d', + 'PROHIBITED', NULL), + + ('hebrew shin', 'efacaa', + 'PROHIBITED', NULL), + + ('arabic ligature', 'efb19b', + 'PROHIBITED', NULL), + + -- Normalization changes bidi properties (L to R). + ('alef after latin', '41e284b5', + 'PROHIBITED', NULL), + + ('alef in RTL', 'd790e284b5d791', + 'SUCCESS', 'd790d790d791'), + + -- Normalization introduces L characters into an RTL string. + ('rupee in RTL', 'd790e282a8d791', + 'PROHIBITED', NULL), + + -- Code points unassigned in Unicode 3.2 that normalize + -- to assigned characters using newer Unicode data. + ('unassigned Greek', 'cfb9', + 'PROHIBITED', NULL), + + ('unassigned modifier', 'e1b4ac', + 'PROHIBITED', NULL), + + -- Control case: valid RTL string containing a digit. + ('valid RTL digits', 'd79031d791', + 'SUCCESS', 'd79031d791') +) +SELECT name, + actual.status, + encode(actual.res, 'hex') AS result, + actual.status = expected_status + AND encode(actual.res, 'hex') + IS NOT DISTINCT FROM expected_res AS passed +FROM cases +CROSS JOIN LATERAL test_saslprep( + decode(cases.src, 'hex') +) AS actual +ORDER BY name; + DROP EXTENSION test_saslprep; -- 2.53.0
saslprep_diff.py
Description: application/python
