Hi Paul, In order to understand the scope of the problem I asked:
> > in which locales the problem occurs: > > - in the "C" locale, > > - in single-byte locales other than "C", > > - or both? > I've tested only OpenBSD 7.9. Apparently it supports only the C and the > en_US.UTF-8 locales I ran your experiment on glibc, Cygwin, OpenBSD (both with and without the modules 'mbrtoc32-regular' and 'uchar-h-c23'). glibc: C OK fr_FR OK Cygwin: C 0xA0..0xFF: c32isprint = 1, isprint = 0 fr_FR.ISO8859-1 OK OpenBSD: C 0xA0..0xFF: c32isprint = 1, isprint = 0 So, the problem exists only for the "C" locale. It is related to the trouble we have with mbrtowc in the C locale: https://sourceware.org/bugzilla/show_bug.cgi?id=19932 https://sourceware.org/bugzilla/show_bug.cgi?id=29511 https://lists.gnu.org/archive/html/bug-gnulib/2023-03/msg00145.html > I see several possible fixes: > > (1) Skip the sed tests on problematic platforms. > > (2) Change Gnulib c32isprint so that in a single-byte locale, it succeeds if > and only if given one of the at most 255 char32_t values corresponding to the > unsigned char value that isprint succeeds on. > > (3) Replace mbrtoc32 on problematic platforms where in single-byte locales it > can yield values that cause isprint to be inconsistent with c32isprint. > > (4) Change 'sed' so that in single-byte locales it uses only traditional > <ctype.h> primitives like isprint, and never uses <uchar.h> functions. (1) is not good. 'sed' is supposed to work alike on all platforms; the unit tests are the means to ensuring that. (2) is not good, IMO. char32_t values, starting with ISO C 23, are supposed to be Unicode code points and therefore behave in a locale-independent way. We need to overcome the old mess [1][2]. (4) This would be an ugly workaround. Similar work would then be needed in other packages as well. So, for me, (3) is the right approach: In the "C" locale, map 0x80..0xFF not to U+0080..U+00FF but to U+DF80..U+DFFF, like musl libc does it. This will ensure that c32isprint returns 0 on these inputs, like isprint does. If you agree with this approach, I can implement it. Bruno [1] https://www.gnu.org/software/libunistring/manual/html_node/The-wchar_005ft-mess.html [2] https://www.gnu.org/software/libunistring/manual/html_node/The-char32_005ft-problem.html
