Re: new module 'c32rtomb'

[Bruno Haible]

I wrote:
> On FreeBSD 12 and Solaris 11.4, the
> two encodings are different. The FreeBSD 12 wchar_t encoding is apparently
> based on ISO 2022 (very old).
> 
> The fix is to use mbrtoc32() on platforms where this is possible, namely
> on FreeBSD.

Actually, FreeBSD has a similar bug as Solaris 11.4. Therefore it is better to
NOT use the system's mbrtoc32() on FreeBSD 12.


2020-01-20  Bruno Haible  

mbrtoc32: Add note about FreeBSD 12.
* m4/mbrtoc32.m4 (gl_MBRTOC32_SANITYCHECK): Guess no also on FreeBSD.
* doc/posix-functions/mbrtoc32.texi: Mention that FreeBSD 12 is also
affected.

diff --git a/doc/posix-functions/mbrtoc32.texi 
b/doc/posix-functions/mbrtoc32.texi
index 9789bef..3151a09 100644
--- a/doc/posix-functions/mbrtoc32.texi
+++ b/doc/posix-functions/mbrtoc32.texi
@@ -20,7 +20,7 @@ glibc 2.19.
 @item
 This function does not recognize multibyte sequences that @code{mbrtowc}
 recognizes on some platforms:
-Solaris 11.4, mingw, MSVC 14.
+FreeBSD 12, Solaris 11.4, mingw, MSVC 14.
 @end itemize
 
 Portability problems not fixed by Gnulib:
diff --git a/m4/mbrtoc32.m4 b/m4/mbrtoc32.m4
index 3dee900..a5dc51a 100644
--- a/m4/mbrtoc32.m4
+++ b/m4/mbrtoc32.m4
@@ -1,4 +1,4 @@
-# mbrtoc32.m4 serial 2
+# mbrtoc32.m4 serial 3
 dnl Copyright (C) 2014-2020 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -136,10 +136,14 @@ AC_DEFUN([gl_MBRTOC32_SANITYCHECK],
 dnl is present.
 changequote(,)dnl
 case "$host_os" in
- # Guess no on Solaris, native Windows.
-  solaris* | mingw*) gl_cv_func_mbrtoc32_sanitycheck="guessing no" ;;
- # Guess yes otherwise.
-  *) gl_cv_func_mbrtoc32_sanitycheck="guessing yes" ;;
+  # Guess no on FreeBSD, Solaris, native Windows.
+  freebsd* | solaris* | mingw*)
+gl_cv_func_mbrtoc32_sanitycheck="guessing no"
+;;
+  # Guess yes otherwise.
+  *)
+gl_cv_func_mbrtoc32_sanitycheck="guessing yes"
+;;
 esac
 changequote([,])dnl
 if test $LOCALE_FR != none || test $LOCALE_ZH_CN != none; then
@@ -176,8 +180,8 @@ int main ()
 result |= 1;
 }
 }
-  /* This fails on Solaris 11.4:
- mbrtoc32 returns (size_t)-1.
+  /* This fails on FreeBSD 12 and Solaris 11.4:
+ mbrtoc32 returns (size_t)-2 or (size_t)-1.
  mbrtowc returns 4 (correct).  */
   if (setlocale (LC_ALL, "$LOCALE_ZH_CN") != NULL)
 {

Re: fstrcmp: memory is not reclaimed on exit

[Bruno Haible]

Hi Akim,

> > +==2837387== 320 bytes in 1 blocks are still reachable in loss record 1 of 1
> > +==2837387==at 0x483A80B: malloc (vg_replace_malloc.c:309)
> > +==2837387==by 0x117CFC: xmalloc (xmalloc.c:41)
> > +==2837387==by 0x11B743: UnknownInlinedFun (xalloc.h:103)
> > +==2837387==by 0x11B743: fstrcmp_bounded (fstrcmp.c:208)
> 
> which points to fstrcmp's allocation of its per-thread internal buffer.

Correct. This is the per-thread buffer.

> AFAICT, glthread takes care of reclaiming the memory when the thread
> exits, that's the whole point of the "free" in the first line of the
> body here:
> 
> > static void
> > keys_init (void)
> > {
> >   gl_tls_key_init (buffer_key, free);
> >   gl_tls_key_init (bufmax_key, NULL);
> >   /* The per-thread initial values are NULL and 0, respectively.  */
> > }
> 
> glthread/tls.h reads
> 
> >  Initialization:gl_tls_key_init (name, destructor);
> > 
> >A destructor is a function pointer of type 'void (*) (void *)', called
> >when a thread exits, and taking the last per-thread value as argument.  
> > It
> >is unspecified whether the destructor function is called when the last
> >per-thread value is NULL.  On some platforms, the destructor function is
> >not called at all.
> 
> I can see that it's not expected to work on some platforms, but in the
> case of the user the platform is fairly mainstream:

Meanwhile this destructor stuff even works on native Windows. The list of
platforms where it does not work is very small (something like mingw with
winpthreads, IIRC).

> So I don't know what to do.  Is this a red herring related to Valgrind
> as a platform?

Do the threads still exist at the moment valgrind does its inventory of left-
over memory? In particular:
  - Did you create threads, in which fstrcmp is run? If yes, are they still
running?
  - Or did you run fstrcmp in the main thread? Most likely valgrind does its
inventory in the main thread, during exit(). This means that at this point
the fstrcmp buffer for the main thread still exists. In other words, you
should treat thread-local memory allocations for the main thread like
static memory allocations (e.g. like in uniqstr.c).

Bruno

Re: FreeBSD 11.2: glthread build failure

[Bruno Haible]

Hi Tim,

> >>>   CCLD wget2
> > 
> > Please make it a habit to use "make V=1" when submitting a report.
> > I don't want to see "CCLD wget2"; I want to see the actual link
> > command line.
> 
> I don't have direct console/shell access - the issue is/was on a FreeBSD
> instance running somewhere that we use for testing via Gitlab CI.

You could, instead of running
   make
run
   make || { echo 'Retrying with verbosity'; make V=1; exit 1; }
This way, the overall log is compact, but it is verbose at the place
where we need it.

> it wasn't obvious in that moment that 'make V=1' would be helpful.

The best practice for bug reporters has always been - for years:
"Show the command that was executed, and show the results that it produced."

Bruno

Re: warning in lib/unistr/u8-uctomb.c

[Bruno Haible]

Correcting the subject.

> cc1: all warnings being treated as errors

"this statement may fall through" is a warning. *You* turned it into an
error by using -Werror or -Werror=implicit-fallthrough. It is therefore
misleading to say that there is a compilation error in lib/unistr/u8-uctomb.c.

> ../../grep/lib/unistr/u8-uctomb.c: In function 'u8_uctomb':
> ../../grep/lib/unistr/u8-uctomb.c:64:65: error: this statement may fall 
> through [-Werror=implicit-fallthrough=]
>  case 4: s[3] = 0x80 | (uc & 0x3f); uc = uc >> 6; uc |= 0x1;
>   ~~~^~
> ../../grep/lib/unistr/u8-uctomb.c:65:13: note: here
>  case 3: s[2] = 0x80 | (uc & 0x3f); uc = uc >> 6; uc |= 0x800;
>  ^~~~
> ../../grep/lib/unistr/u8-uctomb.c:65:65: error: this statement may fall 
> through [-Werror=implicit-fallthrough=]
>  case 3: s[2] = 0x80 | (uc & 0x3f); uc = uc >> 6; uc |= 0x800;
>   ~~~^~~~
> ../../grep/lib/unistr/u8-uctomb.c:66:13: note: here
>  case 2: s[1] = 0x80 | (uc & 0x3f); uc = uc >> 6; uc |= 0xc0;
>  ^~~~

Fixed through the patch below.

But something is wrong with the way you build packages: This code is only seen
by the compiler when HAVE_INLINE is undefined or 0. So, apparently the
autoconfiguration determined that the compiler does not support the 'inline'
keyword. Since the compiler is clang, this finding is not true.

Most likely, you had -Werror in $CC $CFLAGS already during configuration
time. This is NOT SUPPORTED, since it causes many configure tests to produce
wrong results.


2020-01-20  Bruno Haible  

unistr/u8-uctomb: Fix warning.
Reported by Andreas Schwab  in
.
* lib/unistr/u8-uctomb.c (FALLTHROUGH): New macro.
(u8_uctomb): Add FALLTHROUGH markers.

diff --git a/lib/unistr/u8-uctomb.c b/lib/unistr/u8-uctomb.c
index f093822..d998002 100644
--- a/lib/unistr/u8-uctomb.c
+++ b/lib/unistr/u8-uctomb.c
@@ -25,6 +25,14 @@
 /* Specification.  */
 #include "unistr.h"
 
+#ifndef FALLTHROUGH
+# if __GNUC__ < 7
+#  define FALLTHROUGH ((void) 0)
+# else
+#  define FALLTHROUGH __attribute__ ((__fallthrough__))
+# endif
+#endif
+
 #if !HAVE_INLINE
 
 int
@@ -62,7 +70,9 @@ u8_uctomb (uint8_t *s, ucs4_t uc, int n)
   switch (count) /* note: code falls through cases! */
 {
 case 4: s[3] = 0x80 | (uc & 0x3f); uc = uc >> 6; uc |= 0x1;
+  FALLTHROUGH;
 case 3: s[2] = 0x80 | (uc & 0x3f); uc = uc >> 6; uc |= 0x800;
+  FALLTHROUGH;
 case 2: s[1] = 0x80 | (uc & 0x3f); uc = uc >> 6; uc |= 0xc0;
   /*case 1:*/ s[0] = uc;
 }

Re: FreeBSD 11.2: glthread build failure

[Bruno Haible]

> I can confirm that your patch fixes the issue here.

Great. Thanks.

Testing showed a regression: glthread_once is no longer working on FreeBSD 11,
because in this code

# define glthread_once(ONCE_CONTROL, INITFUNCTION) \
(pthread_in_use () \
 ? pthread_once (ONCE_CONTROL, INITFUNCTION)   \
 : (glthread_once_singlethreaded (ONCE_CONTROL) ? (INITFUNCTION (), 0) : 0))

pthread_in_use () now evaluates to 1, and pthread_once from libc returns ENOSYS.
Thus glthread_once returns ENOSYS, and gl_once calls abort().

This patch fixes it.


2020-01-20  Bruno Haible  

lock: Fix test-once1 failure on FreeBSD 11 (regression from 2020-01-19).
* lib/glthread/lock.c (glthread_once_multithreaded): New function.
* lib/glthread/lock.h (glthread_once_multithreaded): New declaration.
(glthread_once): Use it.

diff --git a/lib/glthread/lock.c b/lib/glthread/lock.c
index 6336591..1f6f713 100644
--- a/lib/glthread/lock.c
+++ b/lib/glthread/lock.c
@@ -718,6 +718,26 @@ glthread_once_singlethreaded (pthread_once_t *once_control)
 return 0;
 }
 
+# if !(PTHREAD_IN_USE_DETECTION_HARD || USE_POSIX_THREADS_WEAK)
+
+int
+glthread_once_multithreaded (pthread_once_t *once_control,
+ void (*init_function) (void))
+{
+  int err = pthread_once (once_control, init_function);
+  if (err == ENOSYS)
+{
+  /* This happens on FreeBSD 11: The pthread_once function in libc returns
+ ENOSYS.  */
+  if (glthread_once_singlethreaded (once_control))
+init_function ();
+  return 0;
+}
+  return err;
+}
+
+# endif
+
 #endif
 
 /* = */
diff --git a/lib/glthread/lock.h b/lib/glthread/lock.h
index 9c70f4e..1de86fe 100644
--- a/lib/glthread/lock.h
+++ b/lib/glthread/lock.h
@@ -505,10 +505,19 @@ extern int glthread_recursive_lock_destroy_multithreaded 
(gl_recursive_lock_t *l
 typedef pthread_once_t gl_once_t;
 # define gl_once_define(STORAGECLASS, NAME) \
 STORAGECLASS pthread_once_t NAME = PTHREAD_ONCE_INIT;
-# define glthread_once(ONCE_CONTROL, INITFUNCTION) \
-(pthread_in_use () 
\
- ? pthread_once (ONCE_CONTROL, INITFUNCTION)   
\
- : (glthread_once_singlethreaded (ONCE_CONTROL) ? (INITFUNCTION (), 0) : 
0))
+# if PTHREAD_IN_USE_DETECTION_HARD || USE_POSIX_THREADS_WEAK
+#  define glthread_once(ONCE_CONTROL, INITFUNCTION) \
+ (pthread_in_use ()
\
+  ? pthread_once (ONCE_CONTROL, INITFUNCTION)  
\
+  : (glthread_once_singlethreaded (ONCE_CONTROL) ? (INITFUNCTION (), 0) : 
0))
+# else
+#  define glthread_once(ONCE_CONTROL, INITFUNCTION) \
+ (pthread_in_use ()
\
+  ? glthread_once_multithreaded (ONCE_CONTROL, INITFUNCTION)   
\
+  : (glthread_once_singlethreaded (ONCE_CONTROL) ? (INITFUNCTION (), 0) : 
0))
+extern int glthread_once_multithreaded (pthread_once_t *once_control,
+void (*init_function) (void));
+# endif
 extern int glthread_once_singlethreaded (pthread_once_t *once_control);
 
 # ifdef __cplusplus

lib/unistr/u8-uctomb.c fails to compile

[Andreas Schwab]

../../grep/lib/unistr/u8-uctomb.c: In function 'u8_uctomb':
../../grep/lib/unistr/u8-uctomb.c:64:65: error: this statement may fall through 
[-Werror=implicit-fallthrough=]
 case 4: s[3] = 0x80 | (uc & 0x3f); uc = uc >> 6; uc |= 0x1;
  ~~~^~
../../grep/lib/unistr/u8-uctomb.c:65:13: note: here
 case 3: s[2] = 0x80 | (uc & 0x3f); uc = uc >> 6; uc |= 0x800;
 ^~~~
../../grep/lib/unistr/u8-uctomb.c:65:65: error: this statement may fall through 
[-Werror=implicit-fallthrough=]
 case 3: s[2] = 0x80 | (uc & 0x3f); uc = uc >> 6; uc |= 0x800;
  ~~~^~~~
../../grep/lib/unistr/u8-uctomb.c:66:13: note: here
 case 2: s[1] = 0x80 | (uc & 0x3f); uc = uc >> 6; uc |= 0xc0;
 ^~~~
cc1: all warnings being treated as errors

Andreas.

-- 
Andreas Schwab, SUSE Labs, sch...@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

date -d '...' should support superset of POSIX 'at' dates

[Eric Blake]

On today's Austin Group meeting, the question of whether some form of 
'date -d "..."' should be specified.  While there is no way that POSIX 
would standardize everything that GNU date can parse, there was an idea 
that supporting '@seconds' since Epoch, plus all of the formats already 
required to be supported by the 'at' utility 
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/at.html might 
be acceptable.


But that would imply that we want gnulib's date parser to be a superset 
of the grammar required by 'at'.  We are already most of the way there; 
required support dates such as '2pm + 1 week', '2pm next week', and 
'tomorrow' work; but we are lacking at least 'midnight'.  Thus, I'm 
opening this bug to remind us to add the remaining few items to our date 
grammar, so that we can parse everything 'at' is required to support, 
and in case POSIX does decide to standardize -d using the 'at' grammar 
as a starting point.


--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: FreeBSD 11.2: glthread build failure

[Tim Rühsen]

Hi Bruno,

thanks for your work !

I can confirm that your patch fixes the issue here.

On 1/20/20 4:01 AM, Bruno Haible wrote:
>>> FreeBSD 11.2-RELEASE-p4 (with clang 6.0):
>>>
>>>   CCLD wget2
> 
> Please make it a habit to use "make V=1" when submitting a report.
> I don't want to see "CCLD wget2"; I want to see the actual link
> command line.

I don't have direct console/shell access - the issue is/was on a FreeBSD
instance running somewhere that we use for testing via Gitlab CI.
Despite from just having the output from that CI runner, it wasn't
obvious in that moment that 'make V=1' would be helpful.

Please let me know if something is missing in a report before you waste
time and i try to get that piece of info somehow.

Regards, Tim



signature.asc
Description: OpenPGP digital signature