FWIW, I think I may have figured out what's happening. It's probably something to note in the http://www.graphicsmagick.org/OpenMP.html documentation.
In a nutshell, my understanding is that libgomp keeps track of its thread pool in thread local storage. When that thread local storage happens to be in the main process, all is well, everything works fine, etc. When that thread local storage is in a transient pthread (and, apparently, you're running on a certain class memory/CPU configuration) and your transient thread terminates, that thread local storage doesn't (always) get cleaned up. What this means in practice is making GM API calls from within a pthread can result in undefined behaviour when an API triggers OpenMP calls. I suspect this isn't a particularly desirable situation, as GraphicsMagick appears to have put at least some effort into making the API thread-safe. It likely means that even my mallopt(M_ARENA_MAX,n) fix is still not right, and the only "safe" solution is to disable OpenMP completely in that particular application. I'm going to keep looking for a better workaround/mitigation. It would be ideal if there was a mechanism to keep the thread pool in the main process and still have it used from the subthread. ________________________________ From: Beauregard,Christophe (ECCC) <christophe.beaureg...@ec.gc.ca> Sent: Monday, June 5, 2023 09:20 To: László Böszörményi (GCS) <g...@debian.org>; Bob Friesenhahn <bfrie...@simple.dallas.tx.us>; 1037...@bugs.debian.org <1037...@bugs.debian.org> Subject: Re: Bug#1037042: graphicsmagick: GetImageDepth has a thread arena and memory leak <sigh> Never mind, you can close this. I've managed to reproduce the leak without GM: /* * gcc -o gomparena gomparena.c -fopenmp */ #include <stdio.h> #include <stdlib.h> #include <unistd.h> #include <time.h> #include <pthread.h> static void* go_omp(void* arg) { const int m = 1024; double* a = alloca(m*sizeof(double)); #pragma omp parallel for for( int n = 0; n < m; n ++ ) { a[n] = n*n; } return NULL; } int main ( int argc, char **argv ) { int cycles = 10; while(cycles--) { pid_t myp = getpid(); char s[1024]; #if 1 /* this code path has a thread arena leak */ pthread_t thread_id; pthread_create(&thread_id, NULL, go_omp, NULL); /* pthread_detach(thread_id); also shows the problem */ pthread_join(thread_id, NULL); #else /* this code path doesn't have a thread arena leak */ go_omp(NULL); #endif sleep(1); fprintf(stderr,"%lld\n", (long long)time(0)); snprintf(s,sizeof(s), "ps -o vsz -o rss -o user -o command -p %lld", (long long)myp); system(s); snprintf(s,sizeof(s), "pmap -x %lld | grep 65404 | wc -l", (long long)myp); system(s); } return 0; } ________________________________ From: Beauregard,Christophe (ECCC) <christophe.beaureg...@ec.gc.ca> Sent: Monday, June 5, 2023 08:58 To: László Böszörményi (GCS) <g...@debian.org>; Bob Friesenhahn <bfrie...@simple.dallas.tx.us>; 1037...@bugs.debian.org <1037...@bugs.debian.org> Subject: Re: Bug#1037042: graphicsmagick: GetImageDepth has a thread arena and memory leak >Do you think it might be a problem with another system component, a GCC optimization or this is fixed meanwhile? I'm not sure. It took me most of a week to even properly isolate the problem and find a repeatable test case on that one machine, and even then I feel like it's still a moving target. I've done more poking and it's not 100% exclusive to GetImageDepth(). In fact, I'm seeing the problem now if I comment out the call to GetImageDepth() (but in my original application, ReadImage() without GetImageDepth() in another code path isn't enough to trigger the bug). My working theory is something about how GM uses the OpenMP/libgomp API is tickling a bug in a code path that's only seen with certain CPU/memory configs. The questions I can't answer are (a) is there possibly something incomplete/incorrect in how GM uses OpenMP which could lead to this sort of bug? and (b) just how deep does this rabbit hole go (glibc? kernel?)? I'm starting to think that I might need to peel off a machine from the development cluster and investigate whether I can reproduce the error in a container, at which point I can play around with different configurations of libraries, compilers, etc. c. ________________________________ From: László Böszörményi (GCS) <g...@debian.org> Sent: Sunday, June 4, 2023 06:25 To: Bob Friesenhahn <bfrie...@simple.dallas.tx.us>; 1037...@bugs.debian.org <1037...@bugs.debian.org> Cc: Beauregard,Christophe (ECCC) <christophe.beaureg...@ec.gc.ca> Subject: Re: Bug#1037042: graphicsmagick: GetImageDepth has a thread arena and memory leak [You don't often get email from g...@debian.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] Hi, On Sat, Jun 3, 2023 at 8:30 PM Bob Friesenhahn <bfrie...@simple.dallas.tx.us> wrote: > I am definitely able to confirm that memory consumption builds due to > invoking GetImageDepth() via a POSIX thread. The rate that it builds > is image sensitive since some images cause GetImageDepth() to perform > more OpenMP loops. Unfortunately I can not reproduce. My processor is an Intel K variant CPU, six cores and twelve threads, 64 Gb of RAM. GM is 1.3.40 with two security fixes backported, compiled with GCC v12.2.0. Tried with three PNG images, all memory consumption is static from the beginning. Do I need some special case of PNG files to experience this issue? > My own testing is under Ubuntu 20.04 using GCC 10. Do you think it might be a problem with another system component, a GCC optimization or this is fixed meanwhile? At least I do wonder why this issue is CPU / machine dependent. Regards, Laszlo/GCS