Launchpad has imported 22 comments from the remote bug at http://sourceware.org/bugzilla/show_bug.cgi?id=16159.
If you reply to an imported comment from within Launchpad, your comment will be sent to the remote bug automatically. Read more about Launchpad's inter-bugtracker facilities at https://help.launchpad.net/InterBugTracking. ------------------------------------------------------------------------ On 2013-11-13T03:30:35+00:00 Darryl L. Miles wrote: malloc_printerr() on error detection "free(): invalid next size (fast)" ends up calling into: backtrace.c:init() dl-libc.c:do_dlopen() malloc.c:calloc() malloc.c:malloc_printerr() The malloc error reporting should only report the first error, not attempt to recusively report all error (we knew it was corrupted at the outer most point, so any further work inside malloc is also likely to find corruption). Full stack trace to follow. The main problem is the process does not abort() and die, it hangs around in: pthread_once.S:pthread_one() backtrace.c:__backtrace() I think due to recursive lock, this lock should trylock() on the 2nd time and abort() the process immediately. It does appear to deadlock itself. Reply at: https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1266492/comments/0 ------------------------------------------------------------------------ On 2013-11-13T03:31:30+00:00 Darryl L. Miles wrote: (gdb) bt #0 pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:95 #1 0x00007f8dfb540994 in __backtrace (array=<value optimized out>, size=64) at ../sysdeps/ia64/backtrace.c:85 #2 0x00007f8dfb4b280b in __libc_message (do_abort=2, fmt=0x7f8dfb599fc0 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:178 #3 0x00007f8dfb4b8126 in malloc_printerr (action=3, str=0x7f8dfb5980f9 "malloc(): memory corruption", ptr=<value optimized out>) at malloc.c:6311 #4 0x00007f8dfb4bbba4 in _int_malloc (av=0x7f8dfb7d0e80, bytes=<value optimized out>) at malloc.c:4411 #5 0x00007f8dfb4bc5e6 in __libc_calloc (n=<value optimized out>, elem_size=<value optimized out>) at malloc.c:4075 #6 0x00007f8dfda14d1f in _dl_new_object (realname=0x247de20 "/lib64/libgcc_s.so.1", libname=0x7f8dfb596e3e "libgcc_s.so.1", type=2, loader=0x0, mode=-1879048191, nsid=0) at dl-object.c:77 #7 0x00007f8dfda111ae in _dl_map_object_from_fd (name=0x7f8dfb596e3e "libgcc_s.so.1", fd=6, fbp=0x7fffd2c1ace0, realname=0x247de20 "/lib64/libgcc_s.so.1", loader=0x0, l_type=2, mode=-1879048191, stack_endp=0x7fffd2c1b028, nsid=0) at dl-load.c:975 #8 0x00007f8dfda1236a in _dl_map_object (loader=0x0, name=0x7f8dfb596e3e "libgcc_s.so.1", type=2, trace_mode=0, mode=<value optimized out>, nsid=<value optimized out>) at dl-load.c:2274 #9 0x00007f8dfda1ca34 in dl_open_worker (a=0x7fffd2c1b250) at dl-open.c:227 #10 0x00007f8dfda181a6 in _dl_catch_error (objname=0x7fffd2c1b2a0, errstring=0x7fffd2c1b298, mallocedp=0x7fffd2c1b2af, operate=0x7f8dfda1c910 <dl_open_worker>, args=0x7fffd2c1b250) at dl-error.c:178 #11 0x00007f8dfda1c4ea in _dl_open (file=0x7f8dfb596e3e "libgcc_s.so.1", mode=-2147483647, caller_dlopen=0x0, nsid=-2, argc=8, argv=<value optimized out>, env=0x7fffd2c30020) at dl-open.c:569 #12 0x00007f8dfb568340 in do_dlopen (ptr=<value optimized out>) at dl-libc.c:86 #13 0x00007f8dfda181a6 in _dl_catch_error (objname=0x7fffd2c1b460, errstring=0x7fffd2c1b458, mallocedp=0x7fffd2c1b46f, operate=0x7f8dfb568300 <do_dlopen>, args=0x7fffd2c1b440) at dl-error.c:178 #14 0x00007f8dfb568497 in dlerror_run (name=<value optimized out>, mode=<value optimized out>) at dl-libc.c:47 #15 __libc_dlopen_mode (name=<value optimized out>, mode=<value optimized out>) at dl-libc.c:160 #16 0x00007f8dfb540895 in init () at ../sysdeps/ia64/backtrace.c:41 #17 0x00007f8dfb7e1b23 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:104 #18 0x00007f8dfb540994 in __backtrace (array=<value optimized out>, size=64) at ../sysdeps/ia64/backtrace.c:85 #19 0x00007f8dfb4b280b in __libc_message (do_abort=2, fmt=0x7f8dfb599fc0 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:178 #20 0x00007f8dfb4b8126 in malloc_printerr (action=3, str=0x7f8dfb59a2b8 "free(): invalid next size (fast)", ptr=<value optimized out>) at malloc.c:6311 #21 0x00007f8dfb4bac53 in _int_free (av=0x7f8dfb7d0e80, p=0x24d52c0, have_lock=0) at malloc.c:4811 Reply at: https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1266492/comments/1 ------------------------------------------------------------------------ On 2013-11-13T03:37:53+00:00 Darryl L. Miles wrote: See also bug#956 Reply at: https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1266492/comments/2 ------------------------------------------------------------------------ On 2013-11-13T03:44:48+00:00 Carlos-0 wrote: *** Bug 956 has been marked as a duplicate of this bug. *** Reply at: https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1266492/comments/3 ------------------------------------------------------------------------ On 2013-11-13T03:57:02+00:00 Carlos-0 wrote: This is going to be difficult to fix and invasive. I will do my best to explain why. At the point of the failure we want to be able to print a backtrace. The only way to get a reliable backtrace is to use the unwinder provided by gcc via libgcc_s.so.1 (this may vary by machine). In order to get access to the unwinder we must dlopen that shared library. During the dlopen process we need to calloc enough structures to hookup the new shared library into the structures used by the dynamic linker. One resolution to this problem is to ensure that malloc has a fall-back allocation scheme that is robust against failure and then during the malloc_printerr we flip an internal bit and switch to the temporary reserve allocations. We could also create a new internal API for using the temporary allocations and then dlopen could use that in the event that we are crashing and need to dlopen one last library (the unwinder on demand). That would prevent other threads from consuming the reserve allocations after malloc_printerr is entered by another thread. This is a considerable amount of work and we aren't going to get to this issue until a core developer or someone with serious interest commits to fixing this. Therefore I'm moving this to SUSPENDED until we find the resources to fix the issue. This issue should remain open and new issues submited about this bug should be marked as duplicates of this issue. Reply at: https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1266492/comments/4 ------------------------------------------------------------------------ On 2013-11-13T07:57:31+00:00 Neleai wrote: On Wed, Nov 13, 2013 at 03:57:02AM +0000, carlos at redhat dot com wrote: > One resolution to this problem is to ensure that malloc has a fall-back > allocation scheme that is robust against failure and then during the > malloc_printerr we flip an internal bit and switch to the temporary reserve > allocations. We could also create a new internal API for using the temporary > allocations and then dlopen could use that in the event that we are crashing > and need to dlopen one last library (the unwinder on demand). That would > prevent other threads from consuming the reserve allocations after > malloc_printerr is entered by another thread. > > This is a considerable amount of work and we aren't going to get to this issue > until a core developer or someone with serious interest commits to fixing > this. > Therefore I'm moving this to SUSPENDED until we find the resources to fix the > issue. > Why not reuse a singal-safe malloc for dlopen? Reply at: https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1266492/comments/5 ------------------------------------------------------------------------ On 2013-11-13T13:00:06+00:00 Darryl L. Miles wrote: This fancy backtrace stuff is nice and all but... the process must die! Can't the pthread_once use a non-blocking lock ? Can the lock be a recursive type ? Can pthread_trylock() used in this non-critial path ? if already locked, and if possible to check if locked by our thread-id ? then we immediately abort the process (causing execution of the process to die, like it should). No backtrace is emitted, great! How do I stop this fancy backtrace stuff from working ? I want to setup an environment variable to turn it off as a workaround ? How do I make this fancy backtrace stuff work, by preloading the dlopen() stuff it might need, during initialization of malloc() ? I want to setup an environment variable for that too. There is no need to actually fix the bug, you are over thinking the issue. But this fancy stuff needs to be turned off or preloaded, before the process gets into an undefined state (due to memory bug). Reply at: https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1266492/comments/6 ------------------------------------------------------------------------ On 2013-11-13T13:11:39+00:00 Darryl L. Miles wrote: Another idea, do not backtrace() every malloc() error, only the first one (the outer most one). But right now the process deadlocks itself, on what looks to be a non-recursive mutex trying to do fancy backtrace on every malloc() problem found. The process must die. Reply at: https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1266492/comments/7 ------------------------------------------------------------------------ On 2013-11-13T14:31:28+00:00 Neleai wrote: On Wed, Nov 13, 2013 at 01:00:06PM +0000, darryl.miles at darrylmiles dot org wrote: > How do I stop this fancy backtrace stuff from working ? I want to setup an > environment variable to turn it off as a workaround ? > > How do I make this fancy backtrace stuff work, by preloading the dlopen() > stuff > it might need, during initialization of malloc() ? I want to setup an > environment variable for that too. > As a quick workaround you can add following code to your application/preload this. #include <execinfo.h> static void __attribute__ ((constructor)) init_backtrace() { void *bt[10]; backtrace (bt, 10); } Reply at: https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1266492/comments/8 ------------------------------------------------------------------------ On 2013-11-13T15:50:44+00:00 Bugdal wrote: Carlos, this is yet another reason why dlopen'ing libgcc_s is simply the wrong thing to do, and libgcc_eh should be static-linked into libc. (The other big reason is the possibility of pthread_cancel aborting the program.) At one time in the distant past, it was necessary for there to only be one copy of this code (and its data) in the whole program; otherwise, exception propagation (or backtracing) across DSOs would not work reliably. But modern unwinding code uses dl_iterate_phdr and works fine even if multiple copies of the code are present in the program. Fixing this error in the way I describe will greatly simplify glibc and improve its reliability. Reply at: https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1266492/comments/9 ------------------------------------------------------------------------ On 2013-11-13T16:03:05+00:00 Carlos-0 wrote: (In reply to Rich Felker from comment #9) > Carlos, this is yet another reason why dlopen'ing libgcc_s is simply the > wrong thing to do, and libgcc_eh should be static-linked into libc. (The > other big reason is the possibility of pthread_cancel aborting the program.) > At one time in the distant past, it was necessary for there to only be one > copy of this code (and its data) in the whole program; otherwise, exception > propagation (or backtracing) across DSOs would not work reliably. But modern > unwinding code uses dl_iterate_phdr and works fine even if multiple copies > of the code are present in the program. > > Fixing this error in the way I describe will greatly simplify glibc and > improve its reliability. That sounds like a good idea to me, I also agree that dlopening libgcc_s.so.1 always seemed like a terrible idea to me. We just need the resources to do the rewrite and fixup the linking to use libgcc_eh. I will leave this SUSPENDED until we find someone to clean this up. Reply at: https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1266492/comments/10 ------------------------------------------------------------------------ On 2013-11-13T16:12:53+00:00 Joseph-codesourcery wrote: On Wed, 13 Nov 2013, bugdal at aerifal dot cx wrote: > Carlos, this is yet another reason why dlopen'ing libgcc_s is simply the wrong > thing to do, and libgcc_eh should be static-linked into libc. (The other big Static-linking libgcc_eh into any glibc library is a bad idea because it complicates bootstrapping: it means glibc built with an initial bootstrap compiler (which was built without glibc headers available, implying full EH functionality is not present in libgcc) is not identical to glibc built with a compiler built using full shared glibc and headers. (It's *also* a bad idea because new compilers can start using new DWARF unwind opcodes that an old copy of the unwind code won't understand, causing problems using new programs with old glibc.) The answer for libpthread is for it to dlopen libgcc_s when loaded rather than at pthread_cancel time (or to be made to depend (DT_NEEDED) on libgcc_s in a way that doesn't require libgcc_s to be available when libpthread is built). The answer for other cases is to disable the backtracing by default as discussed in bug 12189 (possibly with an environment variable, not available in setuid programs, that can reenable it - in which case glibc would dlopen libgcc_s at startup). Reply at: https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1266492/comments/11 ------------------------------------------------------------------------ On 2013-11-13T16:23:09+00:00 Neleai wrote: On Wed, Nov 13, 2013 at 04:12:53PM +0000, joseph at codesourcery dot com wrote: > http://sourceware.org/bugzilla/show_bug.cgi?id=16159 > > --- Comment #11 from joseph at codesourcery dot com <joseph at codesourcery > dot com> --- > On Wed, 13 Nov 2013, bugdal at aerifal dot cx wrote: > > > Carlos, this is yet another reason why dlopen'ing libgcc_s is simply the > > wrong > > thing to do, and libgcc_eh should be static-linked into libc. (The other big > > Static-linking libgcc_eh into any glibc library is a bad idea because it > complicates bootstrapping: it means glibc built with an initial bootstrap > compiler (which was built without glibc headers available, implying full > EH functionality is not present in libgcc) is not identical to glibc built > with a compiler built using full shared glibc and headers. (It's *also* a > bad idea because new compilers can start using new DWARF unwind opcodes > that an old copy of the unwind code won't understand, causing problems > using new programs with old glibc.) > Why did you jump from dlopening to static linking? Dynamic linking would work and if there is concern that user does not have one we could provide a stub implementation and function to test if we deal with stub or real one. Reply at: https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1266492/comments/12 ------------------------------------------------------------------------ On 2013-11-13T16:28:31+00:00 Bugdal wrote: Joseph, the bootstrapping issue can presumably be fixed (and bootstrapping made easier) simply by providing a way to install headers without building glibc. This may even allow you to shave one or more steps off of the full bootstrap process. As for the issue of new DWARF opcodes, if they prevent older unwind code from being able to interpret the unwind information at all (rather than just failing to take advantage of the new features) that seems like a fundamental design bug elsewhere that should be reported. I'm not clear whether or not that's really the case. With that said, I find your alternate fix proposal acceptable. For the libpthread issue, I believe the DT_NEEDED could be generated at build time using a fake libgcc_s.so.1 in the glibc source tree. As for disabling backtrace by default, that's perfectly acceptable. Alternatively, glibc could always attempt to load libgcc_s.so.1 at startup and disable backtrace if it's not found. Reply at: https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1266492/comments/13 ------------------------------------------------------------------------ On 2013-11-13T16:29:47+00:00 Bugdal wrote: Ondrej, is having glibc contain a DT_NEEDED entry for libgcc.so.1 really an option that's on the table? I think this would also interfere with bootstrapping issues Joseph and others may be concerned about, as well as hurting load-time performance. Reply at: https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1266492/comments/14 ------------------------------------------------------------------------ On 2013-11-13T16:47:09+00:00 Joseph-codesourcery wrote: On Wed, 13 Nov 2013, neleai at seznam dot cz wrote: > Why did you jump from dlopening to static linking? Dynamic linking would > work and if there is concern that user does not have one we could > provide a stub implementation and function to test if we deal with stub > or real one. I don't think default dlopening libgcc_s from libc at startup is desirable on performance grounds (most programs will never need it), whereas from libpthread it's likely to be less significant. Reply at: https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1266492/comments/15 ------------------------------------------------------------------------ On 2013-11-13T16:53:35+00:00 Joseph-codesourcery wrote: On Wed, 13 Nov 2013, bugdal at aerifal dot cx wrote: > Joseph, the bootstrapping issue can presumably be fixed (and bootstrapping > made > easier) simply by providing a way to install headers without building glibc. There already is. But to install the correct set of headers (some generated at build time) you first need an appropriately configured compiler to configure glibc. That's the old three-compiler bootstrap process: first build a basic compiler, then install headers with it and crt*.o and build a dummy libc.so, then build a second compiler with shared libgcc, then build glibc, then build a third compiler. I changed things in glibc and GCC so that a two-compiler process suffices: the initial compiler built without headers can build glibc and the result is identical to what you get if you repeatedly alternate GCC and glibc builds. (Ideally you'd have a one-compiler process, where the second compiler build only builds/rebuilds GCC's runtime libraries where they depend on system headers or shared glibc, not GCC itself.) Reply at: https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1266492/comments/16 ------------------------------------------------------------------------ On 2013-11-14T14:32:12+00:00 Neleai wrote: Joseph, do you have a benchmark to measure libgcc overhead? I tried a following cat "int main() { return 42; }" > x.c gcc x.c -O3 -o nogcc gcc x.c -O3 -lgcc -o withgcc time for I in `seq 1 10000`; do ./nogcc; done time for I in `seq 1 10000`; do ./withgcc; done And I cannot distinguish these from noise. When I linked with -lpthread there was a noticable slowdown. Reply at: https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1266492/comments/17 ------------------------------------------------------------------------ On 2013-11-14T15:54:30+00:00 Bugdal wrote: Ondrej, did you even check your results with readelf or ldd? -lgcc is a static library and is always linked, so of course it won't make any difference. You need to test with -lgcc_s (and double-check to make sure the dependency really got added). BTW, I'm not sure how well your test will do measuring exec time versus other overhead. If you'd like, I have a test I can post that execs itself and measures the actual time from just before the execve syscall to the start of main. Reply at: https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1266492/comments/18 ------------------------------------------------------------------------ On 2013-11-14T16:47:48+00:00 Neleai wrote: On Thu, Nov 14, 2013 at 03:54:30PM +0000, bugdal at aerifal dot cx wrote: > https://sourceware.org/bugzilla/show_bug.cgi?id=16159 > > --- Comment #18 from Rich Felker <bugdal at aerifal dot cx> --- > Ondrej, did you even check your results with readelf or ldd? -lgcc is a static > library and is always linked, so of course it won't make any difference. You > need to test with -lgcc_s (and double-check to make sure the dependency really > got added). > I asked for benchmark because of that, with a lgcc_s there is difference. plain real 0m3.039s user 0m0.195s sys 0m3.049s with lgcc_s real 0m3.141s user 0m0.169s sys 0m3.179s with lpthread real 0m3.282s user 0m0.182s sys 0m3.308s > BTW, I'm not sure how well your test will do measuring exec time versus other > overhead. If you'd like, I have a test I can post that execs itself and > measures the actual time from just before the execve syscall to the start of > main. > These also count as I wanted to show a relative performance impact. If this is taken into extreme we could improve performance by staticaly linking lm and lpthread Or using prelink. Reply at: https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1266492/comments/19 ------------------------------------------------------------------------ On 2013-11-14T17:08:47+00:00 Bugdal wrote: On Thu, Nov 14, 2013 at 04:47:48PM +0000, neleai at seznam dot cz wrote: > These also count as I wanted to show a relative performance impact. If I agree this approach makes sense, but the relative performance impact could change when the program (possibly linked with libgcc_s) is invoked via posix_spawn or vfork+exec from a high-load server versus as part of an inefficient shell script where the shell may have a lot of additional syscall overhead on each command (this might also vary between shells; dash or busybox ash might perform very differently from bash). So while we may not care about the most extreme impact, I think it's important to consider how large the relative overhead is when the invocation conditions are a low-overhead, real-world scenario. > this is taken into extreme we could improve performance by staticaly linking > lm > and lpthread Yes, of course -- actually, I would recommend merging all of the glibc .so's into libc.so, but I understand that the current situation with symbol versions greatly complicates this, and that there might be other issues. It would certainly improve load-time performance and memory overhead for small programs, though. But I think this is outside the scope of this bug report. The interest in looking at performance here is asking whether a proposed change would make performance noticably worse (a regression), not how we can best optimize startup performance. Reply at: https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1266492/comments/20 ------------------------------------------------------------------------ On 2013-11-28T13:52:17+00:00 Eric Blake wrote: (In reply to Darryl Miles from comment #6) > How do I stop this fancy backtrace stuff from working ? I want to setup an > environment variable to turn it off as a workaround ? According to: https://lists.gnu.org/archive/html/bug-gnulib/2013-11/msg00103.html setting MALLOC_CHECK_=2 in the environment is sufficient to prevent the error message attempts; but that sounds like something you set at program start rather than something we can do via setenv() at the time of reporting the first error (because setenv uses malloc). > There is no need to actually fix the bug, you are over thinking the issue. Yes, there IS a need to fix something. The link above points to a case of a user that is unhappy that their ./configure failed because the conftest program hung after tickling a malloc corruption bug in regex. Configure should never hang (thankfully, configure tests are one case where the MALLOC_CHECK_=2 trick may be sufficient - someone probing for known glibc bugs doesn't care about a bactrace, only about successful exit status). Reply at: https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1266492/comments/21 ** Changed in: eglibc Status: Unknown => Incomplete ** Changed in: eglibc Importance: Unknown => Medium -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to evolution-data-server in Ubuntu. https://bugs.launchpad.net/bugs/1266492 Title: ld:i386 crashes with -static -fPIE -pie Status in Embedded GLIBC: Incomplete Status in “binutils” package in Ubuntu: Confirmed Status in “eglibc” package in Ubuntu: New Status in “evolution-data-server” package in Ubuntu: Triaged Status in “xorg-server” package in Ubuntu: Triaged Status in “binutils” source package in Trusty: Confirmed Status in “eglibc” source package in Trusty: New Status in “evolution-data-server” source package in Trusty: Triaged Status in “xorg-server” source package in Trusty: Triaged Bug description: Making a simple file conftest.c with the following contents: int main() { return 0; } And then compiling it on i386 with gcc -fPIE -pie -static conftest.c returns: *** Error in `/usr/bin/ld': corrupted double-linked list: 0x08dddb38 *** This breaks compilation xorg-server on i386. I believe that -static -fPIE -pie is probably invalid, and it fails on amd64 too. $ gcc -fPIE -pie -static conftest.c /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbeginT.o: relocation R_X86_64_32 against `__TMC_END__' can not be used when making a shared object; recompile with -fPIC /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbeginT.o: error adding symbols: Bad value collect2: error: ld returned 1 exit status But autoconf hangs on the corrupted double-linked list, which times out the xorg-server build. To manage notifications about this bug go to: https://bugs.launchpad.net/eglibc/+bug/1266492/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
