> On 01.04.2018 16:53, Havard Eidnes wrote: >> And some of the internal functions in libexecinfo are apparently >> static, so not present in the symbol table for display in the >> debugger, making debugging all that much harder. >> >> Sigh! >> >> Hints, anyone? > > There is an internal LLVM support for unwinding backtrace with an > attempt to print a meaningful information on a crash signal.
Right... > I assume that there is a crash in the unwinder code causing recursive > execution of a signal handler. :( Would not that cause the stack to overflow? > There was also a post-6.0 patch: > > Fix llvm-config --system-libs output on FreeBSD and NetBSD > > https://github.com/llvm-mirror/llvm/commit/daf294622383687adc281dd695acf4533caf0357 > > Not sure if it is of any help, but it's worth to backport it to 6.0. I applied that locally to the 5.0.1 package I'm building from. I also updated netbsd-8 source, rebuilt kernel and user-land and installed them, then rebuilt llvm, and tried to re-run the tests, and the pointers to which library is what has changed a little (and are probably more accurate now). Included in the update was, I think, an update of gcc from 5.4.0 to 5.5.0. Yes, my 8.0_BETA snapshot was from the middle of last year... (gdb) where #0 0xfbe5a96c in memcpy () from /usr/lib/libc.so.12 #1 0xfbed46a0 in ?? () from /usr/lib/libgcc_s.so.1 #2 0xfbed4cfc in ?? () from /usr/lib/libgcc_s.so.1 #3 0xfbed5bc8 in _Unwind_Backtrace () from /usr/lib/libgcc_s.so.1 #4 0xfc0d0c1c in backtrace () from /usr/lib/libexecinfo.so.0 #5 0xfc56d8d8 in llvm::sys::PrintStackTrace(llvm::raw_ostream&) () from /usr/pkgsrc/lang/llvm/work/build/lib/libLLVM-5.0.so #6 0xfc56dc60 in PrintStackTraceSignalHandler(void*) () from /usr/pkgsrc/lang/llvm/work/build/lib/libLLVM-5.0.so #7 0xfc56bacc in llvm::sys::RunSignalHandlers() () from /usr/pkgsrc/lang/llvm/work/build/lib/libLLVM-5.0.so #8 0xfc56bd4c in SignalHandler(int) () from /usr/pkgsrc/lang/llvm/work/build/lib/libLLVM-5.0.so #9 0xfbdda03c in opendir () from /usr/lib/libc.so.12 Backtrace stopped: frame did not save the PC (gdb) How can I see if it repeatedly hits a signal? I tried: (gdb) b SignalHandler Breakpoint 1 at 0xfc56bb80 (gdb) c Continuing. and ... nothing. So I interrupted and tried some single-stepping: (gdb) si 0xfbed474c in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4750 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4754 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4758 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed475c in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4760 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4764 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed47ac in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed47b0 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed47b4 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed47b8 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed47bc in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4744 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4748 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed474c in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4750 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4754 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4758 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed475c in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4760 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4764 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed47ac in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed47b0 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed47b4 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed47b8 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed47bc in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4744 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4748 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed474c in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4750 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4754 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4758 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed475c in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4760 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4764 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed47ac in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed47b0 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed47b4 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed47b8 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed47bc in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4744 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4748 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed474c in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4750 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4754 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4758 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed475c in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4760 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4764 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed47ac in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed47b0 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed47b4 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed47b8 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed47bc in ?? () from /usr/lib/libgcc_s.so.1 (gdb) 0xfbed4744 in ?? () from /usr/lib/libgcc_s.so.1 (gdb) x/i 0xfbed4744 => 0xfbed4744: lwz r9,4(r27) (gdb) x/20i 0xfbed4748: cmplwi cr7,r9,5 0xfbed474c: bgt cr7,0xfbed47ac 0xfbed4750: lwz r10,-32748(r30) 0xfbed4754: rlwinm r9,r9,2,0,29 0xfbed4758: lwzx r9,r10,r9 0xfbed475c: add r10,r9,r10 0xfbed4760: mtctr r10 0xfbed4764: bctr 0xfbed4768: lwz r9,4(r23) 0xfbed476c: cmpwi cr7,r9,0 0xfbed4770: bne cr7,0xfbed46b8 0xfbed4774: andis. r9,r10,16384 0xfbed4778: lwz r8,584(r22) 0xfbed477c: stw r8,776(r1) 0xfbed4780: beq 0xfbed478c 0xfbed4784: li r8,0 0xfbed4788: stb r8,621(r23) 0xfbed478c: addi r9,r1,776 0xfbed4790: stw r9,4(r23) 0xfbed4794: b 0xfbed46b8 (gdb) 0xfbed4798: bl 0xfbedae00 <00008000.got2.plt_pic32.abort+176> 0xfbed479c: lwz r9,0(r27) 0xfbed47a0: stb r31,0(r26) 0xfbed47a4: add r9,r21,r9 0xfbed47a8: stw r9,0(r25) 0xfbed47ac: addi r26,r26,1 0xfbed47b0: addi r27,r27,8 0xfbed47b4: cmplw cr7,r24,r26 0xfbed47b8: addi r25,r25,4 0xfbed47bc: bne cr7,0xfbed4744 0xfbed47c0: lbz r9,1211(r28) 0xfbed47c4: cmpwi cr7,r9,0 0xfbed47c8: lwz r9,608(r22) 0xfbed47cc: bne cr7,0xfbed49ac 0xfbed47d0: lwz r0,852(r1) 0xfbed47d4: clrlwi r9,r9,1 0xfbed47d8: stw r9,608(r22) 0xfbed47dc: mtlr r0 0xfbed47e0: lwz r21,804(r1) 0xfbed47e4: lwz r22,808(r1) (gdb) (gdb) i regi r0 0xfbed4cfc 4226632956 r1 0xfbb6ee00 4223069696 r2 0xfdf02508 4260373768 r3 0xfbb6ee08 4223069704 r4 0xfbb6f934 4223072564 r5 0x0 0 r6 0x0 0 r7 0x0 0 r8 0x4 4 r9 0xffff9084 4294938756 r10 0xfbed47ac 4226631596 r11 0xfbe5a940 4226132288 r12 0xfdefd000 4260352000 r13 0x1aec8b0 28231856 r14 0x0 0 r15 0x0 0 r16 0x0 0 r17 0x0 0 r18 0x162 354 r19 0xffffe79c 4294961052 r20 0xfbb36168 4222837096 r21 0xfbb6fb80 4223073152 r22 0xfbb6f638 4223071800 r23 0xfbb6ee08 4223069704 r24 0xfbb6f936 4223072566 r25 0xfbb6f800 4223072256 r26 0xfbb6f916 4223072534 r27 0xfbb6f508 4223071496 r28 0xfbb6f178 4223070584 r29 0x0 0 r30 0xfbef5698 4226766488 r31 0x1 1 pc 0xfbed4744 0xfbed4744 msr <unavailable> cr 0x44044884 1141131396 lr 0xfbed46a0 0xfbed46a0 ctr 0xfbed47ac 4226631596 xer 0x0 0 (gdb) This, based on a repetition count of 3 (yes, real scientific conlusion! :) looks suspciously like an infinite loop, at least that appears to be the behaviour. Deciphering this appears to delve into the fringes of gcc, inside libgcc_s.so which appears to come from there. I've not been able to locate the source code corresponding to the above disassembly, and the hidden symbols is frustrating (probably static functions), the library itself claims it's not stripped: ambrosia# file /lib/libgcc_s.so.1.0 /lib/libgcc_s.so.1.0: ELF 32-bit MSB shared object, PowerPC or cisco 4500, version 1 (SYSV), dynamically linked, for NetBSD 8.0, not stripped ambrosia# That's the same size as the one in the obj dir: ambrosia# ls -l /lib/libgcc_s.so.1.0 -r--r--r-- 1 root wheel 67248 Apr 3 12:37 /lib/libgcc_s.so.1.0 ambrosia# find /usr/obj -name libgcc_s.so.1.0 /usr/obj/external/gpl3/gcc/lib/libgcc/libgcc_s/libgcc_s.so.1.0 ambrosia# ls -l /usr/obj/external/gpl3/gcc/lib/libgcc/libgcc_s/libgcc_s.so.1.0 -rwxr-xr-x 1 root wheel 67248 Apr 3 12:37 /usr/obj/external/gpl3/gcc/lib/libgcc/libgcc_s/libgcc_s.so.1.0 ambrosia# Hm, I am suspecting that nobody has actually tested whether backtrace() really works on NetBSD/powerpc... I'll write a simple test of that in C tomorrow. On the other hand, the backtrace gdb was able to provide decidedly looks incomplete -- the program's main function is not opendir() (!), and maybe this has something to do with it? It doesn't look like the SupportTests program is multi-threaded, although it is linked with -lpthread: ambrosia# ldd work/build/unittests/Support/SupportTests work/build/unittests/Support/SupportTests: -lpthread.1 => /usr/lib/libpthread.so.1 -lc.12 => /usr/lib/libc.so.12 -lLLVM-5.0 => /usr/pkg/lib/libLLVM-5.0.so -ledit.3 => /usr/lib/libedit.so.3 -lterminfo.1 => /usr/lib/libterminfo.so.1 -lrt.1 => /usr/lib/librt.so.1 -lexecinfo.0 => /usr/lib/libexecinfo.so.0 -lelf.2 => /usr/lib/libelf.so.2 -lgcc_s.1 => /usr/lib/libgcc_s.so.1 -lz.1 => /usr/lib/libz.so.1 -lstdc++.8 => /usr/lib/libstdc++.so.8 -lm.0 => /usr/lib/libm.so.0 ambrosia# Regards, - HÃ¥vard