Re: llvm self-tests looping(?),Re: llvm self-tests looping(?),Re: llvm self-tests looping(?),Re: llvm self-tests looping(?)

2018-04-09 Thread Havard Eidnes 
>>> I don't see an ATF machine for powerpc, there shall be one available.
>>>
>>> http://releng.netbsd.org/test-results.html
>>
>> Mm, OK, doing the tests on netbsd-8 on this MacMini G4 should be
>> fairly straight-forward.
>
> Not so.  The machine wedged partway through the tests, it's in
> the office and I'm at home...

It looks like one of the net/icmp/t_ping* tests caused the wedge.
I commented them out from Atffile, and the tests now completed.

Regards,

- Håvard

Re: llvm self-tests looping(?),Re: llvm self-tests looping(?),Re: llvm self-tests looping(?),Re: llvm self-tests looping(?)

2018-04-07 Thread Havard Eidnes 
>> I don't see an ATF machine for powerpc, there shall be one available.
>>
>> http://releng.netbsd.org/test-results.html
>
> Mm, OK, doing the tests on netbsd-8 on this MacMini G4 should be
> fairly straight-forward.

Not so.  The machine wedged partway through the tests, it's in
the office and I'm at home...

Regards,

- Håvard

Re: llvm self-tests looping(?),Re: llvm self-tests looping(?),Re: llvm self-tests looping(?),Re: llvm self-tests looping(?)

2018-04-07 Thread Havard Eidnes 
>> Hm, I am suspecting that nobody has actually tested whether
>> backtrace() really works on NetBSD/powerpc...  I'll write a
>> simple test of that in C tomorrow.
>
> Yes, this looks more like dysfunctional backtrace(3).
>
> We have got an ATF test for this:
>
>   tests/lib/libexecinfo/t_backtrace.c
>
> If it will work, it's worth to add a scenario that fails for ppc.

Hmm, that test seems to be working just fine, sigh!

ambrosia: {6} ./t_backtrace backtrace_fmt_basic
t_backtrace: WARNING: Running test cases without atf-run(1) is unsupported
t_backtrace: WARNING: No isolation nor timeout control is being applied; you 
may get unexpected failures; see atf-test-case(4)
got nptrs=19 ncalls=12 (min_frames: 4, max_frames: 9)
backtrace is:
#0: myfunc3
#1: myfunc2
#2: myfunc1
#3: myfunc1
#4: myfunc1
#5: myfunc1
#6: myfunc1
#7: myfunc1
#8: myfunc1
#9: myfunc1
#10: myfunc1
#11: myfunc1
#12: myfunc1
#13: myfunc1
#14: myfunc
#15: atfu_backtrace_fmt_basic_body
#16: atf_tc_run
#17: atf_tp_run
#18: atf_tp_main
passed
ambrosia: {7} pwd
/usr/tests/lib/libexecinfo
ambrosia: {8} 

> I don't see an ATF machine for powerpc, there shall be one available.
>
> http://releng.netbsd.org/test-results.html

Mm, OK, doing the tests on netbsd-8 on this MacMini G4 should be
fairly straight-forward.

>> On the other hand, the backtrace gdb was able to provide
>> decidedly looks incomplete -- the program's main function is not
>> opendir() (!), and maybe this has something to do with it?
>
> This is a bug, it's really a signal trapmpoline. This needs to be fixed
> in GDB.. it's on my TODO list.

OK, good.

>> It doesn't look like the SupportTests program is multi-threaded,
>> although it is linked with -lpthread:
>
> It's common in the LLVM environment to link with everything that could
> be useful.. like libm, librt, libpthread, libdl [for !NetBSD] etc.

Mm, ok.

- Håvard

Re: llvm self-tests looping(?),Re: llvm self-tests looping(?)

2018-04-04 Thread Kamil Rytarowski 
On 05.04.2018 00:55, Havard Eidnes wrote:
> Hm, I am suspecting that nobody has actually tested whether
> backtrace() really works on NetBSD/powerpc...  I'll write a
> simple test of that in C tomorrow.
> 

Yes, this looks more like dysfunctional backtrace(3).

We have got an ATF test for this:

  tests/lib/libexecinfo/t_backtrace.c

If it will work, it's worth to add a scenario that fails for ppc.

I don't see an ATF machine for powerpc, there shall be one available.

http://releng.netbsd.org/test-results.html

> On the other hand, the backtrace gdb was able to provide
> decidedly looks incomplete -- the program's main function is not
> opendir() (!), and maybe this has something to do with it?
> 

This is a bug, it's really a signal trapmpoline. This needs to be fixed
in GDB.. it's on my TODO list.

> It doesn't look like the SupportTests program is multi-threaded,
> although it is linked with -lpthread:
> 

It's common in the LLVM environment to link with everything that could
be useful.. like libm, librt, libpthread, libdl [for !NetBSD] etc.




signature.asc
Description: OpenPGP digital signature

Re: llvm self-tests looping(?),Re: llvm self-tests looping(?)

2018-04-04 Thread Havard Eidnes 
> On 01.04.2018 16:53, Havard Eidnes wrote:
>> And some of the internal functions in libexecinfo are apparently
>> static, so not present in the symbol table for display in the
>> debugger, making debugging all that much harder.
>> 
>> Sigh!
>> 
>> Hints, anyone?
>
> There is an internal LLVM support for unwinding backtrace with an
> attempt to print a meaningful information on a crash signal.

Right...

> I assume that there is a crash in the unwinder code causing recursive
> execution of a signal handler.

:(

Would not that cause the stack to overflow?

> There was also a post-6.0 patch:
>
> Fix llvm-config --system-libs output on FreeBSD and NetBSD
>
> https://github.com/llvm-mirror/llvm/commit/daf294622383687adc281dd695acf4533caf0357
>
> Not sure if it is of any help, but it's worth to backport it to 6.0.

I applied that locally to the 5.0.1 package I'm building from.

I also updated netbsd-8 source, rebuilt kernel and user-land and
installed them, then rebuilt llvm, and tried to re-run the tests,
and the pointers to which library is what has changed a little
(and are probably more accurate now).  Included in the update
was, I think, an update of gcc from 5.4.0 to 5.5.0.  Yes, my
8.0_BETA snapshot was from the middle of last year...

(gdb) where
#0  0xfbe5a96c in memcpy () from /usr/lib/libc.so.12
#1  0xfbed46a0 in ?? () from /usr/lib/libgcc_s.so.1
#2  0xfbed4cfc in ?? () from /usr/lib/libgcc_s.so.1
#3  0xfbed5bc8 in _Unwind_Backtrace () from /usr/lib/libgcc_s.so.1
#4  0xfc0d0c1c in backtrace () from /usr/lib/libexecinfo.so.0
#5  0xfc56d8d8 in llvm::sys::PrintStackTrace(llvm::raw_ostream&) ()
   from /usr/pkgsrc/lang/llvm/work/build/lib/libLLVM-5.0.so
#6  0xfc56dc60 in PrintStackTraceSignalHandler(void*) ()
   from /usr/pkgsrc/lang/llvm/work/build/lib/libLLVM-5.0.so
#7  0xfc56bacc in llvm::sys::RunSignalHandlers() ()
   from /usr/pkgsrc/lang/llvm/work/build/lib/libLLVM-5.0.so
#8  0xfc56bd4c in SignalHandler(int) ()
   from /usr/pkgsrc/lang/llvm/work/build/lib/libLLVM-5.0.so
#9  0xfbdda03c in opendir () from /usr/lib/libc.so.12
Backtrace stopped: frame did not save the PC
(gdb) 

How can I see if it repeatedly hits a signal? I tried:

(gdb) b SignalHandler
Breakpoint 1 at 0xfc56bb80
(gdb) c
Continuing.

and ... nothing.

So I interrupted and tried some single-stepping:

(gdb) si
0xfbed474c in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4750 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4754 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4758 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed475c in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4760 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4764 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed47ac in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed47b0 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed47b4 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed47b8 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed47bc in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4744 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4748 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed474c in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4750 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4754 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4758 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed475c in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4760 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4764 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed47ac in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed47b0 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed47b4 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed47b8 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed47bc in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4744 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4748 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed474c in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4750 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4754 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4758 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed475c in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4760 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4764 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed47ac in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed47b0 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed47b4 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed47b8 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed47bc in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4744 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4748 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed474c in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4750 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4754 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4758 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed475c in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4760 in ?? () from /usr/lib/libgcc_s.so.1
(gdb) 
0xfbed4764 in ?? () from

Re: llvm self-tests looping(?),Re: llvm self-tests looping(?)

2018-04-03 Thread Kamil Rytarowski 
On 03.04.2018 23:36, Havard Eidnes wrote:
>>> And ... as follow-up I thought I'd check whether "make test" in
>>> lang/llvm (5.0.1nb1) works on NetBSD/amd64 8.0_BETA.  And while the
>>> selftest setup seems to work fine on this platform, there are quite a
>>> bit of unexpected failures:
>>>
>>>   Expected Passes: 20309
>>>   Expected Failures  : 130
>>>   Unsupported Tests  : 786
>>>   Unexpected Failures: 211
>>>
>>> and it seems a lot of the tests simply crash.  An example:
>>
>> Most of that fixed in 6.0, MPROTECT violation in JIT code.
> 
> I would have never guessed that as the root cause given the
> diagnostics emitted.  This smells like software which tries to be
> too clever by half.
> 
> Regards,
> 
> - Håvard
> 

All the JIT engines were made compatible with PaX MPROTECT on NetBSD.
There will be few remaining failures in MemoryBlock (or similar class
name) as they are testing W|X protection.

I recall that there were 2 cases where something broke because of
environment setup, like unexpected libstdc++ for clang/NetBSD rather
than hardcoded libc++.

I've not been testing the exact 6.0 release so the exact number of
failures will wary. Probably below 20?



signature.asc
Description: OpenPGP digital signature

Re: llvm self-tests looping(?),Re: llvm self-tests looping(?)

2018-04-03 Thread Havard Eidnes 
>> And ... as follow-up I thought I'd check whether "make test" in
>> lang/llvm (5.0.1nb1) works on NetBSD/amd64 8.0_BETA.  And while the
>> selftest setup seems to work fine on this platform, there are quite a
>> bit of unexpected failures:
>>
>>   Expected Passes: 20309
>>   Expected Failures  : 130
>>   Unsupported Tests  : 786
>>   Unexpected Failures: 211
>>
>> and it seems a lot of the tests simply crash.  An example:
>
> Most of that fixed in 6.0, MPROTECT violation in JIT code.

I would have never guessed that as the root cause given the
diagnostics emitted.  This smells like software which tries to be
too clever by half.

Regards,

- Håvard

Re: llvm self-tests looping(?)

2018-04-03 Thread Kamil Rytarowski 
On 02.04.2018 11:47, Havard Eidnes wrote:
> And ... as follow-up I thought I'd check whether "make test" in
> lang/llvm (5.0.1nb1) works on NetBSD/amd64 8.0_BETA.  And while the
> selftest setup seems to work fine on this platform, there are quite a
> bit of unexpected failures:
> 
>   Expected Passes: 20309
>   Expected Failures  : 130
>   Unsupported Tests  : 786
>   Unexpected Failures: 211
> 
> and it seems a lot of the tests simply crash.  An example:
> 

Most of that fixed in 6.0, MPROTECT violation in JIT code.



signature.asc
Description: OpenPGP digital signature

Re: llvm self-tests looping(?)

2018-04-02 Thread Havard Eidnes 
And ... as follow-up I thought I'd check whether "make test" in
lang/llvm (5.0.1nb1) works on NetBSD/amd64 8.0_BETA.  And while the
selftest setup seems to work fine on this platform, there are quite a
bit of unexpected failures:

  Expected Passes: 20309
  Expected Failures  : 130
  Unsupported Tests  : 786
  Unexpected Failures: 211

and it seems a lot of the tests simply crash.  An example:


Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60
FAIL: LLVM :: ExecutionEngine/test-interp-vec-logical.ll (13657 of 21436)
 TEST 'LLVM :: ExecutionEngine/test-interp-vec-logical.ll' 
FAILED 
Script:
--
/usr/pkgsrc/lang/llvm/work/build/./bin/lli 
/usr/pkgsrc/lang/llvm/work/llvm-5.0.0.src/test/ExecutionEngine/test-interp-vec-logical.ll
 > /dev/null
--
Exit Code: 139

Command Output (stderr):
--
#0 0x7138327c701a llvm::sys::PrintStackTrace(llvm::raw_ostream&) 
(/usr/pkgsrc/lang/llvm/work/build/lib/libLLVM-5.0.so+0x7c701a)
#1 0x7138327c536f llvm::sys::RunSignalHandlers() 
(/usr/pkgsrc/lang/llvm/work/build/lib/libLLVM-5.0.so+0x7c536f)
#2 0x7138327c54b0 SignalHandler(int) 
(/usr/pkgsrc/lang/llvm/work/build/lib/libLLVM-5.0.so+0x7c54b0)
Stack dump:
0.  Program arguments: /usr/pkgsrc/lang/llvm/work/build/./bin/lli 
/usr/pkgsrc/lang/llvm/work/llvm-5.0.0.src/test/ExecutionEngine/test-interp-vec-logical.ll
 
/usr/pkgsrc/lang/llvm/work/build/test/ExecutionEngine/Output/test-interp-vec-logical.ll.script:
 line 1: 18054 Segmentation fault  (core dumped) 
/usr/pkgsrc/lang/llvm/work/build/./bin/lli 
/usr/pkgsrc/lang/llvm/work/llvm-5.0.0.src/test/ExecutionEngine/test-interp-vec-logical.ll
 > /dev/null

However, this doesn't appear to actually give enough information to
zero in on what the actual problem is, sigh!

Isn't the backtracer able to trace through signal delivery events?

I find there's a number of core files left after the selftests have
run:

d3: {10} find work -name '*.core'
work/build/test/MCJITTests.core
work/build/test/OrcJITTests.core
work/build/test/SupportTests.core
work/build/test/BugPoint/opt.core
work/build/test/ExecutionEngine/MCJIT/lli.core
work/build/test/ExecutionEngine/OrcMCJIT/lli.core
work/build/test/ExecutionEngine/lli.core
d3: {11}

but looking at the last one doesn't really give much more useful
information:

d3: {11} gdb -q work/build/./bin/lli
Reading symbols from work/build/./bin/lli...(no debugging symbols found)...done.
(gdb) core-file work/build/test/ExecutionEngine/lli.core
[New process 1]
Core was generated by `lli'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x701d30805000 in main ()
(gdb) x/i 0x701d30805000
=> 0x701d30805000 :   xor%eax,%eax
(gdb) where
#0  0x701d30805000 in main ()
#1  0x701d2e95bf17 in llvm::MCJIT::runFunction(llvm::Function*, 
llvm::ArrayRef) ()
   from /usr/pkgsrc/lang/llvm/work/build/lib/libLLVM-5.0.so
#2  0x701d2e9378ed in 
llvm::ExecutionEngine::runFunctionAsMain(llvm::Function*, 
std::vector const&, char const* 
const*) () from /usr/pkgsrc/lang/llvm/work/build/lib/libLLVM-5.0.so
#3  0x0041cad6 in main ()
(gdb) i reg
rax0x701d2e95be9a   123270637928090
rbx0x7f7fffecd600   140187731285504
rcx0x701d3080b800   123270670104576
rdx0xff4350da   -12365606
rsi0x20 32
rdi0x701d2d1236e0   123270612530912
rbp0x20 0x20
rsp0x7f7fffecd458   0x7f7fffecd458
r8 0x0  0
r9 0x701d2d1151c0   123270612472256
r100x701d30805000   123270670077952
r110x207519
r120x701d30805000   123270670077952
r130x701d2d16c000   123270612828160
r140x701d2d1236e0   123270612530912
r150x701d2c558b00   123270600166144
rip0x701d30805000   0x701d30805000 
eflags 0x10246  [ PF ZF IF RF ]
cs 0x47 71
ss 0x3f 63
ds 0x3f 63
es 0x3f 63
fs 0x0  0
gs 0x0  0
(gdb) 

?!?  SEGV on "xor %eax,%eax"?!?  I don't think so...

Regards,

- Håvard

Re: llvm self-tests looping(?)

2018-04-01 Thread Kamil Rytarowski 
On 01.04.2018 16:53, Havard Eidnes wrote:
> And some of the internal functions in libexecinfo are apparently
> static, so not present in the symbol table for display in the
> debugger, making debugging all that much harder.
> 
> Sigh!
> 
> Hints, anyone?
> 

There is an internal LLVM support for unwinding backtrace with an
attempt to print a meaningful information on a crash signal.

I assume that there is a crash in the unwinder code causing recursive
execution of a signal handler.

There was also a post-6.0 patch:

Fix llvm-config --system-libs output on FreeBSD and NetBSD

https://github.com/llvm-mirror/llvm/commit/daf294622383687adc281dd695acf4533caf0357

Not sure if it is of any help, but it's worth to backport it to 6.0.



signature.asc
Description: OpenPGP digital signature

llvm self-tests looping(?)

2018-04-01 Thread Havard Eidnes 
Hi,

I decided it might be a good idea to run the self-tests in llvm
5.0.1 on powerpc.  However, after the test and utilities are
built, it appears to spin while doing the first test.  The run
log shows:

[100%] Built target LLVMHello_exports
[100%] Built target LLVMHello
Scanning dependencies of target check-llvm
[100%] Running the LLVM regression tests
-- Testing: 21465 tests, 1 threads --
Testing: 0 ..

and the process is spinning consuming CPU (well, it's stopped in
the debugger here):

  PID USERNAME PRI NICE   SIZE   RES STATE  TIME   WCPUCPU COMMAND
0 root 1260 0K   22M pgdaemon 600:47  0.00%  0.00% [system]
19597 root  28443M   15M STOP 306:27  0.00%  0.00% SupportTests

and the debugger points towards libexecinfo:

ambrosia# gdb -q ./work/build/unittests/Support/SupportTests
Reading symbols from ./work/build/unittests/Support/SupportTests...(no 
debugging symbols found)...done.
(gdb) attach 19597
Attaching to program: 
/usr/pkgsrc/lang/llvm/work/build/unittests/Support/SupportTests, process 19597
Couldn't get registers: Device busy.
Couldn't get registers: Device busy.
(gdb) Reading symbols from /usr/lib/libpthread.so.1...(no debugging symbols 
found)...done.
Reading symbols from /usr/pkgsrc/lang/llvm/work/build/lib/libLLVM-5.0.so...(no 
debugging symbols found)...done.
Reading symbols from /usr/lib/librt.so.1...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libexecinfo.so.0...(no debugging symbols 
found)...done.
Reading symbols from /usr/lib/libterminfo.so.1...(no debugging symbols 
found)...done.
Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libstdc++.so.7...(no debugging symbols 
found)...done.
Reading symbols from /usr/lib/libm.so.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libgcc_s.so.1...(no debugging symbols 
found)...done.
Reading symbols from /usr/lib/libc.so.12...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libedit.so.3...(no debugging symbols 
found)...done.
Reading symbols from /usr/lib/libelf.so.2...(no debugging symbols found)...done.
Reading symbols from /usr/libexec/ld.elf_so...(no debugging symbols 
found)...done.
[Switching to LWP 1]
0xfc1d3fcc in ?? () from /usr/lib/libexecinfo.so.0

(gdb) where
#0  0xfc1d3fcc in ?? () from /usr/lib/libexecinfo.so.0
#1  0xfc1d4514 in ?? () from /usr/lib/libexecinfo.so.0
#2  0xfc1d53e0 in _Unwind_Backtrace () from /usr/lib/libexecinfo.so.0
#3  0xfc1d128c in backtrace () from /usr/lib/libexecinfo.so.0
#4  0xfc673ac4 in llvm::sys::PrintStackTrace(llvm::raw_ostream&) ()
   from /usr/pkgsrc/lang/llvm/work/build/lib/libLLVM-5.0.so
#5  0xfc673e4c in PrintStackTraceSignalHandler(void*) ()
   from /usr/pkgsrc/lang/llvm/work/build/lib/libLLVM-5.0.so
#6  0xfc671c08 in llvm::sys::RunSignalHandlers() ()
   from /usr/pkgsrc/lang/llvm/work/build/lib/libLLVM-5.0.so
#7  0xfc671e88 in SignalHandler(int) ()
   from /usr/pkgsrc/lang/llvm/work/build/lib/libLLVM-5.0.so
#8  0xfbe2af58 in opendir () from /usr/lib/libc.so.12
Backtrace stopped: frame did not save the PC
(gdb) i thread
  Id   Target Id Frame 
* 1LWP 1 0xfbeb4c4c in memcpy () from /usr/lib/libc.so.12
(gdb)

and an earlier backtrace of the same process gave

(gdb) where
#0  0xfbeb4c4c in memcpy () from /usr/lib/libc.so.12
#1  0xfc1d2c20 in ?? () from /usr/lib/libexecinfo.so.0
#2  0xfc1d3470 in ?? () from /usr/lib/libexecinfo.so.0
#3  0xfc1d52e8 in _Unwind_Backtrace () from /usr/lib/libexecinfo.so.0
#4  0xfc1d128c in backtrace () from /usr/lib/libexecinfo.so.0
#5  0xfc673ac4 in llvm::sys::PrintStackTrace(llvm::raw_ostream&) ()
   from /usr/pkgsrc/lang/llvm/work/build/lib/libLLVM-5.0.so
#6  0xfc673e4c in PrintStackTraceSignalHandler(void*) ()
   from /usr/pkgsrc/lang/llvm/work/build/lib/libLLVM-5.0.so
#7  0xfc671c08 in llvm::sys::RunSignalHandlers() ()
   from /usr/pkgsrc/lang/llvm/work/build/lib/libLLVM-5.0.so
#8  0xfc671e88 in SignalHandler(int) ()
   from /usr/pkgsrc/lang/llvm/work/build/lib/libLLVM-5.0.so
#9  0xfbe2af58 in opendir () from /usr/lib/libc.so.12
Backtrace stopped: frame did not save the PC
(gdb) 

Setting a breakpoint on the next instruction in backtrace():

(gdb) x/i 0xfc1d128c
   0xfc1d128c :   lwz r3,16(r1)
(gdb) x/i backtrace
   0xfc1d1250 :  stwur1,-48(r1)
(gdb) x/20i
   0xfc1d1254 :mflrr0
   0xfc1d1258 :bcl 20,4*cr7+so,0xfc1d125c 
   0xfc1d125c :   li  r9,-1
   0xfc1d1260 :   stw r30,40(r1)
   0xfc1d1264 :   mflrr30
   0xfc1d1268 :   stw r3,8(r1)
   0xfc1d126c :   stw r4,12(r1)
   0xfc1d1270 :   addir4,r1,8
   0xfc1d1274 :   addis   r30,r30,2
   0xfc1d1278 :   stw r9,16(r1)
   0xfc1d127c :   addir30,r30,-448
   0xfc1d1280 :   stw r0,52(r1)