On Fri, Jan 07, 2022 at 08:34:57PM +0530, Utkarsh Gupta wrote:
> Hello,
>
> On Fri, Jan 7, 2022 at 7:56 PM Utkarsh Gupta
> wrote:
> > LP: #1954854 - (Incomplete) [bind9] - bind
> > 9.16.1-ubuntu on ubuntu 20.04 randomly
> > exits with [...]
> >
> > The bug reporter got back with some logs
> > and files collected via apport. Since the
> > bug is not easily reproducible, I am not quite
> > sure what's the best way forward. That said,
> > me/Paride are looking into it.
>
> Here's an update: we couldn't find anything worthwhile from the
> upstream tracker or changelog or anything and neither of us have
> enough experience with crash files.
Here's a paint-by-numbers way to get stacktraces from .crash files:
0. Enable debug symbols. Various ways to do this, one way is to append
ddebs to your apt sources:
$ echo "deb http://ddebs.ubuntu.com focal main restricted universe
multiverse" | sudo tee -a /etc/apt/sources.list
$ echo "deb http://ddebs.ubuntu.com focal-updates main restricted universe
multiverse" | sudo tee -a /etc/apt/sources.list
$ sudo apt install ubuntu-dbgsym-keyring
$ sudo apt update
1. Install package with debug symbols
$ sudo apt install bind9 bind9*-dbgsym
Often there are also secondary libs or deps with more syms we'll
need. This examines the binary for other possible dbgsym's:
$ sudo apt install debian-goodies
$ sudo apt install $(find-dbgsym-packages /usr/sbin/named 2>/dev/null)
The following NEW packages will be installed:
libcap2-dbgsym libcom-err2-dbgsym libgcc-s1-dbgsym libicu67-dbgsym
libjson-c5-dbgsym libkeyutils1-dbgsym
libkrb5-dbg liblmdb0-dbgsym liblzma5-dbgsym libmaxminddb0-dbgsym
libxml2-dbgsym zlib1g-dbgsym
2. Download the .crash file to a tmp directory
$ mkdir /tmp/bind9-crash
$ cd /tmp/bind9-crash
$ wget
https://bugs.launchpad.net/ubuntu/+source/bind9/+bug/1954854/+attachment/5551855/+files/_usr_sbin_named.114.crash
3. Register the package with the crash file
$ cat <(echo "Package: bind9") _usr_sbin_named.114.crash >
bind9_named.114.crash
Apport-retrace complains if the package isn't specified, but apport
hooks don't insert it, so you have to do it manually. ¯\_(ツ)_/¯
4. Doublecheck that your installed version matches the reporter's
version exactly:
$ apt-cache policy bind9 | grep '^ Installed'
Installed: 1:9.16.1-0ubuntu2.9
Unfortunately the .crash file doesn't include the version number,
but the LP bug report will show it:
Package: bind9 1:9.16.1-0ubuntu2.9
5. Retrace symbols
$ sudo apt-get install apport-retrace
$ apport-retrace bind9_named.114.crash
The above command inserted an empty 'separator: ' line into the crash
file, which apport-unpack will choke on, so delete that line:
$ sed -i '/^separator: *$/d' ./bind9_named.114.crash
6. Unpack crash
$ apport-unpack bind9_named.114.crash crash-114
The 'crash-114/' subdirectory will now include a ThreadStacktrace file
that hopefully(!) should have a usable backtrace. This one looks
great but the full stacktrace is rather TMI.
$ wc -l crash-114/ThreadStacktrace
415 crash-114/ThreadStacktrace
7. (Advanced) GDB
$ apt-get source bind9
$ cd bind9-9.16.1/
$ apport-retrace --gdb /tmp/bind9-crash/bind9_named.114.crash
This will give you an interactive gdb session on the reporter's
coredump. For example, with this particular crash, we can get a
simple backtrace:
(gdb) bt
#0 isc__nm_tcp_send (handle=0x7eff7522dbb0, region=0x7eff7d39a9b8,
cb=0x7eff887675a0 ,
cbarg=0x7eff7d39a9a8) at tcp.c:852
#1 0x7eff88a2e707 in client_sendpkg (client=client@entry=0x7eff754c31b0,
buffer=,
buffer=) at client.c:331
#2 0x7eff88a2ffe9 in ns_client_send (client=client@entry=0x7eff754c31b0)
at client.c:592
#3 0x7eff88a3e9b0 in query_send (client=0x7eff754c31b0) at query.c:552
#4 0x7eff88a469a7 in ns_query_done (qctx=qctx@entry=0x7eff85476850) at
query.c:10914
#5 0x7eff88a4dde6 in query_respond (qctx=0x7eff85476850) at query.c:7407
#6 query_prepresponse (qctx=qctx@entry=0x7eff85476850) at query.c:9906
#7 0x7eff88a49936 in query_gotanswer (qctx=qctx@entry=0x7eff85476850,
res=res@entry=0) at query.c:6823
#8 0x7eff88a4f4c6 in query_resume (qctx=0x7eff85476850) at query.c:6121
#9 fetch_callback (task=, event=) at query.c:5703
#10 0x7eff88770fa1 in dispatch (threadid=,
manager=) at task.c:1152
#11 run (queuep=) at task.c:1344
#12 0x7eff88239609 in start_thread (arg=) at
pthread_create.c:477
#13 0x7eff8815a293 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) list tcp.c:852
file: "src/unix/tcp.c", line number: 852, symbol: "???"
847 src/unix/tcp.c: No such file or directory.
file: "tcp.c", line number: 852, symbol: "???"
847 void *cbarg) {
848 isc_nmsocket_t *sock = handle->sock;
849 isc__netievent_tcpse
