Bug#849821: pdns-recursor: Crash with DNSSEC enabled

[Chris Boot]

On 20/01/17 11:36, Chris Boot wrote:
> On 19 Jan 2017, at 23:04, Christian Hofstaedtler  wrote:
>>
>> Hi Chris,
>>
>> there's a new pdns-recursor 4.0.4-1 in sid, maybe you could give it
>> a try.
> 
> Hi Christian,
> 
> I’ve been running this for a few days now - so far so good. I had a crash 
> (that I didn’t catch through gdb) with 4.0.3-6 but 4.0.4-1 seems solid so 
> far. I’ll let you know if it happens again.

Hi Christian,

I've just realised that it's been ages now and pdns_recursor has not
crashed at all for me. So I think this bug can be closed now, and marked
as fixed in 4.0.4-1.

Thanks for persisting with this bug, and to upstream too for being ever
helpful!

If fact it's so good it's one of those services that Just Work for me
and I no longer have to think about it at all.

HTH,
Chris

-- 
Chris Boot
bo...@debian.org
GPG: 8467 53CB 1921 3142 C56D  C918 F5C8 3C05 D9CE 

Bug#849821: pdns-recursor: Crash with DNSSEC enabled

[Chris Boot]

On 19 Jan 2017, at 23:04, Christian Hofstaedtler  wrote:
> 
> Hi Chris,
> 
> there's a new pdns-recursor 4.0.4-1 in sid, maybe you could give it
> a try.

Hi Christian,

I’ve been running this for a few days now - so far so good. I had a crash (that 
I didn’t catch through gdb) with 4.0.3-6 but 4.0.4-1 seems solid so far. I’ll 
let you know if it happens again.

Cheers,
Chris

-- 
Chris Boot
bo...@bootc.net

Bug#849821: pdns-recursor: Crash with DNSSEC enabled

[Christian Hofstaedtler]

Hi Chris,

there's a new pdns-recursor 4.0.4-1 in sid, maybe you could give it
a try.

Best,
-- 
christian hofstaedtler 

Bug#849821: pdns-recursor: Crash with DNSSEC enabled

[Chris Boot]

On 03/01/17 10:51, Christian Hofstaedtler wrote:
> Hi Chris,
> 
> * Chris Boot  [170102 18:45]:
>> I updated to 4.0.3-5 on Saturday, and have just experienced another
>> crash. This time I also got double-free/corruption errors:
> 
> Hmm. Upstream is also a bit at a loss with those weird crashes.
> Your stacktrace was at least useful to probably rule out the
> libssl and the getZoneCuts issues.
> 
> There's now a 4.0.3-6 in unstable that's closer to the current 4.0.x
> release branch (-5 was missing a few more changesets).
> 
> Maybe you could try that?

Hi Christian,

I've now been running with 4.0.3-6 for a few days and have just
encountered another crash. Unfortunately, I wasn't running
pdns-recursor through gdb this time so all I got was:

*** Error in `/usr/sbin/pdns_recursor': malloc(): smallbin double linked
list corrupted: 0x7fd21c2c0f40 ***
=== Backtrace: =
/lib/x86_64-linux-gnu/libc.so.6(+0x70bcb)[0x7fd22c3d5bcb]
/lib/x86_64-linux-gnu/libc.so.6(+0x76fa6)[0x7fd22c3dbfa6]
/lib/x86_64-linux-gnu/libc.so.6(+0x793c1)[0x7fd22c3de3c1]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7fd22c3dfd94]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_Znwm+0x18)[0x7fd22ccad758]
/usr/sbin/pdns_recursor(+0x187053)[0x558b51345053]
/usr/sbin/pdns_recursor(+0x25eb5d)[0x558b5141cb5d]
/usr/sbin/pdns_recursor(+0x1fb064)[0x558b513b9064]
/usr/sbin/pdns_recursor(+0x16c101)[0x558b5132a101]
/usr/sbin/pdns_recursor(+0x14e330)[0x558b5130c330]
/usr/sbin/pdns_recursor(+0x20cdc7)[0x558b513cadc7]
/usr/sbin/pdns_recursor(+0x21586a)[0x558b513d386a]
/usr/sbin/pdns_recursor(+0x21a32d)[0x558b513d832d]
/usr/sbin/pdns_recursor(+0x21bfa8)[0x558b513d9fa8]
/usr/sbin/pdns_recursor(+0x213435)[0x558b513d1435]
/usr/sbin/pdns_recursor(+0x21a32d)[0x558b513d832d]
/usr/sbin/pdns_recursor(+0x21bfa8)[0x558b513d9fa8]
/usr/sbin/pdns_recursor(+0x213435)[0x558b513d1435]
/usr/sbin/pdns_recursor(+0x21a32d)[0x558b513d832d]
/usr/sbin/pdns_recursor(+0x21bfa8)[0x558b513d9fa8]
/usr/sbin/pdns_recursor(+0x213435)[0x558b513d1435]
/usr/sbin/pdns_recursor(+0x21a32d)[0x558b513d832d]
/usr/sbin/pdns_recursor(+0x21bfa8)[0x558b513d9fa8]
/usr/sbin/pdns_recursor(+0x213435)[0x558b513d1435]
/usr/sbin/pdns_recursor(+0x21a32d)[0x558b513d832d]
/usr/sbin/pdns_recursor(+0x21bfa8)[0x558b513d9fa8]
/usr/sbin/pdns_recursor(+0x213435)[0x558b513d1435]
/usr/sbin/pdns_recursor(+0x21a32d)[0x558b513d832d]
/usr/sbin/pdns_recursor(+0x21bfa8)[0x558b513d9fa8]
/usr/sbin/pdns_recursor(+0x213435)[0x558b513d1435]
/usr/sbin/pdns_recursor(+0x21a32d)[0x558b513d832d]
/usr/sbin/pdns_recursor(+0x21bfa8)[0x558b513d9fa8]
/usr/sbin/pdns_recursor(+0x213435)[0x558b513d1435]
/usr/sbin/pdns_recursor(+0x21a32d)[0x558b513d832d]
/usr/sbin/pdns_recursor(+0x21bfa8)[0x558b513d9fa8]
/usr/sbin/pdns_recursor(+0x213435)[0x558b513d1435]
/usr/sbin/pdns_recursor(+0x21a32d)[0x558b513d832d]
/usr/sbin/pdns_recursor(+0x21bfa8)[0x558b513d9fa8]
/usr/sbin/pdns_recursor(+0x213435)[0x558b513d1435]
/usr/sbin/pdns_recursor(+0x21a32d)[0x558b513d832d]
/usr/sbin/pdns_recursor(+0x21bfa8)[0x558b513d9fa8]
/usr/sbin/pdns_recursor(+0x213435)[0x558b513d1435]
/usr/sbin/pdns_recursor(+0x21a32d)[0x558b513d832d]
/usr/sbin/pdns_recursor(+0x21bfa8)[0x558b513d9fa8]
/usr/sbin/pdns_recursor(+0x213435)[0x558b513d1435]
/usr/sbin/pdns_recursor(+0x21a32d)[0x558b513d832d]
/usr/sbin/pdns_recursor(+0x21bfa8)[0x558b513d9fa8]
/usr/sbin/pdns_recursor(+0x213435)[0x558b513d1435]
/usr/sbin/pdns_recursor(+0x21a32d)[0x558b513d832d]
/usr/sbin/pdns_recursor(+0x21bfa8)[0x558b513d9fa8]
/usr/sbin/pdns_recursor(+0x213435)[0x558b513d1435]
/usr/sbin/pdns_recursor(+0x21a32d)[0x558b513d832d]
/usr/sbin/pdns_recursor(+0x21bfa8)[0x558b513d9fa8]
/usr/sbin/pdns_recursor(+0x213435)[0x558b513d1435]
/usr/sbin/pdns_recursor(+0x21a32d)[0x558b513d832d]
/usr/sbin/pdns_recursor(+0x21bfa8)[0x558b513d9fa8]
/usr/sbin/pdns_recursor(+0x213435)[0x558b513d1435]
/usr/sbin/pdns_recursor(+0x21a32d)[0x558b513d832d]
/usr/sbin/pdns_recursor(+0x21bfa8)[0x558b513d9fa8]
/usr/sbin/pdns_recursor(+0x213435)[0x558b513d1435]
/usr/sbin/pdns_recursor(+0x21a32d)[0x558b513d832d]
/usr/sbin/pdns_recursor(+0x21bfa8)[0x558b513d9fa8]
/usr/sbin/pdns_recursor(+0x213435)[0x558b513d1435]

I'm now running it through gdb so if it happens again I'll let you know.
It has certainly lasted longer this time than on some previous occasions.

Best regards,
Chris

-- 
Chris Boot
bo...@debian.org
GPG: 8467 53CB 1921 3142 C56D  C918 F5C8 3C05 D9CE 

Bug#849821: pdns-recursor: Crash with DNSSEC enabled

[Christian Hofstaedtler]

Hi Chris,

* Chris Boot  [170102 18:45]:
> I updated to 4.0.3-5 on Saturday, and have just experienced another
> crash. This time I also got double-free/corruption errors:

Hmm. Upstream is also a bit at a loss with those weird crashes.
Your stacktrace was at least useful to probably rule out the
libssl and the getZoneCuts issues.

There's now a 4.0.3-6 in unstable that's closer to the current 4.0.x
release branch (-5 was missing a few more changesets).

Maybe you could try that?

Thanks,
-ch

Bug#849821: pdns-recursor: Crash with DNSSEC enabled

[Chris Boot]

On 2016-12-31 16:52, Christian Hofstaedtler wrote:
> Chris,
> 
> * Chris Boot  [161231 16:18]:
>> Sure, that's now installed, now I just need to wait for it to fail
>> again. This will probably take several days; I'll get back to you when
>> it next crashes.
> 
> Great!
> 
> Incidentally, I've just uploaded version 4.0.3-5, which might fix
> the crash. Would be good if you could test againt that version.

Hi Christian,

I updated to 4.0.3-5 on Saturday, and have just experienced another
crash. This time I also got double-free/corruption errors:

*** Error in `/usr/sbin/pdns_recursor': double free or corruption (out): 
0x7f3ce0369a90 ***
=== Backtrace: =
/lib/x86_64-linux-gnu/libc.so.6(+0x70bcb)[0x7f3cec433bcb]
/lib/x86_64-linux-gnu/libc.so.6(+0x76fa6)[0x7f3cec439fa6]
/lib/x86_64-linux-gnu/libc.so.6(+0x7779e)[0x7f3cec43a79e]
/usr/sbin/pdns_recursor(+0x1843d2)[0x55b10f7ac3d2]
/usr/sbin/pdns_recursor(+0x184473)[0x55b10f7ac473]
/usr/sbin/pdns_recursor(+0x18d7a7)[0x55b10f7b57a7]
/usr/sbin/pdns_recursor(+0x16e511)[0x55b10f796511]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7464)[0x7f3ced004464]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x5f)[0x7f3cec4ab9df]
=== Memory map: 
55b10f628000-55b10fa8a000 r-xp  fe:00 526251 
/usr/sbin/pdns_recursor
55b10fc89000-55b10fc9 r--p 00461000 fe:00 526251 
/usr/sbin/pdns_recursor
55b10fc9-55b10fc9d000 rw-p 00468000 fe:00 526251 
/usr/sbin/pdns_recursor
55b10fc9d000-55b10fc9e000 rw-p  00:00 0
55b110d79000-55b110ef7000 rw-p  00:00 0  [heap]
7f3cdc00-7f3cdc9dd000 rw-p  00:00 0
7f3cdc9dd000-7f3ce000 ---p  00:00 0
7f3ce000-7f3ce09fc000 rw-p  00:00 0
7f3ce09fc000-7f3ce400 ---p  00:00 0
7f3ce400-7f3ce4183000 rw-p  00:00 0
7f3ce4183000-7f3ce800 ---p  00:00 0
7f3ce8ed6000-7f3ce8ed7000 ---p  00:00 0
7f3ce8ed7000-7f3ce96d7000 rw-p  00:00 0
7f3ce96d7000-7f3ce96d8000 ---p  00:00 0
7f3ce96d8000-7f3ce9ed8000 rw-p  00:00 0
7f3ce9ed8000-7f3ce9ed9000 ---p  00:00 0
7f3ce9ed9000-7f3cea6d9000 rw-p  00:00 0
7f3cea6d9000-7f3cea6e3000 r-xp  fe:00 132005 
/lib/x86_64-linux-gnu/libnss_files-2.24.so
7f3cea6e3000-7f3cea8e3000 ---p a000 fe:00 132005 
/lib/x86_64-linux-gnu/libnss_files-2.24.so
7f3cea8e3000-7f3cea8e4000 r--p a000 fe:00 132005 
/lib/x86_64-linux-gnu/libnss_files-2.24.so
7f3cea8e4000-7f3cea8e5000 rw-p b000 fe:00 132005 
/lib/x86_64-linux-gnu/libnss_files-2.24.so
7f3cea8e5000-7f3cea8eb000 rw-p  00:00 0
7f3cea8eb000-7f3cea8f6000 r-xp  fe:00 132011 
/lib/x86_64-linux-gnu/libnss_nis-2.24.so
7f3cea8f6000-7f3ceaaf5000 ---p b000 fe:00 132011 
/lib/x86_64-linux-gnu/libnss_nis-2.24.so
7f3ceaaf5000-7f3ceaaf6000 r--p a000 fe:00 132011 
/lib/x86_64-linux-gnu/libnss_nis-2.24.so
7f3ceaaf6000-7f3ceaaf7000 rw-p b000 fe:00 132011 
/lib/x86_64-linux-gnu/libnss_nis-2.24.so
7f3ceaaf7000-7f3ceab0b000 r-xp  fe:00 131831 
/lib/x86_64-linux-gnu/libnsl-2.24.so
7f3ceab0b000-7f3cead0b000 ---p 00014000 fe:00 131831 
/lib/x86_64-linux-gnu/libnsl-2.24.so
7f3cead0b000-7f3cead0c000 r--p 00014000 fe:00 131831 
/lib/x86_64-linux-gnu/libnsl-2.24.so
7f3cead0c000-7f3cead0d000 rw-p 00015000 fe:00 131831 
/lib/x86_64-linux-gnu/libnsl-2.24.so
7f3cead0d000-7f3cead0f000 rw-p  00:00 0
7f3cead0f000-7f3cead16000 r-xp  fe:00 131834 
/lib/x86_64-linux-gnu/libnss_compat-2.24.so
7f3cead16000-7f3ceaf15000 ---p 7000 fe:00 131834 
/lib/x86_64-linux-gnu/libnss_compat-2.24.so
7f3ceaf15000-7f3ceaf16000 r--p 6000 fe:00 131834 
/lib/x86_64-linux-gnu/libnss_compat-2.24.so
7f3ceaf16000-7f3ceaf17000 rw-p 7000 fe:00 131834 
/lib/x86_64-linux-gnu/libnss_compat-2.24.so
7f3ceaf17000-7f3ceafa6000 rw-p  00:00 0
7f3ceafa6000-7f3ceafb9000 r-xp  fe:00 131098 
/lib/x86_64-linux-gnu/libgpg-error.so.0.20.0
7f3ceafb9000-7f3ceb1b8000 ---p 00013000 fe:00 131098 
/lib/x86_64-linux-gnu/libgpg-error.so.0.20.0
7f3ceb1b8000-7f3ceb1b9000 r--p 00012000 fe:00 131098 
/lib/x86_64-linux-gnu/libgpg-error.so.0.20.0
7f3ceb1b9000-7f3ceb1ba000 rw-p 00013000 fe:00 131098 
/lib/x86_64-linux-gnu/libgpg-error.so.0.20.0
7f3ceb1ba000-7f3ceb22c000 r-xp  fe:00 131406 
/lib/x86_64-linux-gnu/libpcre.so.3.13.3
7f3ceb22c000-7f3ceb42b000 ---p 00072000 fe:00 131406 
/lib/x86_64-linux-gnu/libpcre.so.3.13.3
7f3ceb42b000-7f3ceb42c000 r--p 00071000 fe:00 131406 

Bug#849821: pdns-recursor: Crash with DNSSEC enabled

[Christian Hofstaedtler]

Chris,

* Chris Boot  [161231 16:18]:
> Sure, that's now installed, now I just need to wait for it to fail
> again. This will probably take several days; I'll get back to you when
> it next crashes.

Great!

Incidentally, I've just uploaded version 4.0.3-5, which might fix
the crash. Would be good if you could test againt that version.

Thanks,
  -ch

-- 
christian hofstaedtler 

Bug#849821: pdns-recursor: Crash with DNSSEC enabled

[Chris Boot]

On 2016-12-31 15:02, Christian Hofstaedtler wrote:
> Hi,
> 
> * Chris Boot  [161231 15:39]:
>> Version: 4.0.3-4
>>
>> Since the upgrade from 4.0.3-3 to 4.0.3-4 I have been experiencing
>> intermittent crashes in pdns-recursor. I have so far not managed to get
>> a core dump from the daemon, so have resorted to running the process
>> within gdb to catch a backtrace.
> 
> Thank you for your report!
> 
> Upstream has asked if you could install libssl1.1-dbg and capture
> the stack of all threads when this happens? (With "thread apply all
> bt full" in gdb.) 

Hi Christian,

Sure, that's now installed, now I just need to wait for it to fail
again. This will probably take several days; I'll get back to you when
it next crashes.

Cheers,
Chris

-- 
Chris Boot
bo...@debian.org
GPG: 8467 53CB 1921 3142 C56D  C918 F5C8 3C05 D9CE 



signature.asc
Description: OpenPGP digital signature

Bug#849821: pdns-recursor: Crash with DNSSEC enabled

[Christian Hofstaedtler]

Hi,

* Chris Boot  [161231 15:39]:
> Version: 4.0.3-4
> 
> Since the upgrade from 4.0.3-3 to 4.0.3-4 I have been experiencing
> intermittent crashes in pdns-recursor. I have so far not managed to get
> a core dump from the daemon, so have resorted to running the process
> within gdb to catch a backtrace.

Thank you for your report!

Upstream has asked if you could install libssl1.1-dbg and capture
the stack of all threads when this happens? (With "thread apply all
bt full" in gdb.) 

Cheers,
-- 
christian hofstaedtler 

Bug#849821: pdns-recursor: Crash with DNSSEC enabled

[Chris Boot]

Package: pdns-recursor
Version: 4.0.3-4
Severity: important

Dear maintainers,

Since the upgrade from 4.0.3-3 to 4.0.3-4 I have been experiencing
intermittent crashes in pdns-recursor. I have so far not managed to get
a core dump from the daemon, so have resorted to running the process
within gdb to catch a backtrace. I have included this below:

(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:58
#1  0x7f405816440a in __GI_abort () at abort.c:89
#2  0x7f40581a0bd0 in __libc_message (do_abort=do_abort@entry=2, 
fmt=fmt@entry=0x7f4058295c70 "*** Error in `%s': %s: 0x%s ***\n") at 
../sysdeps/posix/libc_fatal.c:175
#3  0x7f40581a6fa6 in malloc_printerr (action=3, str=0x7f4058295d80 
"free(): invalid next size (fast)", ptr=, ar_ptr=) at malloc.c:5046
#4  0x7f40581a779e in _int_free (av=0x7f404c20, p=0x7f404c9985f0, 
have_lock=0) at malloc.c:3902
#5  0x55b3dcea35e3 in 
__gnu_cxx::new_allocator 
>::deallocate (this=, __p=) at 
/usr/include/c++/6/ext/new_allocator.h:110
#6  std::allocator_traits > >::deallocate (__a=..., __n=1, __p=) at 
/usr/include/c++/6/bits/alloc_traits.h:442
#7  std::_Rb_tree, std::less, 
std::allocator >::_M_put_node 
(this=0x7f404cb6bb60, __p=) at 
/usr/include/c++/6/bits/stl_tree.h:509
#8  std::_Rb_tree, std::less, 
std::allocator >::_M_drop_node 
(__p=, this=0x7f404cb6bb60) at 
/usr/include/c++/6/bits/stl_tree.h:576
#9  std::_Rb_tree, std::less, 
std::allocator >::_M_erase 
(this=this@entry=0x7f404cb6bb60, __x=) at 
/usr/include/c++/6/bits/stl_tree.h:1640
#10 0x55b3dcea35f4 in std::_Rb_tree, 
std::less, std::allocator 
>::_M_erase (this=this@entry=0x7f404cb6bb60, __x=0x7f404c0e4d70) at 
/usr/include/c++/6/bits/stl_tree.h:1638
#11 0x55b3dcea35f4 in std::_Rb_tree, 
std::less, std::allocator 
>::_M_erase (this=this@entry=0x7f404cb6bb60, __x=0x7f404c7b3220) at 
/usr/include/c++/6/bits/stl_tree.h:1638
#12 0x55b3dcea35f4 in std::_Rb_tree, 
std::less, std::allocator 
>::_M_erase (this=this@entry=0x7f404cb6bb60, __x=0x7f404c9fdc30) at 
/usr/include/c++/6/bits/stl_tree.h:1638
#13 0x55b3dcea35f4 in std::_Rb_tree, 
std::less, std::allocator 
>::_M_erase (this=this@entry=0x7f404cb6bb60, __x=0x7f404c9fc200) at 
/usr/include/c++/6/bits/stl_tree.h:1638
#14 0x55b3dcea9a43 in std::_Rb_tree, 
std::less, std::allocator 
>::~_Rb_tree (this=0x7f404cb6bb60, __in_chrg=) at 
/usr/include/c++/6/bits/stl_tree.h:873
#15 std::map >::~map (this=0x7f404cb6bb60, 
__in_chrg=) at /usr/include/c++/6/bits/stl_map.h:96
#16 SyncRes::shuffleInSpeedOrder (this=this@entry=0x7f404cb76c60, 
tnameservers=..., prefix=...) at syncres.cc:885
#17 0x55b3dce96276 in SyncRes::doResolveAt (this=this@entry=0x7f404cb76c60, 
nameservers=..., auth=..., flawedNSSet=flawedNSSet@entry=false, qname=..., 
qtype=..., ret=..., depth=, beenthere=...) at syncres.cc:989
#18 0x55b3dce9fd5d in SyncRes::doResolve (this=this@entry=0x7f404cb76c60, 
qname=..., qtype=..., ret=..., depth=depth@entry=15, beenthere=...) at 
syncres.cc:470
#19 0x55b3dcea19d8 in SyncRes::getAddrs (this=this@entry=0x7f404cb76c60, 
qname=..., depth=depth@entry=14, beenthere=...) at syncres.cc:517
#20 0x55b3dce98ed2 in SyncRes::doResolveAt (this=this@entry=0x7f404cb76c60, 
nameservers=..., auth=..., flawedNSSet=flawedNSSet@entry=false, qname=..., 
qtype=..., ret=..., depth=, beenthere=...) at syncres.cc:1041
#21 0x55b3dce9fd5d in SyncRes::doResolve (this=this@entry=0x7f404cb76c60, 
qname=..., qtype=..., ret=..., depth=depth@entry=12, beenthere=...) at 
syncres.cc:470
#22 0x55b3dcea19d8 in SyncRes::getAddrs (this=this@entry=0x7f404cb76c60, 
qname=..., depth=depth@entry=11, beenthere=...) at syncres.cc:517
#23 0x55b3dce98ed2 in SyncRes::doResolveAt (this=this@entry=0x7f404cb76c60, 
nameservers=..., auth=..., flawedNSSet=flawedNSSet@entry=false, qname=..., 
qtype=..., ret=..., depth=, beenthere=...) at syncres.cc:1041
#24 0x55b3dce9fd5d in SyncRes::doResolve (this=this@entry=0x7f404cb76c60, 
qname=..., qtype=..., ret=..., depth=depth@entry=9, beenthere=...) at 
syncres.cc:470
#25 0x55b3dcea19d8 in SyncRes::getAddrs (this=this@entry=0x7f404cb76c60, 
qname=..., depth=depth@entry=8, beenthere=...) at syncres.cc:517
#26 0x55b3dce98ed2 in SyncRes::doResolveAt (this=this@entry=0x7f404cb76c60, 
nameservers=..., auth=..., flawedNSSet=flawedNSSet@entry=false, qname=...,