Re: [Pdns-users] tcp listener issue - hopefully fixed
Wonderful news! This truly closes this truly longstanding bug ;-) Bert On Tue, Nov 30, 2010 at 09:43:24AM +, Simon Bedford wrote: Bert, I have now rolled out the new 3.3 version to our entire estate of servers and have seen no further bugs, the process has been stable for a number of weeks now and all looks good, many thanks for all of your help. Simon From: pdns-users-boun...@mailman.powerdns.com [pdns-users-boun...@mailman.powerdns.com] On Behalf Of Brad Dameron [brad.dame...@clearwire.com] Sent: 09 September 2010 20:04 To: bert hubert Cc: Mike; pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] tcp listener issue - hopefully fixed Bert, I re-downloaded the src and rebuilt my RPM. It appears to be up and running now. Unknown why the first compiles had this issue. I have been testing the last 2 hours and so far so good. I'll let you know tomorrow the final outcome. Thanks, Brad -Original Message- From: bert hubert [mailto:bert.hub...@netherlabs.nl] Sent: Thursday, September 09, 2010 5:06 AM To: Brad Dameron Cc: Mike; pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] tcp listener issue - hopefully fixed On Wed, Sep 08, 2010 at 11:16:41PM +, Brad Dameron wrote: Bert, I can't seem to get this latest version to run right. I build my RPM. I launch it with the following: This is not good! Does it crash immediately, even without accepting any traffic? So if you bind it to port 54 (for example), does it crash too? Bert /usr/sbin/pdns_recursor --local-address=172.26.68.42,127.0.0.1 / --allow-from= --max-cache-entries=300 --log-common-errors=no / --threads=4 --socket-dir=/var/run/recursor1 --daemon --dont-query= pdns_recursor[32283]: Operating in 64 bits mode pdns_recursor[32283]: Reading random entropy from '/dev/urandom' pdns_recursor[32283]: WARNING: Allowing queries from all IP addresses - this can be a security risk! pdns_recursor[32283]: Inserting rfc 1918 private space zones pdns_recursor[32283]: Listening for UDP queries on 172.26.68.42:53 pdns_recursor[32283]: Listening for UDP queries on 127.0.0.1:53 pdns_recursor[32283]: Enabled TCP data-ready filter for (slight) DoS protection pdns_recursor[32283]: Listening for TCP queries on 172.26.68.42:53 pdns_recursor[32283]: Listening for TCP queries on 127.0.0.1:53 pdns_recursor[32283]: Calling daemonize, going to background pdns_recursor[32284]: Launching 4 threads pdns_recursor[32284]: Done priming cache with root hints kernel: pdns_recursor[32288] general protection rip:4ea75c rsp:42802840 error:0 pdns_recursor[32284]: Done priming cache with root hints pdns_recursor[32298]: PowerDNS recursor 3.3-pre (C) 2001-2010 PowerDNS.COM BV (Sep 8 2010, 22:53:00, gcc 4.1.2 20080704 (Red Hat pdns_recursor[32298]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it ac the GPL version 2. pdns_recursor[32298]: Operating in 64 bits mode pdns_recursor[32298]: Reading random entropy from '/dev/urandom' pdns_recursor[32298]: WARNING: Allowing queries from all IP addresses - this can be a security risk! pdns_recursor[32298]: Inserting rfc 1918 private space zones pdns_recursor[32298]: Listening for UDP queries on 172.26.68.42:53 pdns_recursor[32298]: Listening for UDP queries on 127.0.0.1:53 pdns_recursor[32298]: Enabled TCP data-ready filter for (slight) DoS protection pdns_recursor[32298]: Listening for TCP queries on 172.26.68.42:53 pdns_recursor[32298]: Listening for TCP queries on 127.0.0.1:53 pdns_recursor[32298]: Calling daemonize, going to background pdns_recursor[32299]: Launching 4 threads pdns_recursor[32299]: Done priming cache with root hints kernel: pdns_recursor[32300] general protection rip:4ea75c rsp:409ff840 error:0 pdns_recursor[32307]: PowerDNS recursor 3.3-pre (C) 2001-2010 PowerDNS.COM BV (Sep 8 2010, 22:53:00, gcc 4.1.2 20080704 (Red Hat pdns_recursor[32307]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it ac the GPL version 2. pdns_recursor[32307]: Operating in 64 bits mode pdns_recursor[32307]: Reading random entropy from '/dev/urandom' pdns_recursor[32307]: WARNING: Allowing queries from all IP addresses - this can be a security risk! pdns_recursor[32307]: Inserting rfc 1918 private space zones pdns_recursor[32307]: Listening for UDP queries on 172.26.68.42:53 pdns_recursor[32307]: Listening for UDP queries on 127.0.0.1:53 pdns_recursor[32307]: Enabled TCP data-ready filter for (slight) DoS protection pdns_recursor[32307]: Listening for TCP queries on 172.26.68.42:53 pdns_recursor[32307]: Listening for TCP queries on 127.0.0.1:53 pdns_recursor[32307]: Calling daemonize, going to background pdns_recursor[32308]: Launching 4 threads pdns_recursor[32308
Re: [Pdns-users] tcp listener issue - hopefully fixed
On Wed, Sep 08, 2010 at 11:16:41PM +, Brad Dameron wrote: Bert, I can’t seem to get this latest version to run right. I build my RPM. I launch it with the following: This is not good! Does it crash immediately, even without accepting any traffic? So if you bind it to port 54 (for example), does it crash too? Bert /usr/sbin/pdns_recursor --local-address=172.26.68.42,127.0.0.1 / --allow-from= --max-cache-entries=300 --log-common-errors=no / --threads=4 --socket-dir=/var/run/recursor1 --daemon --dont-query= pdns_recursor[32283]: Operating in 64 bits mode pdns_recursor[32283]: Reading random entropy from '/dev/urandom' pdns_recursor[32283]: WARNING: Allowing queries from all IP addresses - this can be a security risk! pdns_recursor[32283]: Inserting rfc 1918 private space zones pdns_recursor[32283]: Listening for UDP queries on 172.26.68.42:53 pdns_recursor[32283]: Listening for UDP queries on 127.0.0.1:53 pdns_recursor[32283]: Enabled TCP data-ready filter for (slight) DoS protection pdns_recursor[32283]: Listening for TCP queries on 172.26.68.42:53 pdns_recursor[32283]: Listening for TCP queries on 127.0.0.1:53 pdns_recursor[32283]: Calling daemonize, going to background pdns_recursor[32284]: Launching 4 threads pdns_recursor[32284]: Done priming cache with root hints kernel: pdns_recursor[32288] general protection rip:4ea75c rsp:42802840 error:0 pdns_recursor[32284]: Done priming cache with root hints pdns_recursor[32298]: PowerDNS recursor 3.3-pre (C) 2001-2010 PowerDNS.COM BV (Sep 8 2010, 22:53:00, gcc 4.1.2 20080704 (Red Hat pdns_recursor[32298]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it ac the GPL version 2. pdns_recursor[32298]: Operating in 64 bits mode pdns_recursor[32298]: Reading random entropy from '/dev/urandom' pdns_recursor[32298]: WARNING: Allowing queries from all IP addresses - this can be a security risk! pdns_recursor[32298]: Inserting rfc 1918 private space zones pdns_recursor[32298]: Listening for UDP queries on 172.26.68.42:53 pdns_recursor[32298]: Listening for UDP queries on 127.0.0.1:53 pdns_recursor[32298]: Enabled TCP data-ready filter for (slight) DoS protection pdns_recursor[32298]: Listening for TCP queries on 172.26.68.42:53 pdns_recursor[32298]: Listening for TCP queries on 127.0.0.1:53 pdns_recursor[32298]: Calling daemonize, going to background pdns_recursor[32299]: Launching 4 threads pdns_recursor[32299]: Done priming cache with root hints kernel: pdns_recursor[32300] general protection rip:4ea75c rsp:409ff840 error:0 pdns_recursor[32307]: PowerDNS recursor 3.3-pre (C) 2001-2010 PowerDNS.COM BV (Sep 8 2010, 22:53:00, gcc 4.1.2 20080704 (Red Hat pdns_recursor[32307]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it ac the GPL version 2. pdns_recursor[32307]: Operating in 64 bits mode pdns_recursor[32307]: Reading random entropy from '/dev/urandom' pdns_recursor[32307]: WARNING: Allowing queries from all IP addresses - this can be a security risk! pdns_recursor[32307]: Inserting rfc 1918 private space zones pdns_recursor[32307]: Listening for UDP queries on 172.26.68.42:53 pdns_recursor[32307]: Listening for UDP queries on 127.0.0.1:53 pdns_recursor[32307]: Enabled TCP data-ready filter for (slight) DoS protection pdns_recursor[32307]: Listening for TCP queries on 172.26.68.42:53 pdns_recursor[32307]: Listening for TCP queries on 127.0.0.1:53 pdns_recursor[32307]: Calling daemonize, going to background pdns_recursor[32308]: Launching 4 threads pdns_recursor[32308]: Done priming cache with root hints pdns_recursor[32308]: Enabled 'epoll' multiplexer kernel: pdns_recursor[32309]: segfault at 00723a41 rip 00723a41 rsp 409ffb18 error 15 Thanks, Brad From: pdns-users-boun...@mailman.powerdns.com [mailto:pdns-users-boun...@mailman.powerdns.com] On Behalf Of bert.hub...@netherlabs.nl Sent: Tuesday, September 07, 2010 5:45 AM To: Mike Cc: Brad Dameron; pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] tcp listener issue - hopefully fixed Simon, Brief reply, am on the road. 3.3 will be released the moment you, brad or laurent confirm the issue is truly gone. What I can do is make packages that will be binary identical to the real 3.3 once you 'bless' them as having solved your issue. This would save you an upgrade. Would this work for you? Bert. Sent from my phone. - Reply message - From: Simon Bedford sbedf...@plus.net Date: Mon, Sep 6, 2010 14:16 Subject: tcp listener issue - hopefully fixed To: bert hubert bert.hub...@netherlabs.nl Cc: Brad Dameron brad.dame...@clearwire.com, pdns-users@mailman.powerdns.com pdns-users@mailman.powerdns.com Hi Bert, Apologies for the delay in replying I have been assigned to some different work for the moment, I
Re: [Pdns-users] tcp listener issue - hopefully fixed
Bert, I re-downloaded the src and rebuilt my RPM. It appears to be up and running now. Unknown why the first compiles had this issue. I have been testing the last 2 hours and so far so good. I'll let you know tomorrow the final outcome. Thanks, Brad -Original Message- From: bert hubert [mailto:bert.hub...@netherlabs.nl] Sent: Thursday, September 09, 2010 5:06 AM To: Brad Dameron Cc: Mike; pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] tcp listener issue - hopefully fixed On Wed, Sep 08, 2010 at 11:16:41PM +, Brad Dameron wrote: Bert, I can't seem to get this latest version to run right. I build my RPM. I launch it with the following: This is not good! Does it crash immediately, even without accepting any traffic? So if you bind it to port 54 (for example), does it crash too? Bert /usr/sbin/pdns_recursor --local-address=172.26.68.42,127.0.0.1 / --allow-from= --max-cache-entries=300 --log-common-errors=no / --threads=4 --socket-dir=/var/run/recursor1 --daemon --dont-query= pdns_recursor[32283]: Operating in 64 bits mode pdns_recursor[32283]: Reading random entropy from '/dev/urandom' pdns_recursor[32283]: WARNING: Allowing queries from all IP addresses - this can be a security risk! pdns_recursor[32283]: Inserting rfc 1918 private space zones pdns_recursor[32283]: Listening for UDP queries on 172.26.68.42:53 pdns_recursor[32283]: Listening for UDP queries on 127.0.0.1:53 pdns_recursor[32283]: Enabled TCP data-ready filter for (slight) DoS protection pdns_recursor[32283]: Listening for TCP queries on 172.26.68.42:53 pdns_recursor[32283]: Listening for TCP queries on 127.0.0.1:53 pdns_recursor[32283]: Calling daemonize, going to background pdns_recursor[32284]: Launching 4 threads pdns_recursor[32284]: Done priming cache with root hints kernel: pdns_recursor[32288] general protection rip:4ea75c rsp:42802840 error:0 pdns_recursor[32284]: Done priming cache with root hints pdns_recursor[32298]: PowerDNS recursor 3.3-pre (C) 2001-2010 PowerDNS.COM BV (Sep 8 2010, 22:53:00, gcc 4.1.2 20080704 (Red Hat pdns_recursor[32298]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it ac the GPL version 2. pdns_recursor[32298]: Operating in 64 bits mode pdns_recursor[32298]: Reading random entropy from '/dev/urandom' pdns_recursor[32298]: WARNING: Allowing queries from all IP addresses - this can be a security risk! pdns_recursor[32298]: Inserting rfc 1918 private space zones pdns_recursor[32298]: Listening for UDP queries on 172.26.68.42:53 pdns_recursor[32298]: Listening for UDP queries on 127.0.0.1:53 pdns_recursor[32298]: Enabled TCP data-ready filter for (slight) DoS protection pdns_recursor[32298]: Listening for TCP queries on 172.26.68.42:53 pdns_recursor[32298]: Listening for TCP queries on 127.0.0.1:53 pdns_recursor[32298]: Calling daemonize, going to background pdns_recursor[32299]: Launching 4 threads pdns_recursor[32299]: Done priming cache with root hints kernel: pdns_recursor[32300] general protection rip:4ea75c rsp:409ff840 error:0 pdns_recursor[32307]: PowerDNS recursor 3.3-pre (C) 2001-2010 PowerDNS.COM BV (Sep 8 2010, 22:53:00, gcc 4.1.2 20080704 (Red Hat pdns_recursor[32307]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it ac the GPL version 2. pdns_recursor[32307]: Operating in 64 bits mode pdns_recursor[32307]: Reading random entropy from '/dev/urandom' pdns_recursor[32307]: WARNING: Allowing queries from all IP addresses - this can be a security risk! pdns_recursor[32307]: Inserting rfc 1918 private space zones pdns_recursor[32307]: Listening for UDP queries on 172.26.68.42:53 pdns_recursor[32307]: Listening for UDP queries on 127.0.0.1:53 pdns_recursor[32307]: Enabled TCP data-ready filter for (slight) DoS protection pdns_recursor[32307]: Listening for TCP queries on 172.26.68.42:53 pdns_recursor[32307]: Listening for TCP queries on 127.0.0.1:53 pdns_recursor[32307]: Calling daemonize, going to background pdns_recursor[32308]: Launching 4 threads pdns_recursor[32308]: Done priming cache with root hints pdns_recursor[32308]: Enabled 'epoll' multiplexer kernel: pdns_recursor[32309]: segfault at 00723a41 rip 00723a41 rsp 409ffb18 error 15 Thanks, Brad From: pdns-users-boun...@mailman.powerdns.com [mailto:pdns-users-boun...@mailman.powerdns.com] On Behalf Of bert.hub...@netherlabs.nl Sent: Tuesday, September 07, 2010 5:45 AM To: Mike Cc: Brad Dameron; pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] tcp listener issue - hopefully fixed Simon, Brief reply, am on the road. 3.3 will be released the moment you, brad or laurent confirm the issue is truly gone. What I can do is make packages that will be binary identical to the real 3.3 once you 'bless' them as having solved your issue
Re: [Pdns-users] tcp listener issue - hopefully fixed
On Sun, Aug 29, 2010 at 09:17:01PM +, Brad Dameron wrote: The release process for 3.3 can now start - only 1 feature request left to finish. Good to hear Bert. I'll run it through the ringer on Monday and see if we can reproduce the problem. Cross fingers that it is fixed. Brad, Any news? If I get confirmation from you, Simon or Laurent that the problem is gone, we can go for release. Thanks ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] tcp listener issue - hopefully fixed
On Mon, Aug 23, 2010 at 11:37:42AM +0100, Simon Bedford wrote: Just returned from annual leave to this welcome news, has anyone had a chance to try it as yet and investigate whether the bug still manifests? And does it no longer happen when running Brad's test script? Simon, Sadly we found that the previous code drop, while stable, still exhibited the same problem. We've fixed yet another bug that might be causing the issue, and we are currently testing that. This drop can be found on http://svn.powerdns.com/snapshots/pdns-recursor-3.3-pre.tar.bz2 If you compile the program below, and run it on your pdns server, you'll probably see it print '0' and then a very large number. If you could share that output with us, it could tell us if we are on the right track. http://pastie.org/1124088 http://pastie.org/1124088.txt To compile, save as .cpp file and compile with 'g++ -O2 testing.cpp -o testing', and run with './testing'. Good luck! Bert ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] tcp listener issue - hopefully fixed
On Sun, Aug 29, 2010 at 04:56:14PM +0200, bert hubert wrote: We've fixed yet another bug that might be causing the issue, and we are currently testing that. This drop can be found on http://svn.powerdns.com/snapshots/pdns-recursor-3.3-pre.tar.bz2 Our testing shows that the problem disappeared! Many thanks to Christian Hofstaedtler. If this was the final problem, this also explains why only some people are seeing it. There are hardware platforms that are highly immune to this bug (Michel Stol measured this), but we've seen an Openvirtuozo hosted Linux being very vulnerable to it. On my own testing hardware, the problem was relatively rare. For the people who care, the root cause appeared to be doing non-atomic updates to the TCP client count. The release process for 3.3 can now start - only 1 feature request left to finish. Bert ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] tcp listener issue - hopefully fixed
-Original Message- From: pdns-users-boun...@mailman.powerdns.com [mailto:pdns-users- boun...@mailman.powerdns.com] On Behalf Of bert hubert Sent: Sunday, August 29, 2010 11:31 AM To: Mike Cc: Brad Dameron; pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] tcp listener issue - hopefully fixed On Sun, Aug 29, 2010 at 04:56:14PM +0200, bert hubert wrote: We've fixed yet another bug that might be causing the issue, and we are currently testing that. This drop can be found on http://svn.powerdns.com/snapshots/pdns-recursor-3.3-pre.tar.bz2 Our testing shows that the problem disappeared! Many thanks to Christian Hofstaedtler. If this was the final problem, this also explains why only some people are seeing it. There are hardware platforms that are highly immune to this bug (Michel Stol measured this), but we've seen an Openvirtuozo hosted Linux being very vulnerable to it. On my own testing hardware, the problem was relatively rare. For the people who care, the root cause appeared to be doing non-atomic updates to the TCP client count. The release process for 3.3 can now start - only 1 feature request left to finish. Bert Good to hear Bert. I'll run it through the ringer on Monday and see if we can reproduce the problem. Cross fingers that it is fixed. Thanks, Brad This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
[Pdns-users] tcp listener issue - hopefully fixed
Hi everybody, After an *exhaustive* search, no definitive cause of the TCP/IP issue has been found, even though we were eventually able to reproduce it (with the help of Brad). To make headway in solving this, we've now removed most of the old TCP/IP connection accounting code, and replaced it by something a lot cleaner. Details can be seen on http://wiki.powerdns.com/trac/changeset/1685 If anything, this has made PowerDNS a bit smaller ;-) A PowerDNS Recursor 3.3 pre-release is available on http://svn.powerdns.com/snapshots/pdns-recursor-3.3-pre.tar.bz2 and this contains the improvements. This is not the preferred way to solve bugs, nor can we be sure that the problem has in fact been solved. Brad, can you see if you can reproduce the issue using this version? It usually takes us around 2 hours to even see the problem, but for you it usually hit within a few minuts I understand. We hope to hear from you! If this issue is resolved, 3.3 is around the corner. Kind regards, Bert On Wed, Aug 04, 2010 at 09:30:03PM +, Brad Dameron wrote: Bert, Any update on this? Thanks, Brad -Original Message- From: pdns-users-boun...@mailman.powerdns.com [mailto:pdns-users-boun...@mailman.powerdns.com] On Behalf Of bert hubert Sent: Tuesday, July 06, 2010 9:01 AM To: Mike Cc: Brad Dameron; pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] Possible tcp listener issue The issue has now been reproduced on one of the PowerDNS servers, thanks to Brad and Christian. However, it is quite some work to make it happen and we've not yet been able to make sense of *why* it is happening. We'll keep you posted. Once this is solved, 3.3 will be released. Bert On Tue, Jul 06, 2010 at 04:11:35PM +0100, Simon Bedford wrote: Hi Christian, Have we had any joy in replicating this issue or highlighting the root cause of the issue? Thanks Simon Christian Hofstädtler wrote: Brad, Josh, I'm trying to reproduce the behaviour you see here, but fail to do so at this time. Maybe you can give me more details about your environment, at least OS version, architecture, and any compile options you've used to build the binary (say, STATIC or LUA). Also of interest would be the last SVN revision you've tried out. If there is anything else you can think of which is special in your environment, please let me know of that, too. Thanks, Christian ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users