Re: dnsperf and BIND memory consumption
Danny Mayer wrote: > Doug Barton wrote: >> You'd have to dig into the source and really understand what's happening >> now vs. what was happening before in order for me to answer this >> question, and by the time you had done that work I would not need to >> answer this question for you. :) > > You would have a very hard time finding it just using code inspection ... I think one of us is talking about something different that what the other person is talking about. :) I was referring to the different defaults (and the subsequent decisions for include files, etc.) that come when configure picks the right architecture. Good work on finding that bug though. :) Doug ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
Doug Barton wrote: > Vinny Abello wrote: >> Just for clarification, is there any downside to this autoconf fix >> vs. how it was previously working? > > It was not working correctly previously, so no. > >> Does autoconf still not understand AMD64 on FreeBSD > > You're confusing "autoconf" and "the configure script provided in the > sources for BIND >= 9.5.x." There is also a variable of "what the > FreeBSD ports provide to configure by default" that I have deliberately > omitted for the sake of clarity. > >> and is your fix just defaulting to something lesser? > > No. > >> It appeared that the switches being run were for 32 bit after this >> fix, but I'm no expert on compilers by far. > > You'd have to dig into the source and really understand what's happening > now vs. what was happening before in order for me to answer this > question, and by the time you had done that work I would not need to > answer this question for you. :) You would have a very hard time finding it just using code inspection since the bug was found several levels under the code since there were a number of Macros in place and the source of the problem was in the creation but the fix had to be elsewhere. I only found this leak because I had accidentally left on a debugging tool when test running on a Windows box and the debugging tool was causing it to break at the creation point. It took me quite a while to peel back the layers and find out why it was really complaining. The fix was easy, but understanding what needed to be fixed and why was not. Danny > The short version is that right now the port is doing what it should be. > > > hth, > > Doug > > ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
At Mon, 5 Jan 2009 18:58:07 -0500, Vinny Abello wrote: > My basic question is: Is there any advantage to compiling BIND in > the previous manner now that there is a fix in the BIND source code? Do you mean compiling BIND with the memory leak fix and without the FreeBSD port change by "in the previous manner"? For this particular memory leak problem, there's nothing wrong with it. But then you won't benefit from the other (main) advantage of atomic operations: performance improvement. So, you'll still be happier with applying two patches. --- JINMEI, Tatuya Internet Systems Consortium, Inc. ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
At Thu, 1 Jan 2009 00:47:10 -0500, Vinny Abello wrote: > I just loaded up the BIND 9.5.1 port on FreeBSD 7.0 AMD64 with > threads. I don't see the prominent memory leak present on my system > any longer. I lost track of this thread and think two different > changes might have been made, however. One via the port and one in > BIND itself, so I don't know which fix is actually in effect here, > but the results seem good so far. Maybe Doug and Jinmei and comment > on their respective changes just for the record. The memory leak only happens when autoconf fails to detect the availability of hardware-dependent atomic operations. The patch for the FreeBSD port fixes the autoconf problem. The patch I mentioned fixes the memory leak itself. The result is the same either way; the leak will disappear. --- JINMEI, Tatuya Internet Systems Consortium, Inc. ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
At Thu, 01 Jan 2009 00:04:49 -0500, Danny Mayer wrote: > Personally, I'm not convinced that it will make a difference outside of > Windows. The fix is to make sure a lock gets destroyed when done and the > function exits. On Windows the lock gets created and memory is allocated > for it outside of the function using it and needs to be explicitly > destroyed, but my understanding of pthreads was that this worked > differently on Unix and the lock structure would be automatically > destroyed when the function is exited since the lock was local to the > function. The pthread implementation of FreeBSD I tested allocated memory for a lock. That was the reason for the leak, and that was why the "Windows-specific" fix also helps FreeBSD (that happens to use pthread locks instead of atomic operations). --- JINMEI, Tatuya Internet Systems Consortium, Inc. ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
Vinny Abello wrote: >> -Original Message- >> From: Danny Mayer [mailto:ma...@gis.net] >> Sent: Thursday, January 01, 2009 12:05 AM >> To: jinmei_tat...@isc.org >> Cc: Vinny Abello; do...@freebsd.org; bind-us...@isc.org >> Subject: Re: dnsperf and BIND memory consumption >> >> JINMEI Tatuya / ...@l@C#:H wrote: >>> At Tue, 9 Dec 2008 15:26:25 -0500, >>> Vinny Abello wrote: >>> >>>> Has anybody else tried this patch for you? I haven't had time to >>>> look into this at all. If nobody has tried this yet, I'll get around >>>> to it when I can and let you know the result. >>> No one else other than by myself. It worked perfectly for me, i.e., >> I >>> could reproduce the problem and I could completely eliminate the leak >>> with the patch. One thing I was not certain about in an off-list >>> discussion that led to this patch was that the patch reportedly >> solved >>> the leak only partially. I've been hoping to confirm that, but >>> unfortunately I've not got any followup since then. >>> >>> So, basically, I believe the problem was solved, it would also help >> if >>> you could confirm it. >> Personally, I'm not convinced that it will make a difference outside of >> Windows. The fix is to make sure a lock gets destroyed when done and >> the >> function exits. On Windows the lock gets created and memory is >> allocated >> for it outside of the function using it and needs to be explicitly >> destroyed, but my understanding of pthreads was that this worked >> differently on Unix and the lock structure would be automatically >> destroyed when the function is exited since the lock was local to the >> function. >> >> I'd be interested in seeing the results of this. >> w >> Danny > > I just loaded up the BIND 9.5.1 port on FreeBSD 7.0 AMD64 with threads. I don't see the prominent memory leak present on my system any longer. I lost track of this thread and think two different changes might have been made, however. One via the port and one in BIND itself, so I don't know which fix is actually in effect here, but the results seem good so far. Maybe Doug and Jinmei and comment on their respective changes just for the record. The fix that Jinmei mentioned should be in the 9.5.1 release since it was found during the 9.5.0-P2-W2 development so you would not see it in 9.5.1. In addition, that code was new in 9.5.0 so you would not have seen it in prior releases. Danny > > -Vinny > > ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
RE: dnsperf and BIND memory consumption
> -Original Message- > From: Danny Mayer [mailto:ma...@gis.net] > Sent: Thursday, January 01, 2009 12:05 AM > To: jinmei_tat...@isc.org > Cc: Vinny Abello; do...@freebsd.org; bind-us...@isc.org > Subject: Re: dnsperf and BIND memory consumption > > JINMEI Tatuya / ...@l@C#:H wrote: > > At Tue, 9 Dec 2008 15:26:25 -0500, > > Vinny Abello wrote: > > > >> Has anybody else tried this patch for you? I haven't had time to > >> look into this at all. If nobody has tried this yet, I'll get around > >> to it when I can and let you know the result. > > > > No one else other than by myself. It worked perfectly for me, i.e., > I > > could reproduce the problem and I could completely eliminate the leak > > with the patch. One thing I was not certain about in an off-list > > discussion that led to this patch was that the patch reportedly > solved > > the leak only partially. I've been hoping to confirm that, but > > unfortunately I've not got any followup since then. > > > > So, basically, I believe the problem was solved, it would also help > if > > you could confirm it. > > Personally, I'm not convinced that it will make a difference outside of > Windows. The fix is to make sure a lock gets destroyed when done and > the > function exits. On Windows the lock gets created and memory is > allocated > for it outside of the function using it and needs to be explicitly > destroyed, but my understanding of pthreads was that this worked > differently on Unix and the lock structure would be automatically > destroyed when the function is exited since the lock was local to the > function. > > I'd be interested in seeing the results of this. > w > Danny I just loaded up the BIND 9.5.1 port on FreeBSD 7.0 AMD64 with threads. I don't see the prominent memory leak present on my system any longer. I lost track of this thread and think two different changes might have been made, however. One via the port and one in BIND itself, so I don't know which fix is actually in effect here, but the results seem good so far. Maybe Doug and Jinmei and comment on their respective changes just for the record. -Vinny ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
JINMEI Tatuya / ...@l@C#:H wrote: > At Tue, 9 Dec 2008 15:26:25 -0500, > Vinny Abello wrote: > >> Has anybody else tried this patch for you? I haven't had time to >> look into this at all. If nobody has tried this yet, I'll get around >> to it when I can and let you know the result. > > No one else other than by myself. It worked perfectly for me, i.e., I > could reproduce the problem and I could completely eliminate the leak > with the patch. One thing I was not certain about in an off-list > discussion that led to this patch was that the patch reportedly solved > the leak only partially. I've been hoping to confirm that, but > unfortunately I've not got any followup since then. > > So, basically, I believe the problem was solved, it would also help if > you could confirm it. Personally, I'm not convinced that it will make a difference outside of Windows. The fix is to make sure a lock gets destroyed when done and the function exits. On Windows the lock gets created and memory is allocated for it outside of the function using it and needs to be explicitly destroyed, but my understanding of pthreads was that this worked differently on Unix and the lock structure would be automatically destroyed when the function is exited since the lock was local to the function. I'd be interested in seeing the results of this. Danny ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
I've just committed the update to 9.5.1, including the ARCH=x86_64 trick (which I verified on one of our amd64 test platforms). Thanks for bringing this to my attention, and thanks to everyone that helped out. Doug ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
On Mon, 22 Dec 2008, ivan jr sy wrote: I have confirmed that the ARCH=x86_64 trick resolved the issues with my configuration. I have tested this with an authoritative and recursive dns/bind95 port with modified Makefile. I have not fully tested the acl.c and iptable.c since the patch suit my need. Thanks! Ok, thanks to all. Since 9.5.1 should be out soon, I'll test the port with that version and make any appropriate changes. Doug -- If you're never wrong, you're not trying hard enough ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
I have confirmed that the ARCH=x86_64 trick resolved the issues with my configuration. I have tested this with an authoritative and recursive dns/bind95 port with modified Makefile. I have not fully tested the acl.c and iptable.c since the patch suit my need. Thanks! --- On Tue, 12/23/08, Doug Barton wrote: > From: Doug Barton > Subject: Re: dnsperf and BIND memory consumption > To: "bind-us...@isc.org" > Date: Tuesday, December 23, 2008, 1:15 PM > JINMEI Tatuya / wrote: > > At Tue, 2 Dec 2008 00:35:32 -0500, > > Vinny Abello wrote: > > > >>For what it's worth, I just want to > contribute that I can > >>confirm this behavior on my systems as well. On > BIND 9.5.0-P2, > > > > From an off-list discussion, I found there was indeed > memory leak in > > the code of 9.5.0-P2 (so I was wrong in suspecting it > might be the > > FreeBSD thread library). > > > > Can you try the patched copied below to see whether it > solves the > > problem? > > Folks, > > I have been following this issue, but I haven't seen > any conclusive > statements that any of the proposed solutions are effective > and I > don't want to make a change to the port without knowing > for sure that > it's the right one (and unfortunately I don't have > an amd64 system to > do testing on). > > So, can someone please state affirmatively if the proposed > patch to > the port (ARCH= x86_64) results in the right > stuff in > config.log, AND results in a named that doesn't > experience the leaks? > > And can someone please state affirmatively that the patches > to acl.c > and iptable.c do the right thing, with or without the patch > to the port? > > > Thanks, > > Doug > > -- > > This .signature sanitized for your protection > > > ___ > bind-users mailing list > bind-users@lists.isc.org > https://lists.isc.org/mailman/listinfo/bind-users ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
At Mon, 22 Dec 2008 16:13:10 -0800, Doug Barton wrote: > And can someone please state affirmatively that the patches to acl.c > and iptable.c do the right thing, with or without the patch to the port? This patch completely fixed the hole in my test environment. With this patch you don't need the ARCH=x86_64 trick. I'm also pretty sure there's no bad side effect with this patch. So, basically, there's no reason not to apply this patch anyway. Alternatively, you may just want to wait until 9.5.1 ships, which will be released within a few days. --- JINMEI, Tatuya Internet Systems Consortium, Inc. ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
JINMEI Tatuya / wrote: > At Tue, 2 Dec 2008 00:35:32 -0500, > Vinny Abello wrote: > >> For what it's worth, I just want to contribute that I can >> confirm this behavior on my systems as well. On BIND 9.5.0-P2, > > From an off-list discussion, I found there was indeed memory leak in > the code of 9.5.0-P2 (so I was wrong in suspecting it might be the > FreeBSD thread library). > > Can you try the patched copied below to see whether it solves the > problem? Folks, I have been following this issue, but I haven't seen any conclusive statements that any of the proposed solutions are effective and I don't want to make a change to the port without knowing for sure that it's the right one (and unfortunately I don't have an amd64 system to do testing on). So, can someone please state affirmatively if the proposed patch to the port (ARCH= x86_64) results in the right stuff in config.log, AND results in a named that doesn't experience the leaks? And can someone please state affirmatively that the patches to acl.c and iptable.c do the right thing, with or without the patch to the port? Thanks, Doug -- This .signature sanitized for your protection ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
At Thu, 18 Dec 2008 17:14:06 +0300,
Dmitry Rybin wrote:
> > If you can allow the multiple views to share a single cache, one
> > possibility is to create a separate "localhost" view as the single
> > caching view and forward all recursive queries to that view:
>
> I add this lines in named.conf
> clients IP in this view can't resolve recursive queries.
It's hard to help with this too simplified problem description...
Note that my previous config example is just a conceptual one. You
should probably adjust it for your own environment.
> view "localhost" {
> match-clients { ::1; };
> };
One obvious point in this example is that named has to listen on ::1,
which is not the case for the default configuration.
---
JINMEI, Tatuya
Internet Systems Consortium, Inc.
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
JINMEI Tatuya / 神明達哉 wrote:
>>
>>> Have you any ideas how to limit memory usage?
>> Unfortunately not, unless you can consolidate the caching views to a
>> small number of views.
>
> If you can allow the multiple views to share a single cache, one
> possibility is to create a separate "localhost" view as the single
> caching view and forward all recursive queries to that view:
>
I add this lines in named.conf
clients IP in this view can't resolve recursive queries.
view "localhost" {
match-clients { ::1; };
};
view "view7" {
max-cache-ttl 0;
zone "." {
type forward;
forwarders { ::1; };
forward only;
};
};
Trying to use
view "localhost" {
match-clients { ::1; };
zone "." {
type hint;
file "named.root";
};
};
has the same result.
> view "localhost" {
> match-clients { ::1; };
> };
>
> view "view1" {
> match-clients { view1-clients; };
>
> max-cache-ttl 0; //disable caching to save memory
> zone "." {
> type forward;
> forwarders { ::1; };
> forward only;
> };
> // add other authoritative zones if necessary
> };
>
> view "view2" {
> match-clients { view2-clients; };
> max-cache-ttl 0;
> zone "." {
> type forward;
> forwarders { ::1; };
> forward only;
> };
> // add other authoritative zones if necessary
> };
>
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
JINMEI Tatuya / 神明達哉 wrote: > At Mon, 15 Dec 2008 09:53:23 +0300, > Dmitry Rybin wrote: > >> Thank's to JINMEI Tatuya for support. >> I have over 40 views, defined in named.conf, max-memory for cache - >> 32Mb. Named daemon allocate over 2 Gb per 24 hours of work. > > Each view has a separate cache DB. So if each of these 40 views > really needs to cache a certain amount of data, a footprint of 2GB is > not a surprising situation, even with a 32MB of max-cache-size for > each view. OK. I Just limit max-cache-size 16MB. 16MB * 50 views = over 800 MB of memory. How much total memory bind can accrue? -- ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
At Wed, 17 Dec 2008 17:07:12 -0800,
JINMEI Tatuya wrote:
>
> At Mon, 15 Dec 2008 09:53:23 +0300,
> Dmitry Rybin wrote:
>
> > Thank's to JINMEI Tatuya for support.
> > I have over 40 views, defined in named.conf, max-memory for cache -
> > 32Mb. Named daemon allocate over 2 Gb per 24 hours of work.
>
> Each view has a separate cache DB. So if each of these 40 views
> really needs to cache a certain amount of data, a footprint of 2GB is
> not a surprising situation, even with a 32MB of max-cache-size for
> each view.
>
> > Have you any ideas how to limit memory usage?
>
> Unfortunately not, unless you can consolidate the caching views to a
> small number of views.
If you can allow the multiple views to share a single cache, one
possibility is to create a separate "localhost" view as the single
caching view and forward all recursive queries to that view:
view "localhost" {
match-clients { ::1; };
};
view "view1" {
match-clients { view1-clients; };
max-cache-ttl 0; //disable caching to save memory
zone "." {
type forward;
forwarders { ::1; };
forward only;
};
// add other authoritative zones if necessary
};
view "view2" {
match-clients { view2-clients; };
max-cache-ttl 0;
zone "." {
type forward;
forwarders { ::1; };
forward only;
};
// add other authoritative zones if necessary
};
---
JINMEI, Tatuya
Internet Systems Consortium, Inc.
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
At Mon, 15 Dec 2008 09:53:23 +0300, Dmitry Rybin wrote: > Thank's to JINMEI Tatuya for support. > I have over 40 views, defined in named.conf, max-memory for cache - > 32Mb. Named daemon allocate over 2 Gb per 24 hours of work. Each view has a separate cache DB. So if each of these 40 views really needs to cache a certain amount of data, a footprint of 2GB is not a surprising situation, even with a 32MB of max-cache-size for each view. > Have you any ideas how to limit memory usage? Unfortunately not, unless you can consolidate the caching views to a small number of views. --- JINMEI, Tatuya Internet Systems Consortium, Inc. ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
Thank's to JINMEI Tatuya for support.
I have over 40 views, defined in named.conf, max-memory for cache -
32Mb. Named daemon allocate over 2 Gb per 24 hours of work.
Have you any ideas how to limit memory usage?
Dmitry Rybin wrote:
> max-cache-size 64M;
> # /usr/bin/limits -v 1200M /usr/local/sbin/named-test -c
> /etc/namedb/named.conf
>
> Over 10 minutes of work and core dumped:
>
> (gdb) bt
> #0 0x0058c3fc in thr_kill ()
> #1 0x005c5a68 in abort ()
> #2 0x00597af7 in malloc ()
> #3 0x0056645a in isc_mem_createx2 (init_max_size=0, target_size=0,
> memalloc=0x564400 , memfree=0x563320
> , arg=0x0,
> ctxp=0xcb29b978, flags=Variable "flags" is not available.
> ) at mem.c:790
> #4 0x00566730 in isc_mem_create (init_max_size=Variable
> "init_max_size" is not available.
> ) at mem.c:859
> #5 0x004d83ae in dns_resolver_create (view=0xca46e800,
> taskmgr=0x80828000,
> ntasks=31, socketmgr=Variable "socketmgr" is not available.
> ) at resolver.c:6514
> #6 0x004ee860 in dns_view_createresolver (view=0xca46e800,
> taskmgr=0x80828000,
> ntasks=31, socketmgr=0x8082b000, timermgr=0x80829000, options=0,
> dispatchmgr=0x8083c000,
> dispatchv4=0x864c9000, dispatchv6=0x864c9800) at view.c:580
> #7 0x0041bba2 in configure_view (view=0xca46e800,
> config=0x80abb4c0,
> vconfig=0x8085ada8, mctx=0x8080d140, actx=0x7eff8860,
> need_hints=isc_boolean_true)
> at server.c:1290
> #8 0x00420f42 in load_configuration (filename=Variable
> "filename" is not available.
> ) at server.c:3285
> #9 0x00422095 in loadconfig (server=0x8082f000) at server.c:4121
> #10 0x00422426 in reload (server=0x8082f000) at server.c:4141
> #11 0x004225c2 in ns_server_reloadcommand (server=0x8082f000,
> args=Variable "args" is not available.
> ) at server.c:4334
> #12 0x00407682 in ns_control_docommand (message=Variable
> "message" is not available.
> ) at control.c:102
> #13 0x0040a8b7 in control_recvmessage (task=0x80839000,
> event=Variable "event" is not available.
> ) at controlconf.c:456
> #14 0x0057052c in run (uap=Variable "uap" is not available.
> ) at task.c:862
> #15 0x005868a7 in thread_start ()
> #16 0x in ?? ()
> Cannot access memory at address 0x7eff9000
>
>
>
> JINMEI Tatuya / 神明達哉 wrote:
>
>> At Wed, 10 Dec 2008 15:50:22 +0300,
>> Dmitry Rybin wrote:
>>
>>> JINMEI Tatuya / 神明達哉 wrote:
>>>
At Tue, 09 Dec 2008 18:05:27 +0300,
Dmitry Rybin wrote:
> I test patch, add to bind95/Makefile
> .if (${ARCH} == "amd64")
> ARCH= x86_64
> .endif
>
Future versions of BIND9 will support amd64 in its configure script to
workaround the FreeBSD port for amd64.
Regarding the memory leak, I believe it's already solved in 9.5.1rc1
(even with threads and without atomic).
>>> I just make port bind 9.5.1rc1. It has same problem with memory leak.
>>> It grows from 670M on startup, to 1,4Gb after 20 minutes of work.
>>>
>> Can you first fall back to the vanilla 9.5.1rc1 (i.e., not FreeBSD
>> port) so that we can separate FreeBSD-port specific issue and BIND9
>> specific leak?
>>
>> Second, what if you stop named by 'rndc stop'? If there's memory leak
>> in BIND9, it normally detects it during a cleanup process and
>> indicates the bug by aborting (core dumping) itself.
>>
>> If it doesn't cause an abort, please then try the diagnosing I
>> suggested before:
>> http://marc.info/?l=bind-users&m=121811979629090&w=2
>>
>> To summarize it:
>>
>> 1. create a symbolic link from "/etc/malloc.conf" to "X":
>># ln -s X /etc/malloc.conf
>> 2. - start named with a moderate limitation of virtual memory size, e.g.
>># /usr/bin/limits -v 384m $path_to_named/named
>> (note that "384m" should be reasonably large compared with
>> max-cache-size. I'd suggest setting max-cache-size to 128M and
>> setting 'limits -v' to 512m).
>> 3. Then the named process will eventually abort itself with a core dump
>>due to malloc failure. Please show us the stack trace at that point.
>>Hopefully it will reveal the malloc call that keeps consuming memory.
>>
>> In fact, I myself successfully identified one leak in 9.5.0-P2 with
>> FreeBSD port this way.
>>
>> ---
>> JINMEI, Tatuya
>> Internet Systems Consortium, Inc.
>>
>
> ___
> bind-users mailing list
> bind-users@lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users
>
>
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
Thank's to JINMEI Tatuya for support.
I have over 40 views, defined in named.conf, max-memory for cache -
32Mb. Named daemon allocate over 2 Gb per 24 hours of work.
Have you any ideas how to limit memory usage?
Dmitry Rybin wrote:
> max-cache-size 64M;
> # /usr/bin/limits -v 1200M /usr/local/sbin/named-test -c
> /etc/namedb/named.conf
>
> Over 10 minutes of work and core dumped:
>
> (gdb) bt
> #0 0x0058c3fc in thr_kill ()
> #1 0x005c5a68 in abort ()
> #2 0x00597af7 in malloc ()
> #3 0x0056645a in isc_mem_createx2 (init_max_size=0, target_size=0,
> memalloc=0x564400 , memfree=0x563320
> , arg=0x0,
> ctxp=0xcb29b978, flags=Variable "flags" is not available.
> ) at mem.c:790
> #4 0x00566730 in isc_mem_create (init_max_size=Variable
> "init_max_size" is not available.
> ) at mem.c:859
> #5 0x004d83ae in dns_resolver_create (view=0xca46e800,
> taskmgr=0x80828000,
> ntasks=31, socketmgr=Variable "socketmgr" is not available.
> ) at resolver.c:6514
> #6 0x004ee860 in dns_view_createresolver (view=0xca46e800,
> taskmgr=0x80828000,
> ntasks=31, socketmgr=0x8082b000, timermgr=0x80829000, options=0,
> dispatchmgr=0x8083c000,
> dispatchv4=0x864c9000, dispatchv6=0x864c9800) at view.c:580
> #7 0x0041bba2 in configure_view (view=0xca46e800,
> config=0x80abb4c0,
> vconfig=0x8085ada8, mctx=0x8080d140, actx=0x7eff8860,
> need_hints=isc_boolean_true)
> at server.c:1290
> #8 0x00420f42 in load_configuration (filename=Variable
> "filename" is not available.
> ) at server.c:3285
> #9 0x00422095 in loadconfig (server=0x8082f000) at server.c:4121
> #10 0x00422426 in reload (server=0x8082f000) at server.c:4141
> #11 0x004225c2 in ns_server_reloadcommand (server=0x8082f000,
> args=Variable "args" is not available.
> ) at server.c:4334
> #12 0x00407682 in ns_control_docommand (message=Variable
> "message" is not available.
> ) at control.c:102
> #13 0x0040a8b7 in control_recvmessage (task=0x80839000,
> event=Variable "event" is not available.
> ) at controlconf.c:456
> #14 0x0057052c in run (uap=Variable "uap" is not available.
> ) at task.c:862
> #15 0x005868a7 in thread_start ()
> #16 0x in ?? ()
> Cannot access memory at address 0x7eff9000
>
>
>
> JINMEI Tatuya / 神明達哉 wrote:
>
>> At Wed, 10 Dec 2008 15:50:22 +0300,
>> Dmitry Rybin wrote:
>>
>>> JINMEI Tatuya / 神明達哉 wrote:
>>>
At Tue, 09 Dec 2008 18:05:27 +0300,
Dmitry Rybin wrote:
> I test patch, add to bind95/Makefile
> .if (${ARCH} == "amd64")
> ARCH= x86_64
> .endif
>
Future versions of BIND9 will support amd64 in its configure script to
workaround the FreeBSD port for amd64.
Regarding the memory leak, I believe it's already solved in 9.5.1rc1
(even with threads and without atomic).
>>> I just make port bind 9.5.1rc1. It has same problem with memory leak.
>>> It grows from 670M on startup, to 1,4Gb after 20 minutes of work.
>>>
>> Can you first fall back to the vanilla 9.5.1rc1 (i.e., not FreeBSD
>> port) so that we can separate FreeBSD-port specific issue and BIND9
>> specific leak?
>>
>> Second, what if you stop named by 'rndc stop'? If there's memory leak
>> in BIND9, it normally detects it during a cleanup process and
>> indicates the bug by aborting (core dumping) itself.
>>
>> If it doesn't cause an abort, please then try the diagnosing I
>> suggested before:
>> http://marc.info/?l=bind-users&m=121811979629090&w=2
>>
>> To summarize it:
>>
>> 1. create a symbolic link from "/etc/malloc.conf" to "X":
>># ln -s X /etc/malloc.conf
>> 2. - start named with a moderate limitation of virtual memory size, e.g.
>># /usr/bin/limits -v 384m $path_to_named/named
>> (note that "384m" should be reasonably large compared with
>> max-cache-size. I'd suggest setting max-cache-size to 128M and
>> setting 'limits -v' to 512m).
>> 3. Then the named process will eventually abort itself with a core dump
>>due to malloc failure. Please show us the stack trace at that point.
>>Hopefully it will reveal the malloc call that keeps consuming memory.
>>
>> In fact, I myself successfully identified one leak in 9.5.0-P2 with
>> FreeBSD port this way.
>>
>> ---
>> JINMEI, Tatuya
>> Internet Systems Consortium, Inc.
>>
>
> ___
> bind-users mailing list
> bind-users@lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users
>
>
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
JINMEI Tatuya / 神明達哉 wrote: > At Thu, 11 Dec 2008 11:25:42 +0300, > Dmitry Rybin wrote: > >> OK. I just make bind from src with ./configure --enable-threads & gcc >> option -static. >> >> file /usr/local/sbin/named-test >> /usr/local/sbin/named-test: ELF 64-bit LSB executable, x86-64, version 1 >> (FreeBSD), for FreeBSD 7.1 (701100), statically linked, FreeBSD-style, >> not stripped > > please let me check some points: > > 1. you used 9.5.1rc1 without any patch, right? > 2. did you try 'rndc stop' at some point? If so, did named stop >cleanly or did it abort itself? > 3. were you periodically reloading the server during the test? I'm >not sure if this is coincidence but the self-abort happened while >reload operation in your both cases. > 4. if the answer to question#2 is yes, is it possible to not reload >the server and see if it changes anything? > 5. is it possible to install (if not yet) libxml2 port to your system >and enable statistics-channels? then you can see more detailed >information about how named uses memory. > 1. Yes. With STD_CDEFINES='-DFD_SETSIZE=16384' --enable-threads and static linking. 2. Cleanly 3. No. Then reloading result is the same. 4. 5. Yes. Building named with libxml2 done. Say me private your's IP to make access to it. ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
At Thu, 11 Dec 2008 11:25:42 +0300, Dmitry Rybin wrote: > OK. I just make bind from src with ./configure --enable-threads & gcc > option -static. > > file /usr/local/sbin/named-test > /usr/local/sbin/named-test: ELF 64-bit LSB executable, x86-64, version 1 > (FreeBSD), for FreeBSD 7.1 (701100), statically linked, FreeBSD-style, > not stripped please let me check some points: 1. you used 9.5.1rc1 without any patch, right? 2. did you try 'rndc stop' at some point? If so, did named stop cleanly or did it abort itself? 3. were you periodically reloading the server during the test? I'm not sure if this is coincidence but the self-abort happened while reload operation in your both cases. 4. if the answer to question#2 is yes, is it possible to not reload the server and see if it changes anything? 5. is it possible to install (if not yet) libxml2 port to your system and enable statistics-channels? then you can see more detailed information about how named uses memory. --- JINMEI, Tatuya Internet Systems Consortium, Inc. ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
max-cache-size 64M;
# /usr/bin/limits -v 1200M /usr/local/sbin/named-test -c
/etc/namedb/named.conf
Over 10 minutes of work and core dumped:
(gdb) bt
#0 0x0058c3fc in thr_kill ()
#1 0x005c5a68 in abort ()
#2 0x00597af7 in malloc ()
#3 0x0056645a in isc_mem_createx2 (init_max_size=0, target_size=0,
memalloc=0x564400 , memfree=0x563320
, arg=0x0,
ctxp=0xcb29b978, flags=Variable "flags" is not available.
) at mem.c:790
#4 0x00566730 in isc_mem_create (init_max_size=Variable
"init_max_size" is not available.
) at mem.c:859
#5 0x004d83ae in dns_resolver_create (view=0xca46e800,
taskmgr=0x80828000,
ntasks=31, socketmgr=Variable "socketmgr" is not available.
) at resolver.c:6514
#6 0x004ee860 in dns_view_createresolver (view=0xca46e800,
taskmgr=0x80828000,
ntasks=31, socketmgr=0x8082b000, timermgr=0x80829000, options=0,
dispatchmgr=0x8083c000,
dispatchv4=0x864c9000, dispatchv6=0x864c9800) at view.c:580
#7 0x0041bba2 in configure_view (view=0xca46e800,
config=0x80abb4c0,
vconfig=0x8085ada8, mctx=0x8080d140, actx=0x7eff8860,
need_hints=isc_boolean_true)
at server.c:1290
#8 0x00420f42 in load_configuration (filename=Variable
"filename" is not available.
) at server.c:3285
#9 0x00422095 in loadconfig (server=0x8082f000) at server.c:4121
#10 0x00422426 in reload (server=0x8082f000) at server.c:4141
#11 0x004225c2 in ns_server_reloadcommand (server=0x8082f000,
args=Variable "args" is not available.
) at server.c:4334
#12 0x00407682 in ns_control_docommand (message=Variable
"message" is not available.
) at control.c:102
#13 0x0040a8b7 in control_recvmessage (task=0x80839000,
event=Variable "event" is not available.
) at controlconf.c:456
#14 0x0057052c in run (uap=Variable "uap" is not available.
) at task.c:862
#15 0x005868a7 in thread_start ()
#16 0x in ?? ()
Cannot access memory at address 0x7eff9000
JINMEI Tatuya / 神明達哉 wrote:
> At Wed, 10 Dec 2008 15:50:22 +0300,
> Dmitry Rybin <[EMAIL PROTECTED]> wrote:
>> JINMEI Tatuya / 神明達哉 wrote:
>>> At Tue, 09 Dec 2008 18:05:27 +0300,
>>> Dmitry Rybin <[EMAIL PROTECTED]> wrote:
>>>
I test patch, add to bind95/Makefile
.if (${ARCH} == "amd64")
ARCH= x86_64
.endif
>>> Future versions of BIND9 will support amd64 in its configure script to
>>> workaround the FreeBSD port for amd64.
>>>
>>> Regarding the memory leak, I believe it's already solved in 9.5.1rc1
>>> (even with threads and without atomic).
>> I just make port bind 9.5.1rc1. It has same problem with memory leak.
>> It grows from 670M on startup, to 1,4Gb after 20 minutes of work.
>
> Can you first fall back to the vanilla 9.5.1rc1 (i.e., not FreeBSD
> port) so that we can separate FreeBSD-port specific issue and BIND9
> specific leak?
>
> Second, what if you stop named by 'rndc stop'? If there's memory leak
> in BIND9, it normally detects it during a cleanup process and
> indicates the bug by aborting (core dumping) itself.
>
> If it doesn't cause an abort, please then try the diagnosing I
> suggested before:
> http://marc.info/?l=bind-users&m=121811979629090&w=2
>
> To summarize it:
>
> 1. create a symbolic link from "/etc/malloc.conf" to "X":
># ln -s X /etc/malloc.conf
> 2. - start named with a moderate limitation of virtual memory size, e.g.
># /usr/bin/limits -v 384m $path_to_named/named
> (note that "384m" should be reasonably large compared with
> max-cache-size. I'd suggest setting max-cache-size to 128M and
> setting 'limits -v' to 512m).
> 3. Then the named process will eventually abort itself with a core dump
>due to malloc failure. Please show us the stack trace at that point.
>Hopefully it will reveal the malloc call that keeps consuming memory.
>
> In fact, I myself successfully identified one leak in 9.5.0-P2 with
> FreeBSD port this way.
>
> ---
> JINMEI, Tatuya
> Internet Systems Consortium, Inc.
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
OK. I just make bind from src with ./configure --enable-threads & gcc
option -static.
file /usr/local/sbin/named-test
/usr/local/sbin/named-test: ELF 64-bit LSB executable, x86-64, version 1
(FreeBSD), for FreeBSD 7.1 (701100), statically linked, FreeBSD-style,
not stripped
fresh system (yesterday cvsup to RELENG_7)
$ uname -a
FreeBSD XXX.XXX.XX 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #1: Wed Dec 10
17:07:03 MSK 2008 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/XXX amd64
2.
max-cache-size 128M;
start:
/usr/bin/limits -c 1000M -v 500M /usr/local/sbin/named-test -c
/etc/namedb/named.conf
$ gdb -c named-test.core -se /usr/local/sbin/named-test
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Core was generated by `named-test'.
Program terminated with signal 6, Aborted.
#0 0x0058c3fc in thr_kill ()
[New Thread 0x80902f00 (LWP 100404)]
[New Thread 0x80902d80 (LWP 100400)]
[New Thread 0x80902c00 (LWP 100356)]
[New Thread 0x80902a80 (LWP 100318)]
[New Thread 0x80902900 (LWP 100239)]
[New Thread 0x80902780 (LWP 100237)]
[New Thread 0x80902600 (LWP 100222)]
[New Thread 0x80902480 (LWP 100209)]
[New Thread 0x80902300 (LWP 100175)]
[New Thread 0x80902180 (LWP 100092)]
[New Thread 0x80803180 (LWP 100177)]
(gdb) bt
#0 0x0058c3fc in thr_kill ()
#1 0x005c5a68 in abort ()
#2 0x00597af7 in malloc ()
#3 0x00564247 in mem_getunlocked (ctx=0x8080d140, size=94) at
mem.c:385
#4 0x00564b68 in isc__mem_get (ctx=0x8080d140, size=94,
file=0x61bd78 "rbt.c", line=1425) at mem.c:1096
#5 0x00480121 in create_node (mctx=0x8080d140,
name=0x7fbfcff0, nodep=0x7fbfd2e0) at rbt.c:1424
#6 0x0048080f in dns_rbt_addnode (rbt=0x80a925e8,
name=0x88cf2020, nodep=0x7fbfd3a8) at rbt.c:624
#7 0x0048be53 in loading_addrdataset (arg=0x94b07ff0,
name=0x88cf2020, rdataset=0x7fbfd810) at rbtdb.c:5657
#8 0x00463761 in commit (callbacks=0x7fbfe5c0,
lctx=0x80834000, head=0x7fbfe480, owner=0x88cf2020,
source=0x94c2afd8 "co/brand.bak", line=611215) at master.c:2729
#9 0x004668df in load_text (lctx=0x80834000) at master.c:1427
#10 0x0046b61b in dns_master_loadfile2 (master_file=0x878a7098
"co/broad.bak", top=Variable "top" is not available.
)
at master.c:2350
#11 0x00506126 in zone_load (zone=0x878ec000, flags=Variable
"flags" is not available.
) at zone.c:1504
#12 0x005082b9 in load (zone=Variable "zone" is not available.
) at zt.c:246
#13 0x00507ec2 in dns_zt_apply2 (zt=Variable "zt" is not available.
) at zt.c:379
#14 0x00508144 in dns_zt_load (zt=0x86adb750,
stop=isc_boolean_false) at zt.c:237
#15 0x004223c7 in load_zones (server=0x8082f000,
stop=isc_boolean_false) at server.c:3659
#16 0x004232fc in run_server (task=Variable "task" is not available.
) at server.c:3751
#17 0x0057052c in run (uap=Variable "uap" is not available.
) at task.c:862
#18 0x005868a7 in thread_start ()
#19 0x in ?? ()
Cannot access memory at address 0x7fbff000
At normal situation after startup memory usage over 700 MB.
-
JINMEI Tatuya / 神明達哉 wrote:
> At Wed, 10 Dec 2008 15:50:22 +0300,
> Dmitry Rybin <[EMAIL PROTECTED]> wrote:
>> JINMEI Tatuya / 神明達哉 wrote:
>>> At Tue, 09 Dec 2008 18:05:27 +0300,
>>> Dmitry Rybin <[EMAIL PROTECTED]> wrote:
>>>
I test patch, add to bind95/Makefile
.if (${ARCH} == "amd64")
ARCH= x86_64
.endif
>>> Future versions of BIND9 will support amd64 in its configure script to
>>> workaround the FreeBSD port for amd64.
>>>
>>> Regarding the memory leak, I believe it's already solved in 9.5.1rc1
>>> (even with threads and without atomic).
>> I just make port bind 9.5.1rc1. It has same problem with memory leak.
>> It grows from 670M on startup, to 1,4Gb after 20 minutes of work.
>
> Can you first fall back to the vanilla 9.5.1rc1 (i.e., not FreeBSD
> port) so that we can separate FreeBSD-port specific issue and BIND9
> specific leak?
>
> Second, what if you stop named by 'rndc stop'? If there's memory leak
> in BIND9, it normally detects it during a cleanup process and
> indicates the bug by aborting (core dumping) itself.
>
> If it doesn't cause an abort, please then try the diagnosing I
> suggested before:
> http://marc.info/?l=bind-users&m=121811979629090&w=2
>
> To summarize it:
>
> 1. create a symbolic link from "/etc/malloc.conf" to "X":
># ln -s X /etc/malloc.conf
> 2. - start named with a moderate limitation of virtual memory size, e.g.
># /usr/bin/limits -v 384m $path_to_named/named
> (note that "384m" should be reasonably lar
Re: dnsperf and BIND memory consumption
At Wed, 10 Dec 2008 15:50:22 +0300,
Dmitry Rybin <[EMAIL PROTECTED]> wrote:
>
> JINMEI Tatuya / 神明達哉 wrote:
> > At Tue, 09 Dec 2008 18:05:27 +0300,
> > Dmitry Rybin <[EMAIL PROTECTED]> wrote:
> >
> >> I test patch, add to bind95/Makefile
> >> .if (${ARCH} == "amd64")
> >> ARCH= x86_64
> >> .endif
> >
> > Future versions of BIND9 will support amd64 in its configure script to
> > workaround the FreeBSD port for amd64.
> >
> > Regarding the memory leak, I believe it's already solved in 9.5.1rc1
> > (even with threads and without atomic).
>
> I just make port bind 9.5.1rc1. It has same problem with memory leak.
> It grows from 670M on startup, to 1,4Gb after 20 minutes of work.
Can you first fall back to the vanilla 9.5.1rc1 (i.e., not FreeBSD
port) so that we can separate FreeBSD-port specific issue and BIND9
specific leak?
Second, what if you stop named by 'rndc stop'? If there's memory leak
in BIND9, it normally detects it during a cleanup process and
indicates the bug by aborting (core dumping) itself.
If it doesn't cause an abort, please then try the diagnosing I
suggested before:
http://marc.info/?l=bind-users&m=121811979629090&w=2
To summarize it:
1. create a symbolic link from "/etc/malloc.conf" to "X":
# ln -s X /etc/malloc.conf
2. - start named with a moderate limitation of virtual memory size, e.g.
# /usr/bin/limits -v 384m $path_to_named/named
(note that "384m" should be reasonably large compared with
max-cache-size. I'd suggest setting max-cache-size to 128M and
setting 'limits -v' to 512m).
3. Then the named process will eventually abort itself with a core dump
due to malloc failure. Please show us the stack trace at that point.
Hopefully it will reveal the malloc call that keeps consuming memory.
In fact, I myself successfully identified one leak in 9.5.0-P2 with
FreeBSD port this way.
---
JINMEI, Tatuya
Internet Systems Consortium, Inc.
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
Hi, Ivan!
Methodology test - one of production dns servers. It has over 900
authoritative zones and 50'000 online clients via recursive. 1
queries per 8,9 second. Also server use views.
=== named.conf ===
key "rndc-key" {
algorithm hmac-md5;
secret "XXx";
};
controls {
inet XXX.XXX.XXX.XXX port 1953
allow { 127.0.0.1; }
keys { "rndc-key"; };
};
options {
directory "/etc/namedb";
max-cache-size 256M;
max-cache-ttl 3600;
max-ncache-ttl 1800;
cleaning-interval 10;
acache-enable yes;
acache-cleaning-interval 30;
max-acache-size 256M;
transfers-in 1;
transfers-out 1;
transfers-per-ns 100;
recursive-clients 1;
clients-per-query 80;
max-clients-per-query 100 ;
listen-on {
127.0.0.1;
XXX.XXX.XXX.XXX;
};
rrset-order {order cyclic;};
pid-file "/var/run/named/pid";
version "DNS";
minimal-responses yes;
allow-recursion {
XXX.XXX.XXX.XXX/XXX;
};
allow-transfer {
XXX.XXX.XXX.XXX/XXX;
};
allow-notify {
XXX.XXX.XXX.XXX/XXX;
};
};
view "world" {
zone "." {
type hint;
file "named.root";
};
zone "0.0.127.IN-ADDR.ARPA" {
type master;
file "localhost.rev";
};
include "xx.conf";
};
view "v0"{
match-destinations { XXX.XXX.XXX.XXX; };
match-clients {
10.10.0.0/16;
};
include "v/v0.conf";
};
=== named.conf ===
50 minutes of work... 1563M in mem. LA 0.80, 0.82, 0.79. 8 threads.
$ldd /usr/local/sbin/named
/usr/local/sbin/named:
libcrypto.so.5 => /lib/libcrypto.so.5 (0x807bf000)
libthr.so.3 => /lib/libthr.so.3 (0x80a51000)
libc.so.7 => /lib/libc.so.7 (0x80b67000)
ivan jr sy wrote:
> Hi,
>
> is it possible to see your named.conf
> what is the methodology of the test? is it for authoritative queries?
> recursive? or both? at the same time?
>
> my patch for the port is the same as yours...
>
> thanks!
> ===
> ..if ${ARCH} == "amd64"
> ARCH=x86_64
> ..endif
>
>
>
> --- On Thu, 12/11/08, Dmitry Rybin <[EMAIL PROTECTED]> wrote:
>
>> From: Dmitry Rybin <[EMAIL PROTECTED]>
>> Subject: Re: dnsperf and BIND memory consumption
>> To: "JINMEI Tatuya / 神明達哉" <[EMAIL PROTECTED]>
>> Cc: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>, "[EMAIL PROTECTED]" <[EMAIL
>> PROTECTED]>
>> Date: Thursday, December 11, 2008, 1:50 AM
>> JINMEI Tatuya / 神明達哉 wrote:
>>> At Tue, 09 Dec 2008 18:05:27 +0300,
>>> Dmitry Rybin <[EMAIL PROTECTED]> wrote:
>>>
>>>> I test patch, add to bind95/Makefile
>>>> .if (${ARCH} == "amd64")
>>>> ARCH= x86_64
>>>> .endif
>>> Future versions of BIND9 will support amd64 in its
>> configure script to
>>> workaround the FreeBSD port for amd64.
>>>
>>> Regarding the memory leak, I believe it's already
>> solved in 9.5.1rc1
>>> (even with threads and without atomic).
>> I just make port bind 9.5.1rc1. It has same problem with
>> memory leak.
>> It grows from 670M on startup, to 1,4Gb after 20 minutes of
>> work.
>>
>> grep x86 work/bind-9.5.1rc1/config.log
>> ISC_ARCH_DIR='x86_32'
>> build='x86_64-portbld-freebsd7.0'
>> build_alias='x86_64-portbld-freebsd7.0'
>> build_cpu='x86_64'
>> host='x86_64-portbld-freebsd7.0'
>> host_cpu='x86_64'
>>
>> ___
>> bind-users mailing list
>> bind-users@lists.isc.org
>> https://lists.isc.org/mailman/listinfo/bind-users
>
>
>
>
--
Рыбин Дмитрий
Управление магистральной сети
Отдел Информационных Систем
Руководитель группы АВР
Corbina Telecom
Tel: +7(495) 728-4000
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
Hi,
is it possible to see your named.conf
what is the methodology of the test? is it for authoritative queries?
recursive? or both? at the same time?
my patch for the port is the same as yours...
thanks!
===
.if ${ARCH} == "amd64"
ARCH=x86_64
.endif
--- On Thu, 12/11/08, Dmitry Rybin <[EMAIL PROTECTED]> wrote:
> From: Dmitry Rybin <[EMAIL PROTECTED]>
> Subject: Re: dnsperf and BIND memory consumption
> To: "JINMEI Tatuya / 神明達哉" <[EMAIL PROTECTED]>
> Cc: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>, "[EMAIL PROTECTED]" <[EMAIL
> PROTECTED]>
> Date: Thursday, December 11, 2008, 1:50 AM
> JINMEI Tatuya / 神明達哉 wrote:
> > At Tue, 09 Dec 2008 18:05:27 +0300,
> > Dmitry Rybin <[EMAIL PROTECTED]> wrote:
> >
> >> I test patch, add to bind95/Makefile
> >> .if (${ARCH} == "amd64")
> >> ARCH= x86_64
> >> .endif
> >
> > Future versions of BIND9 will support amd64 in its
> configure script to
> > workaround the FreeBSD port for amd64.
> >
> > Regarding the memory leak, I believe it's already
> solved in 9.5.1rc1
> > (even with threads and without atomic).
>
> I just make port bind 9.5.1rc1. It has same problem with
> memory leak.
> It grows from 670M on startup, to 1,4Gb after 20 minutes of
> work.
>
> grep x86 work/bind-9.5.1rc1/config.log
> ISC_ARCH_DIR='x86_32'
> build='x86_64-portbld-freebsd7.0'
> build_alias='x86_64-portbld-freebsd7.0'
> build_cpu='x86_64'
> host='x86_64-portbld-freebsd7.0'
> host_cpu='x86_64'
>
> ___
> bind-users mailing list
> bind-users@lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
JINMEI Tatuya / 神明達哉 wrote:
> At Tue, 09 Dec 2008 18:05:27 +0300,
> Dmitry Rybin <[EMAIL PROTECTED]> wrote:
>
>> I test patch, add to bind95/Makefile
>> .if (${ARCH} == "amd64")
>> ARCH= x86_64
>> .endif
>
> Future versions of BIND9 will support amd64 in its configure script to
> workaround the FreeBSD port for amd64.
>
> Regarding the memory leak, I believe it's already solved in 9.5.1rc1
> (even with threads and without atomic).
I just make port bind 9.5.1rc1. It has same problem with memory leak.
It grows from 670M on startup, to 1,4Gb after 20 minutes of work.
grep x86 work/bind-9.5.1rc1/config.log
ISC_ARCH_DIR='x86_32'
build='x86_64-portbld-freebsd7.0'
build_alias='x86_64-portbld-freebsd7.0'
build_cpu='x86_64'
host='x86_64-portbld-freebsd7.0'
host_cpu='x86_64'
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
Memory statistic
start - 570M
1 min - 913M
2 min - 958M
3 min - 1092M
4 min - 1074M
5 min - 1082M
10 min - 1217M
15 min - 1234M
60 min - 1513M
max-cache-size 800M;
Port installed only with Threads parameter, and patch in Makefile
.if (${ARCH} == "amd64")
ARCH= x86_64
.endif
===
# ps axw|grep named
/usr/local/sbin/named -t /var/named -u bind -c /etc/namedb/named.conf -t
/var/named -u bind
===
$ rndc status
version: 9.5.0-P2 (Unknown DNS1)
number of zones: 899
debug level: 0
xfers running: 0
xfers deferred: 0
soa queries in progress: 2
query logging is OFF
recursive clients: 286/9900/1
tcp clients: 0/100
server is up and running
===
(port installed)
$ldd /usr/local/sbin/named
/usr/local/sbin/named:
libcrypto.so.5 => /lib/libcrypto.so.5 (0x807bb000)
libthr.so.3 => /lib/libthr.so.3 (0x80a4d000)
libc.so.7 => /lib/libc.so.7 (0x80b63000)
(system standart)
$ldd /usr/sbin/named
/usr/sbin/named:
libcrypto.so.5 => /lib/libcrypto.so.5 (0x807a9000)
libthr.so.3 => /lib/libthr.so.3 (0x80a3b000)
libc.so.7 => /lib/libc.so.7 (0x80b51000)
===
ivan jr sy wrote:
> Hi
>
> can you verify if you're using the newly installed named.
>
> did you configure your options to replace the base?
>
> can you give us:
>
> ldd /usr/sbin/named
> ldd /usr/local/sbin/named
>
> to my understanding, there should be no memory leak issue at all if you
> disable threads..
>
> this post has always been directed to the concern of FreeBSD + amd64 platform
> + FreeBSD port dns/bind95 (BIND 9.5.0-P2) + threading enabled
>
> thanks!
>
> --- On Wed, 12/10/08, Dmitry Rybin <[EMAIL PROTECTED]> wrote:
>
>> From: Dmitry Rybin <[EMAIL PROTECTED]>
>> Subject: Re: dnsperf and BIND memory consumption
>> To: "Vinny Abello" <[EMAIL PROTECTED]>
>> Cc: "JINMEI Tatuya / 神明達哉" <[EMAIL PROTECTED]>, "[EMAIL PROTECTED]" <[EMAIL
>> PROTECTED]>, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>> Date: Wednesday, December 10, 2008, 4:05 AM
>> Hello!
>>
>> I test patch, add to bind95/Makefile
>> .if (${ARCH} == "amd64")
>> ARCH= x86_64
>> .endif
>>
>> work/bind-9.5.0-P2/config.log
>> uname -m = amd64
>> /usr/bin/uname -p = amd64
>> Target: amd64-undermydesk-freebsd
>> Configured with: FreeBSD/amd64 system compiler
>> ISC_ARCH_DIR='x86_32'
>> build='x86_64-portbld-freebsd7.0'
>> build_alias='x86_64-portbld-freebsd7.0'
>> build_cpu='x86_64'
>> host='x86_64-portbld-freebsd7.0'
>> host_cpu='x86_64'
>>
>> I didn't find any affect, memory leak very quickly with
>> threads support,
>> and slowly without threads.
>>
>> FreeBSD xxx 7.0-STABLE FreeBSD 7.0-STABLE #0: Wed Jul 2
>> 14:18:35 MSD
>> 2008 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/H1 amd64
>>
>>
>> Vinny Abello wrote:
>>
>>>>> so does this memory leak only occur if
>>>>> @ISC_ARCH_DIR@ is "noatomic" under
>> FreeBSD amd64?
>>>>> and not when its "x86_32" ?
>>>> First off, note that I have no explicit evidence
>> of memory leak. But
>>>> *if there is indeed leak in the FreeBSD pthread
>> library*, the key is
>>>> "noatomic". With this configuration
>> named will call pthread
>>>> locks/unlocks much, much heavier, so the problem
>> may be observable
>>>> more clearly. named still uses pthread locks Even
>> with x86_32, so it
>>>> may just be leaking memory more slowly.
>>>>
>>>> Again, everything is just a guess and could be
>> wrong. We should seek
>>>> advice from someone who knows FreeBSD library
>> well.
>>> Just out of curiosity, why in theory is this not seen
>> in prior versions of BIND such as 9.4.2-P2 or 9.4.3 on the
>> same FreeBSD 7.0 AMD64 platforms with threading enabled in
>> BIND?
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
RE: dnsperf and BIND memory consumption
> -Original Message- > From: JINMEI Tatuya / 神明達哉 [mailto:[EMAIL PROTECTED] > Sent: Tuesday, December 09, 2008 3:38 PM > To: Vinny Abello > Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] > Subject: Re: dnsperf and BIND memory consumption > > At Tue, 9 Dec 2008 15:26:25 -0500, > Vinny Abello <[EMAIL PROTECTED]> wrote: > > > Has anybody else tried this patch for you? I haven't had time to > > look into this at all. If nobody has tried this yet, I'll get around > > to it when I can and let you know the result. > > No one else other than by myself. It worked perfectly for me, i.e., I > could reproduce the problem and I could completely eliminate the leak > with the patch. One thing I was not certain about in an off-list > discussion that led to this patch was that the patch reportedly solved > the leak only partially. I've been hoping to confirm that, but > unfortunately I've not got any followup since then. > > So, basically, I believe the problem was solved, it would also help if > you could confirm it. > > Thanks, > > --- > JINMEI, Tatuya > Internet Systems Consortium, Inc. Jinmei, I'll try to confirm when I have some spare time and let you know. -Vinny ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
At Tue, 9 Dec 2008 15:26:25 -0500, Vinny Abello <[EMAIL PROTECTED]> wrote: > Has anybody else tried this patch for you? I haven't had time to > look into this at all. If nobody has tried this yet, I'll get around > to it when I can and let you know the result. No one else other than by myself. It worked perfectly for me, i.e., I could reproduce the problem and I could completely eliminate the leak with the patch. One thing I was not certain about in an off-list discussion that led to this patch was that the patch reportedly solved the leak only partially. I've been hoping to confirm that, but unfortunately I've not got any followup since then. So, basically, I believe the problem was solved, it would also help if you could confirm it. Thanks, --- JINMEI, Tatuya Internet Systems Consortium, Inc. ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
RE: dnsperf and BIND memory consumption
Hi Jinmei,
Has anybody else tried this patch for you? I haven't had time to look into this
at all. If nobody has tried this yet, I'll get around to it when I can and let
you know the result.
-Vinny
> -Original Message-
> From: JINMEI Tatuya / 神明達哉 [mailto:[EMAIL PROTECTED]
> Sent: Tuesday, December 02, 2008 1:06 AM
> To: Vinny Abello
> Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]
> Subject: Re: dnsperf and BIND memory consumption
>
> At Tue, 2 Dec 2008 00:35:32 -0500,
> Vinny Abello <[EMAIL PROTECTED]> wrote:
>
> > For what it's worth, I just want to contribute that I can
> > confirm this behavior on my systems as well. On BIND 9.5.0-P2,
>
> From an off-list discussion, I found there was indeed memory leak in
> the code of 9.5.0-P2 (so I was wrong in suspecting it might be the
> FreeBSD thread library).
>
> Can you try the patched copied below to see whether it solves the
> problem?
>
> Note: this leak was already fixed in 9.5.1rc1:
>
> 2435. [bug] Fixed an ACL memory leak affecting win32.
>
> while the change description seems to indicate it's win32 specific,
> it can actually happen any build with threads and without atomic
> operations.
>
> ---
> JINMEI, Tatuya
> Internet Systems Consortium, Inc.
>
> Index: acl.c
> ===
> RCS file: /proj/cvs/prod/bind9/lib/dns/acl.c,v
> retrieving revision 1.37.2.7
> diff -u -r1.37.2.7 acl.c
> --- acl.c 29 Apr 2008 01:04:14 - 1.37.2.7
> +++ acl.c 1 Dec 2008 03:17:36 -
> @@ -217,6 +217,7 @@
>
> /* Search radix. */
> result = isc_radix_search(acl->iptable->radix, &node, &pfx);
> + isc_refcount_destroy(&pfx.refcount);
>
> /* Found a match. */
> if (result == ISC_R_SUCCESS && node != NULL) {
> Index: iptable.c
> ===
> RCS file: /proj/cvs/prod/bind9/lib/dns/iptable.c,v
> retrieving revision 1.5.46.3
> diff -u -r1.5.46.3 iptable.c
> --- iptable.c 21 Jan 2008 21:02:24 - 1.5.46.3
> +++ iptable.c 1 Dec 2008 19:25:27 -
> @@ -74,6 +74,7 @@
> family = bitlen ? pfx.family : AF_INET;
>
> result = isc_radix_insert(tab->radix, &node, NULL, &pfx);
> + isc_refcount_destroy(&pfx.refcount);
>
> if (result != ISC_R_SUCCESS)
> return(result);
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
At Tue, 09 Dec 2008 18:05:27 +0300,
Dmitry Rybin <[EMAIL PROTECTED]> wrote:
> I test patch, add to bind95/Makefile
> .if (${ARCH} == "amd64")
> ARCH= x86_64
> .endif
Future versions of BIND9 will support amd64 in its configure script to
workaround the FreeBSD port for amd64.
Regarding the memory leak, I believe it's already solved in 9.5.1rc1
(even with threads and without atomic).
---
JINMEI, Tatuya
Internet Systems Consortium, Inc.
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
Hi
can you verify if you're using the newly installed named.
did you configure your options to replace the base?
can you give us:
ldd /usr/sbin/named
ldd /usr/local/sbin/named
to my understanding, there should be no memory leak issue at all if you disable
threads..
this post has always been directed to the concern of FreeBSD + amd64 platform +
FreeBSD port dns/bind95 (BIND 9.5.0-P2) + threading enabled
thanks!
--- On Wed, 12/10/08, Dmitry Rybin <[EMAIL PROTECTED]> wrote:
> From: Dmitry Rybin <[EMAIL PROTECTED]>
> Subject: Re: dnsperf and BIND memory consumption
> To: "Vinny Abello" <[EMAIL PROTECTED]>
> Cc: "JINMEI Tatuya / 神明達哉" <[EMAIL PROTECTED]>, "[EMAIL PROTECTED]" <[EMAIL
> PROTECTED]>, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Date: Wednesday, December 10, 2008, 4:05 AM
> Hello!
>
> I test patch, add to bind95/Makefile
> .if (${ARCH} == "amd64")
> ARCH= x86_64
> .endif
>
> work/bind-9.5.0-P2/config.log
> uname -m = amd64
> /usr/bin/uname -p = amd64
> Target: amd64-undermydesk-freebsd
> Configured with: FreeBSD/amd64 system compiler
> ISC_ARCH_DIR='x86_32'
> build='x86_64-portbld-freebsd7.0'
> build_alias='x86_64-portbld-freebsd7.0'
> build_cpu='x86_64'
> host='x86_64-portbld-freebsd7.0'
> host_cpu='x86_64'
>
> I didn't find any affect, memory leak very quickly with
> threads support,
> and slowly without threads.
>
> FreeBSD xxx 7.0-STABLE FreeBSD 7.0-STABLE #0: Wed Jul 2
> 14:18:35 MSD
> 2008 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/H1 amd64
>
>
> Vinny Abello wrote:
>
> >>> so does this memory leak only occur if
> >>> @ISC_ARCH_DIR@ is "noatomic" under
> FreeBSD amd64?
> >>> and not when its "x86_32" ?
> >> First off, note that I have no explicit evidence
> of memory leak. But
> >> *if there is indeed leak in the FreeBSD pthread
> library*, the key is
> >> "noatomic". With this configuration
> named will call pthread
> >> locks/unlocks much, much heavier, so the problem
> may be observable
> >> more clearly. named still uses pthread locks Even
> with x86_32, so it
> >> may just be leaking memory more slowly.
> >>
> >> Again, everything is just a guess and could be
> wrong. We should seek
> >> advice from someone who knows FreeBSD library
> well.
> >
> > Just out of curiosity, why in theory is this not seen
> in prior versions of BIND such as 9.4.2-P2 or 9.4.3 on the
> same FreeBSD 7.0 AMD64 platforms with threading enabled in
> BIND?
>
>
> --
> Рыбин Дмитрий
> Управление магистральной сети
> Отдел Информационных Систем
> Руководитель группы АВР
> Corbina Telecom
> Tel: +7(495) 728-4000
> ___
> bind-users mailing list
> bind-users@lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
Hello!
I test patch, add to bind95/Makefile
.if (${ARCH} == "amd64")
ARCH= x86_64
.endif
work/bind-9.5.0-P2/config.log
uname -m = amd64
/usr/bin/uname -p = amd64
Target: amd64-undermydesk-freebsd
Configured with: FreeBSD/amd64 system compiler
ISC_ARCH_DIR='x86_32'
build='x86_64-portbld-freebsd7.0'
build_alias='x86_64-portbld-freebsd7.0'
build_cpu='x86_64'
host='x86_64-portbld-freebsd7.0'
host_cpu='x86_64'
I didn't find any affect, memory leak very quickly with threads support,
and slowly without threads.
FreeBSD xxx 7.0-STABLE FreeBSD 7.0-STABLE #0: Wed Jul 2 14:18:35 MSD
2008 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/H1 amd64
Vinny Abello wrote:
>>> so does this memory leak only occur if
>>> @ISC_ARCH_DIR@ is "noatomic" under FreeBSD amd64?
>>> and not when its "x86_32" ?
>> First off, note that I have no explicit evidence of memory leak. But
>> *if there is indeed leak in the FreeBSD pthread library*, the key is
>> "noatomic". With this configuration named will call pthread
>> locks/unlocks much, much heavier, so the problem may be observable
>> more clearly. named still uses pthread locks Even with x86_32, so it
>> may just be leaking memory more slowly.
>>
>> Again, everything is just a guess and could be wrong. We should seek
>> advice from someone who knows FreeBSD library well.
>
> Just out of curiosity, why in theory is this not seen in prior versions of
> BIND such as 9.4.2-P2 or 9.4.3 on the same FreeBSD 7.0 AMD64 platforms with
> threading enabled in BIND?
--
Рыбин Дмитрий
Управление магистральной сети
Отдел Информационных Систем
Руководитель группы АВР
Corbina Telecom
Tel: +7(495) 728-4000
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
At Tue, 2 Dec 2008 00:35:32 -0500,
Vinny Abello <[EMAIL PROTECTED]> wrote:
> For what it's worth, I just want to contribute that I can
> confirm this behavior on my systems as well. On BIND 9.5.0-P2,
>From an off-list discussion, I found there was indeed memory leak in
the code of 9.5.0-P2 (so I was wrong in suspecting it might be the
FreeBSD thread library).
Can you try the patched copied below to see whether it solves the
problem?
Note: this leak was already fixed in 9.5.1rc1:
2435. [bug] Fixed an ACL memory leak affecting win32.
while the change description seems to indicate it's win32 specific,
it can actually happen any build with threads and without atomic
operations.
---
JINMEI, Tatuya
Internet Systems Consortium, Inc.
Index: acl.c
===
RCS file: /proj/cvs/prod/bind9/lib/dns/acl.c,v
retrieving revision 1.37.2.7
diff -u -r1.37.2.7 acl.c
--- acl.c 29 Apr 2008 01:04:14 - 1.37.2.7
+++ acl.c 1 Dec 2008 03:17:36 -
@@ -217,6 +217,7 @@
/* Search radix. */
result = isc_radix_search(acl->iptable->radix, &node, &pfx);
+ isc_refcount_destroy(&pfx.refcount);
/* Found a match. */
if (result == ISC_R_SUCCESS && node != NULL) {
Index: iptable.c
===
RCS file: /proj/cvs/prod/bind9/lib/dns/iptable.c,v
retrieving revision 1.5.46.3
diff -u -r1.5.46.3 iptable.c
--- iptable.c 21 Jan 2008 21:02:24 - 1.5.46.3
+++ iptable.c 1 Dec 2008 19:25:27 -
@@ -74,6 +74,7 @@
family = bitlen ? pfx.family : AF_INET;
result = isc_radix_insert(tab->radix, &node, NULL, &pfx);
+ isc_refcount_destroy(&pfx.refcount);
if (result != ISC_R_SUCCESS)
return(result);
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
RE: dnsperf and BIND memory consumption
Hi, For what it's worth, I just want to contribute that I can confirm this behavior on my systems as well. On BIND 9.5.0-P2, building manually outside of ports with threads enabled does not induce the massively noticeable memory leak like it does from the version in ports. The BIND 9.4.2-P2 port is fine though even with threads enabled. I'm currently running off a manually built version of 9.4.3 which resulted in x86_32 for @ISC_ARCH_DIR@ from the configure script on my AMD64 FreeBSD 7.0 systems and also works just fine. Thanks for your additional investigation into this issue, Ivan. Hopefully the root cause is or will be identified and corrected. I'm fine building BIND manually from source. Can someone comment what changes are actually in the versions from the FreeBSD ports tree that differs from the source and what those changes intend to do exactly? -Vinny > -Original Message- > From: JINMEI Tatuya / 神明達哉 [mailto:[EMAIL PROTECTED] > Sent: Friday, November 28, 2008 4:09 AM > To: [EMAIL PROTECTED] > Cc: [EMAIL PROTECTED]; Vinny Abello; [EMAIL PROTECTED] > Subject: Re: dnsperf and BIND memory consumption > > At Thu, 27 Nov 2008 23:35:30 -0800 (PST), > ivan jr sy <[EMAIL PROTECTED]> wrote: > > > so does this memory leak only occur if > > @ISC_ARCH_DIR@ is "noatomic" under FreeBSD amd64? > > and not when its "x86_32" ? > > First off, note that I have no explicit evidence of memory leak. But > *if there is indeed leak in the FreeBSD pthread library*, the key is > "noatomic". With this configuration named will call pthread > locks/unlocks much, much heavier, so the problem may be observable > more clearly. named still uses pthread locks Even with x86_32, so it > may just be leaking memory more slowly. > > Again, everything is just a guess and could be wrong. We should seek > advice from someone who knows FreeBSD library well. > > --- > JINMEI, Tatuya > Internet Systems Consortium, Inc. ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
RE: dnsperf and BIND memory consumption
> -Original Message- > From: JINMEI Tatuya / 神明達哉 [mailto:[EMAIL PROTECTED] > Sent: Friday, November 28, 2008 4:09 AM > To: [EMAIL PROTECTED] > Cc: [EMAIL PROTECTED]; Vinny Abello; [EMAIL PROTECTED] > Subject: Re: dnsperf and BIND memory consumption > > At Thu, 27 Nov 2008 23:35:30 -0800 (PST), > ivan jr sy <[EMAIL PROTECTED]> wrote: > > > so does this memory leak only occur if > > @ISC_ARCH_DIR@ is "noatomic" under FreeBSD amd64? > > and not when its "x86_32" ? > > First off, note that I have no explicit evidence of memory leak. But > *if there is indeed leak in the FreeBSD pthread library*, the key is > "noatomic". With this configuration named will call pthread > locks/unlocks much, much heavier, so the problem may be observable > more clearly. named still uses pthread locks Even with x86_32, so it > may just be leaking memory more slowly. > > Again, everything is just a guess and could be wrong. We should seek > advice from someone who knows FreeBSD library well. Just out of curiosity, why in theory is this not seen in prior versions of BIND such as 9.4.2-P2 or 9.4.3 on the same FreeBSD 7.0 AMD64 platforms with threading enabled in BIND? -Vinny ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption -- FreeBSD port dns/bind95 for AMD64 - memory leak?
Thanks for the advise. I'm currently testing the performance of authoritative queries. Test data is 100 zones with a total of 50,000 RRs, using dnsperf and invoked 6 times simultaneously. That's 300,000 queries per run of the script. The script was invoked 5 (runs) successively. memory consumption was observed by just using 'top'. result: 1. FreeBSD dns/bind95 with the memory leak issue: 1st run:30,241QPS - mem: 220MB 2nd run:28,121QPS - mem: 640MB 3rd run:14,854QPS - mem: 990MB 4th run:9,521QPS - mem: 1780MB 5th run:7,545QPS - mem: 2540MB (1.5 million queries) note: the physical memory of the system is just 2GIG, swap was 20% during the last test. i restarted the box (just to be sure) and... 2. FreeBSD dns/bind95 with the patch below. ARCH=x86_64 1st run:34,213QPS - mem: 65MB 2nd run:33,505QPS - mem: 65MB 3rd run:34,251QPS - mem: 65MB 4th run:34,345QPS - mem: 65MB 5th run:34,012QPS - mem: 65MB (1.5 million queries) note: there was no movement in mem as shown by 'top'. not even a single KB basically, the memory consumption did not move (as this is an isolated test setup).. i'll be looping this script for a day or two, and after that test it as a recursive server for another day or two, and for another day and two as caching and authoritative at the same time. After that I can then migrate from my old linux box to this freebsd. my tools are just: http://www.nominum.com/services/measurement_tools.php Thanks - ivan --- On Fri, 11/28/08, JINMEI Tatuya / 神明達哉 <[EMAIL PROTECTED]> wrote: > From: JINMEI Tatuya / 神明達哉 <[EMAIL PROTECTED]> > Subject: Re: dnsperf and BIND memory consumption > To: [EMAIL PROTECTED] > Cc: [EMAIL PROTECTED], "Vinny Abello" <[EMAIL PROTECTED]>, "[EMAIL > PROTECTED]" <[EMAIL PROTECTED]> > Date: Friday, November 28, 2008, 10:09 PM > At Thu, 27 Nov 2008 23:35:30 -0800 (PST), > ivan jr sy <[EMAIL PROTECTED]> wrote: > > > so does this memory leak only occur if > > @ISC_ARCH_DIR@ is "noatomic" under FreeBSD > amd64? > > and not when its "x86_32" ? > > First off, note that I have no explicit evidence of memory > leak. But > *if there is indeed leak in the FreeBSD pthread library*, > the key is > "noatomic". With this configuration named will > call pthread > locks/unlocks much, much heavier, so the problem may be > observable > more clearly. named still uses pthread locks Even with > x86_32, so it > may just be leaking memory more slowly. > > Again, everything is just a guess and could be wrong. We > should seek > advice from someone who knows FreeBSD library well. > > --- > JINMEI, Tatuya > Internet Systems Consortium, Inc. ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
At Thu, 27 Nov 2008 23:35:30 -0800 (PST), ivan jr sy <[EMAIL PROTECTED]> wrote: > so does this memory leak only occur if > @ISC_ARCH_DIR@ is "noatomic" under FreeBSD amd64? > and not when its "x86_32" ? First off, note that I have no explicit evidence of memory leak. But *if there is indeed leak in the FreeBSD pthread library*, the key is "noatomic". With this configuration named will call pthread locks/unlocks much, much heavier, so the problem may be observable more clearly. named still uses pthread locks Even with x86_32, so it may just be leaking memory more slowly. Again, everything is just a guess and could be wrong. We should seek advice from someone who knows FreeBSD library well. --- JINMEI, Tatuya Internet Systems Consortium, Inc. ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
so does this memory leak only occur if @ISC_ARCH_DIR@ is "noatomic" under FreeBSD amd64? and not when its "x86_32" ? --- On Fri, 11/28/08, JINMEI Tatuya / 神明達哉 <[EMAIL PROTECTED]> wrote: > From: JINMEI Tatuya / 神明達哉 <[EMAIL PROTECTED]> > Subject: Re: dnsperf and BIND memory consumption > To: [EMAIL PROTECTED] > Cc: [EMAIL PROTECTED], "Vinny Abello" <[EMAIL PROTECTED]>, "[EMAIL > PROTECTED]" <[EMAIL PROTECTED]> > Date: Friday, November 28, 2008, 8:21 PM > At Thu, 27 Nov 2008 23:04:58 -0800 (PST), > ivan jr sy <[EMAIL PROTECTED]> wrote: > > > - with a minor tweak, it fixed the memory consumption > problem. > > > > if you compile BIND9.5.0-P2 on FreeBSD by hand, the > host_cpu is 'x86_32' > > # grep -e "host_cpu" -e "ARCH" > config.log > > ISC_ARCH_DIR='x86_32' > > host_cpu='x86_64' > > > > and with threads enabled... you won't experience > the high memory consumption... > > > > But if you install using ports/bind95 (FreeBSD ports) > > #grep -e "host_cpu" -e "ARCH" > config.log > > ISC_ARCH_DIR='noatomic' > > host_cpu='amd64' > > > > with threads enabled... then you will get high memory > consumption after a several thousands of queries.. > > These seem to me to indicate that there's a memory leak > in FreeBSD's > pthread_mutex_lock() and/or pthread_mutex_unlock(). > > --- > JINMEI, Tatuya > Internet Systems Consortium, Inc. ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
At Thu, 27 Nov 2008 23:04:58 -0800 (PST), ivan jr sy <[EMAIL PROTECTED]> wrote: > - with a minor tweak, it fixed the memory consumption problem. > > if you compile BIND9.5.0-P2 on FreeBSD by hand, the host_cpu is 'x86_32' > # grep -e "host_cpu" -e "ARCH" config.log > ISC_ARCH_DIR='x86_32' > host_cpu='x86_64' > > and with threads enabled... you won't experience the high memory > consumption... > > But if you install using ports/bind95 (FreeBSD ports) > #grep -e "host_cpu" -e "ARCH" config.log > ISC_ARCH_DIR='noatomic' > host_cpu='amd64' > > with threads enabled... then you will get high memory consumption after a > several thousands of queries.. These seem to me to indicate that there's a memory leak in FreeBSD's pthread_mutex_lock() and/or pthread_mutex_unlock(). --- JINMEI, Tatuya Internet Systems Consortium, Inc. ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
Could it be as simple as the ARCH not compatible with BIND 9.5.0 FreeBSD port
under FreeBSD AMD??
- with a minor tweak, it fixed the memory consumption problem.
if you compile BIND9.5.0-P2 on FreeBSD by hand, the host_cpu is 'x86_32'
# grep -e "host_cpu" -e "ARCH" config.log
ISC_ARCH_DIR='x86_32'
host_cpu='x86_64'
and with threads enabled... you won't experience the high memory consumption...
But if you install using ports/bind95 (FreeBSD ports)
#grep -e "host_cpu" -e "ARCH" config.log
ISC_ARCH_DIR='noatomic'
host_cpu='amd64'
with threads enabled... then you will get high memory consumption after a
several thousands of queries..
Do we need to patch Makefile of ports/bind95 to make use "x86_64"? (cc'ed in
this thread is port maintenair for his comments...)
--- Makefile.orig 2008-11-28 13:16:32.0 +0800
+++ Makefile2008-11-28 13:08:09.0 +0800
@@ -45,6 +45,10 @@
.include
# We are ok by default from 7.0-RELEASE on
+.if (${ARCH} == "amd64")
+ARCH= x86_64
+.endif
+
.if ${OSVERSION} >= 700055
OPTIONS+= THREADS "Compile with thread support" on
.else
Another thing... (not affected thou) is that if you compile BIND by hand under
FreeBSD, the compiler is gcc while in ports thats cc...
(BIND95 port uses cc by default)
Jin,
or how about make BIND aware of it?
There are some conditions in configure.in about "x86_64" and the compiler being
used. Should'nt it be able to interpret "amd64" and "cc" ?
I'll reply again to this post after some performance testing with this
configuration.. if 9.5.0-P2 threaded and non-threaded differ in FreeBSD am64
with x number of AMD64 cores...
- ivan
--- On Thu, 11/27/08, JINMEI Tatuya / 神明達哉 <[EMAIL PROTECTED]> wrote:
> From: JINMEI Tatuya / 神明達哉 <[EMAIL PROTECTED]>
> Subject: Re: dnsperf and BIND memory consumption
> To: [EMAIL PROTECTED]
> Cc: "Vinny Abello" <[EMAIL PROTECTED]>, "[EMAIL PROTECTED]" <[EMAIL
> PROTECTED]>
> Date: Thursday, November 27, 2008, 12:06 PM
> At Wed, 26 Nov 2008 10:34:59 -0800 (PST),
> ivan jr sy <[EMAIL PROTECTED]> wrote:
>
> > I know this is a an old thread, but I wish to
> resurrect this in
> > hopes to find answers..
> >
> > 9.5 + threads on FreeBSD 7 is better performance wise,
> but there is
> > this problem.
> >
> > 9.4 + threads on FreeBSD 7 is almost 50% of the
> performance, but
> > there is no issues like this. 9.5 without threads
> doesnt have this
> > issue but same in performance.
>
> > more data below... its basically the same as
> Vinny's but im
> > stressing out that 9.5 with threads has a good
> performance.
> >
> > hoping there's some shed of light as to where to
> get a patch for this issue.
>
> Unfortunately, I've never succeeded in reproducing this
> problem. If
> possible, can I see your test configuration (including
> named.conf and
> query data)?
>
> Also, just out of curiosity (I actually don't think
> this matters), do
> you see the same problem if you build named by hand, not
> from FreeBSD
> ports?
>
> ---
> JINMEI, Tatuya
> Internet Systems Consortium, Inc.
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
At Wed, 26 Nov 2008 10:34:59 -0800 (PST), ivan jr sy <[EMAIL PROTECTED]> wrote: > I know this is a an old thread, but I wish to resurrect this in > hopes to find answers.. > > 9.5 + threads on FreeBSD 7 is better performance wise, but there is > this problem. > > 9.4 + threads on FreeBSD 7 is almost 50% of the performance, but > there is no issues like this. 9.5 without threads doesnt have this > issue but same in performance. > more data below... its basically the same as Vinny's but im > stressing out that 9.5 with threads has a good performance. > > hoping there's some shed of light as to where to get a patch for this issue. Unfortunately, I've never succeeded in reproducing this problem. If possible, can I see your test configuration (including named.conf and query data)? Also, just out of curiosity (I actually don't think this matters), do you see the same problem if you build named by hand, not from FreeBSD ports? --- JINMEI, Tatuya Internet Systems Consortium, Inc. ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
RE: dnsperf and BIND memory consumption
Hi all, I know this is a an old thread, but I wish to resurrect this in hopes to find answers.. 9.5 + threads on FreeBSD 7 is better performance wise, but there is this problem. 9.4 + threads on FreeBSD 7 is almost 50% of the performance, but there is no issues like this. 9.5 without threads doesnt have this issue but same in performance. more data below... its basically the same as Vinny's but im stressing out that 9.5 with threads has a good performance. hoping there's some shed of light as to where to get a patch for this issue. Thanks! - Ivan system: FreeBSD 7.0 RELEASE AMD64 Server is a Dell SC1435 with 4 CPU's, no Hyperthreading, 2GB of RAM and a 150GB RAID1 Dnsperf run from a different server on the same network segment over Gig-E 1. FreeBSD 7-RELEASE+BIND 9.4.2-P2 = 34,000 QPS, 94MB mem 2. FreeBSD 7-RELEASE+BIND 9.5.0-P2 threaded = 82,000 QPS, 1.5GIG mem! (and it wont stop until the test script ends, and does not go back to its original state) 3. FreeBSD 7-RELEASE+BIND 9.5.0-P2 non-threaded = 34,000 QPS, 95MB mem FIRST TEST # pkg_info | grep bind bind94-base-9.4.2.2 The BIND DNS suite with updated DNSSEC and threads # named -v BIND 9.4.2-P2 # ldd /usr/sbin/named /usr/sbin/named: libcrypto.so.5 => /lib/libcrypto.so.5 (0x8007a9000) libthr.so.3 => /lib/libthr.so.3 (0x800a3b000) libc.so.7 => /lib/libc.so.7 (0x800b51000) PID USERNAME THR PRI NICE SIZERES STATE C TIME WCPU COMMAND 13677 bind7 1000 93704K 77912K select 1 6:13 194.43% named Notes: 1. regardless how many times the script was used, memory consumption remained the same.. 2. a few seconds after the script was terminated... the CPU normalize.. PID USERNAME THR PRI NICE SIZERES STATE C TIME WCPU COMMAND 13677 bind7 980 93704K 77912K select 3 7:57 0.00% named SECOND TEST # pkg_info | grep bind bind95-base-9.5.0.2 The BIND DNS suite with updated DNSSEC and threads # named -v BIND 9.5.0-P2 # ldd /usr/sbin/named /usr/sbin/named: libcrypto.so.5 => /lib/libcrypto.so.5 (0x8007bf000) libxml2.so.5 => /usr/local/lib/libxml2.so.5 (0x800a51000) libz.so.4 => /lib/libz.so.4 (0x800c95000) libiconv.so.3 => /usr/local/lib/libiconv.so.3 (0x800da9000) libm.so.5 => /lib/libm.so.5 (0x800fa2000) libthr.so.3 => /lib/libthr.so.3 (0x8010bc000) libc.so.7 => /lib/libc.so.7 (0x8011d2000) PID USERNAME THR PRI NICE SIZERES STATE C TIME WCPU COMMAND 67304 bind7 990 1524M 1509M select 1 2:10 200.54% named Notes: 1. memory consumption of 1.5G after only running the script 26 times. thats 1.3 million authoritative queries. 2. the script was terminated and the memory consumption was still the same. 3RD TEST (very similar to 1st test) Hardware Details CPU: Quad-Core AMD Opteron(tm) Processor 2350 (1995.01-MHz K8-class CPU) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 SMP: AP CPU #2 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #1 Launched! usable memory = 2133491712 (2034 MB) avail memory = 2058821632 (1963 MB) # uname -a FreeBSD jaljeb.infoweapons.com 7.0-RELEASE FreeBSD 7.0-RELEASE #0: Sun Feb 24 10:35:36 UTC 2008 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/GENERIC FreeBSD 7.0 RELEASE AMD64 Server is a Dell SC1435 with 4 CPU's, Hyperthreading disabled, 2GB of RAM and a 150GB RAID1 Dnsperf run from a different server on the same network segment over Gig-E --- On Fri, 8/8/08, Vinny Abello <[EMAIL PROTECTED]> wrote: > From: Vinny Abello <[EMAIL PROTECTED]> > Subject: RE: dnsperf and BIND memory consumption > To: "JINMEI Tatuya / 神明達哉" <[EMAIL PROTECTED]> > Cc: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> > Date: Friday, August 8, 2008, 2:33 AM > > -Original Message- > > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On > > Behalf Of JINMEI Tatuya / > > Sent: Thursday, August 07, 2008 3:56 AM > > To: Vinny Abello > > Cc: [EMAIL PROTECTED] > > Subject: Re: dnsperf and BIND memory consumption > > > > At Thu, 7 Aug 2008 00:58:23 -0400, > > Vinny Abello <[EMAIL PROTECTED]> wrote: > > > > > OK. I've recompiled BIND 9.5.0-P2 (from > ports) without threads > > > enabled. I no longer see the memory leak at all. > I'm running dnsperf > > > and I see a constant of 18MB which is much more > reasonable for what > > > I am doing. For me it's easy to reproduce. > Some more information > > > that may help reproduce it: > > > > > FreeBSD 7.0 STABLE AMD64 (cvsup'ed within the > past week) > > > BIND 9.5.0-P2 installed via ports with threads > enabled > > > Server is a Dell PowerEdge 2850 with 2 CPU's,
