[Unbound-users] Is there any memory leak in unbound?
Hi guys,I got a problem. I installed unbound 1.3.3 and found when the unbound is running,it use more and more memory.. I even found it occupied 96% of memory and then was killed by the system.When it was killed,the following log was found in /var/log/messages("Dnsserv" is the hostname): Sep 8 00:21:02 Dnsserv Out of memory: kill process 23999 (unbound) score 125734 or a child Sep 8 00:21:02 Dnsserv Killed process 23999 (unbound) I only specify some normal items in unbound.conf and it was shown bellow: server: verbosity: 1 statistics-interval: 1 num-threads: 1 interface: 0.0.0.0 interface-automatic: no port: 53 msg-cache-size: 4m rrset-cache-size: 4m do-ip4: yes do-ip6: no do-udp: yes access-control: 10.0.0.0/8 allow access-control: 127.0.0.0/8 allow statistics-interval: 10 #chroot: "/var/unbound" username: "unbound" directory: "/var/unbound" logfile: "" use-syslog: no log-time-ascii: yes pidfile: "/var/run/unbound.pid" root-hints: "/var/unbound/named.cache" hide-version: no version: "hiahia" local-zone: "mydns123.com." static local-data: "mydns123.com. SOA ns.mydns123.com tj.mydns123.com. 1 28800 7200 604800 86400" local-data: "mydns123.com. IN NS ns.mydns123.com." local-data: "mydns123.com. IN NS ns2.mydns123.com." local-data: "ns.mydns123.com. IN A 192.1.1.1" local-data: "ns1.mydns123.com. IN A 192.1.1.2" local-data: "www.mydns123.com. IN A 192.1.1.3" local-data: "ftp.mydns123.com. IN A 192.1.1.4" local-data: "wa.mydns123.com. IN A 192.1.1.5" local-data: "pp.mydns123.com. IN CNAME wa.mydns123.com." local-zone:"1.1.192.in-addr.arpa" static local-data: "mydns123.com. SOA ns.mydns123.com tj.mydns123.com. 1 28800 7200 604800 86400" local-data: "mydns123.com. IN NS ns.mydns123.com." local-data: "mydns123.com. IN NS ns2.mydns123.com." local-data:"1.1.1.192.in-addr.arpa IN PTR ns.mydns123.com." local-data:"2.1.1.192.in-addr.arpa IN PTR ns1.mydns123.com." local-data:"3.1.1.192.in-addr.arpa IN PTR www.mydns123.com." local-data:"4.1.1.192.in-addr.arpa IN PTR ftp.mydns123.com." local-data:"5.1.1.192.in-addr.arpa IN PTR was.mydns123.com." Is there any mistake in my configuration?Or is it a bug of unbound? ___ 好玩贺卡等你发,邮箱贺卡全新上线! http://card.mail.cn.yahoo.com/___ Unbound-users mailing list Unbound-users@unbound.net http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users
Re: [Unbound-users] Is there any memory leak in unbound?
On Tue, 8 Sep 2009, Tao Ma wrote: I installed unbound 1.3.3 and found when the unbound is running,it use more and more memory.. I even found it occupied 96% of memory and then was killed by the Sep 8 00:21:02 Dnsserv Out of memory: kill process 23999 (unbound) score I only specify some normal items in unbound.conf and it was shown bellow: server: msg-cache-size: 4m rrset-cache-size: 4m Is this an embedded device with little memory? eg 16mb or 32mb? The OOM killer might be killing the biggest memory consumer, which not neccessarilly means that process is the one leaking memory. unbound might be running stable at just over 8mb, and some other process might be slowly allocating more memory over time, but still end up using a smaller memory footprint, for which unbound takes the blame (and the shot) Paul ___ Unbound-users mailing list Unbound-users@unbound.net http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users
Re: [Unbound-users] Is there any memory leak in unbound?
Hi Wouter,thanks very much for your reply. I tried the method you mentioned,here is the info I got from the log: debug: cache memory msg=16528 rrset=33040 infra=672 val=24888 This info keeps the same from the start of the unbound(I have only about 5 records in unbound). I also found the value of mem.total.sbrk printed by "unbound-control stats" keeps growing.The item "mem.total.sbrk" is an estimate of the heap size of unbound in bytes,it is close to the memory used by unbound. I have a script keeps recording mem.total.sbrk in a file,and found when mem.total.sbrk grew to 2017873920,unbound was killed by system(My server has a 2G memory). Unbound runs in a Gentoo Linux with 2.6.23 kernel,does the problem relate to the OS? Thanks & Best regards, Tao Ma --- 09年9月10日,周四, W.C.A. Wijngaards 写道: 发件人: W.C.A. Wijngaards 主题: Re: [Unbound-users] 回复: Re: Is there any memory leak in unbound? 收件人: "Tao Ma" 抄送: "Paul Wouters" , unbound-users@unbound.net 日期: 2009年9月10日,周四,下午10:25 -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Tao Ma, The memory statistics that unbound makes can help? It prints them into the log file (depending on verbosity level and --enable-debug to configure) like this: debug: cache memory msg=1620127 rrset=2964258 infra=738306 val=128832 Also the memory statistics are printed when you enable extended statistics and do unbound-control stats, like this: mem.cache.rrset=2980222 mem.cache.message=1633031 mem.mod.iterator=16460 mem.mod.validator=130126 Best regards, Wouter ___ 好玩贺卡等你发,邮箱贺卡全新上线! http://card.mail.cn.yahoo.com/___ Unbound-users mailing list Unbound-users@unbound.net http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users
Re: [Unbound-users] Is there any memory leak in unbound?
What's more,I found that the memory occupied by unbound would not be freed when all requets finished,that means,unbound only consume memory but never free them. What does unbound hold the memory for?It quite like that a memory leak happens... Anyone else has been in the same trouble? Best Regards, Tao Ma --- 09年9月11日,周五, Tao Ma 写道: 发件人: Tao Ma 主题: Re: [Unbound-users] Is there any memory leak in unbound? 收件人: "W.C.A. Wijngaards" 抄送: unbound-users@unbound.net 日期: 2009年9月11日,周五,上午11:50 Hi Wouter,thanks very much for your reply. I tried the method you mentioned,here is the info I got from the log: debug: cache memory msg=16528 rrset=33040 infra=672 val=24888 This info keeps the same from the start of the unbound(I have only about 5 records in unbound). I also found the value of mem.total.sbrk printed by "unbound-control stats" keeps growing.The item "mem.total.sbrk" is an estimate of the heap size of unbound in bytes,it is close to the memory used by unbound. I have a script keeps recording mem.total.sbrk in a file,and found when mem.total.sbrk grew to 2017873920,unbound was killed by system(My server has a 2G memory). Unbound runs in a Gentoo Linux with 2.6.23 kernel,does the problem relate to the OS? Thanks & Best regards, Tao Ma --- 09年9月10日,周四, W.C.A. Wijngaards 写道: 发件人: W.C.A. Wijngaards 主题: Re: [Unbound-users] 回复: Re: Is there any memory leak in unbound? 收件人: "Tao Ma" 抄送: "Paul Wouters" , unbound-users@unbound.net 日期: 2009年9月10日,周四,下午10:25 -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Tao Ma, The memory statistics that unbound makes can help? It prints them into the log file (depending on verbosity level and --enable-debug to configure) like this: debug: cache memory msg=1620127 rrset=2964258 infra=738306 val=128832 Also the memory statistics are printed when you enable extended statistics and do unbound-control stats, like this: mem.cache.rrset=2980222 mem.cache.message=1633031 mem.mod.iterator=16460 mem.mod.validator=130126 Best regards, Wouter 好玩贺卡等你发,邮箱贺卡全新上线! -下面为附件内容- ___ Unbound-users mailing list Unbound-users@unbound.net http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users ___ 好玩贺卡等你发,邮箱贺卡全新上线! http://card.mail.cn.yahoo.com/___ Unbound-users mailing list Unbound-users@unbound.net http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users
Re: [Unbound-users] Is there any memory leak in unbound?
2009/9/11 Tao Ma > > What's more,I found that the memory occupied by unbound would not be freed > when all requets finished,that means,unbound only consume memory but never > free them. > What does unbound hold the memory for?It quite like that a memory leak > happens... > Anyone else has been in the same trouble? Hi, do you reload unbound while it's running ? -- Artis Caune Everything should be made as simple as possible, but not simpler. ___ Unbound-users mailing list Unbound-users@unbound.net http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users
Re: [Unbound-users] Is there any memory leak in unbound?
Hi,I didn't reload unbound while it's running,I always start it and then send requests to the server,then it will consume more and more memory. --- 09年9月11日,周五, Artis Caune 写道: 发件人: Artis Caune 主题: Re: [Unbound-users] Is there any memory leak in unbound? 收件人: "Tao Ma" 抄送: wou...@nlnetlabs.nl, unbound-users@unbound.net 日期: 2009年9月11日,周五,下午7:02 Hi, do you reload unbound while it's running ? -- Artis Caune 聽 聽Everything should be made as simple as possible, but not simpler. ___ 好玩贺卡等你发,邮箱贺卡全新上线! http://card.mail.cn.yahoo.com/___ Unbound-users mailing list Unbound-users@unbound.net http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users
Re: [Unbound-users] Is there any memory leak in unbound?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Tao Ma, > debug: cache memory msg=16528 rrset=33040 infra=672 val=24888 The values from your log file look very reasonable. No problems here. The sbrk keeps growing? Ok, it is possible to enable even more memory debug, configure with - --enable-alloc-checks and compile again. Unbound prints: info: Memory conditions: %u front=%u back=%u mesh=%u msg=%u rrset=%u infra=%u iter=%u val=%u alloccache=%u globalalloccache=%u me=%u But with values. It also prints a log line for *every* alloc call. So, you can see where in the code it allocates all that memory. This will make the server very slow. If you send me the log file (off-list to wou...@nlnetlabs.nl , then I can parse it with testcode/memstats.c; or you can do that yourself: make memstats ; ./memstats < logfile ). To answer your question, unbound keeps some memory around, up to a maximum, because free-ing something that you need later again is slow. This should top out at 40Mb for the standard config. Gentoo? Is it 64bit? Does it use a 'special' memory allocator in libc? Best regards, Wouter On 09/11/2009 10:06 AM, Tao Ma wrote: > What's more,I found that the memory occupied by unbound would not be > freed when all requets finished,that means,unbound only consume memory > but never free them. > What does unbound hold the memory for?It quite like that a memory leak > happens... > Anyone else has been in the same trouble? > > > Best Regards, > Tao Ma > > > --- *09年9月11日,周五, Tao Ma //* 写道: > > > 发件人: Tao Ma > 主题: Re: [Unbound-users] Is there any memory leak in unbound? > 收件人: "W.C.A. Wijngaards" > 抄送: unbound-users@unbound.net > 日期: 2009年9月11日,周五,上午11:50 > > Hi Wouter,thanks very much for your reply. > > I tried the method you mentioned,here is the info I got from the log: > debug: cache memory msg=16528 rrset=33040 infra=672 val=24888 > This info keeps the same from the start of the unbound(I have only > about 5 records in unbound). > > I also found the value of mem.total.sbrk printed by "unbound-control > stats" keeps growing.The item "mem.total.sbrk" is an estimate of the > heap size of unbound in bytes,it is close to the memory used by unbound. > > I have a script keeps recording mem.total.sbrk in a file,and found > when mem.total.sbrk grew to 2017873920,unbound was killed by > system(My server has a 2G memory). > > Unbound runs in a Gentoo Linux with 2.6.23 kernel,does the problem > relate to the OS? > > Thanks & Best regards, > Tao Ma > > > > --- *09年9月10日,周四, W.C.A. Wijngaards //* > 写道: > > > 发件人: W.C.A. Wijngaards > 主题: Re: [Unbound-users] 回复: Re: Is there any memory leak in > unbound? > 收件人: "Tao Ma" > 抄送: "Paul Wouters" , unbound-users@unbound.net > 日期: 2009年9月10日,周四,下午10:25 > > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Hi Tao Ma, > > The memory statistics that unbound makes can help? > > It prints them into the log file (depending on verbosity > level and --enable-debug to configure) like this: > debug: cache memory msg=1620127 rrset=2964258 infra=738306 > val=128832 > > Also the memory statistics are printed when you enable extended > statistics and do unbound-control stats, like this: > mem.cache.rrset=2980222 > mem.cache.message=1633031 > mem.mod.iterator=16460 > mem.mod.validator=130126 > > Best regards, >Wouter > > > > 好玩贺卡等你发,邮箱贺卡全新上线! > > <http://cn.rd.yahoo.com/mail_cn/tagline/card/*http://card.mail.cn.yahoo.com/> > > -下面为附件内容- > > ___ > Unbound-users mailing list > Unbound-users@unbound.net > http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users > > > > 好玩贺卡等你发,邮箱贺卡全新上线! > <http://cn.rd.yahoo.com/mail_cn/tagline/card/*http://card.mail.cn.yahoo.com/> > -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iEYEARECAAYFAkqvXcAACgkQkDLqNwOhpPggMACgt9whFNmwi52mp0CzxiBvtIZD xlEAoIgt2xRAmW7nsBP1CgvMi6tNykcw =Rw2R -END PGP SIGNATURE- ___ Unbound-users mailing list Unbound-users@unbound.net http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users
Re: [Unbound-users] Is there any memory leak in unbound?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Tao Ma, Just realized something else, the sbrk estimate is a dirty hack. Gentoo may report a different sbrk() value, and the dirty hack does not work. The sbrk() value keeps growing but maybe it is not true that unbound's memory keeps growing. Does the 'top' or 'ps' memory number also grow? Best regards, Wouter On 09/15/2009 11:26 AM, W.C.A. Wijngaards wrote: > Hi Tao Ma, > >> debug: cache memory msg=16528 rrset=33040 infra=672 val=24888 > > The values from your log file look very reasonable. No problems here. > The sbrk keeps growing? > > Ok, it is possible to enable even more memory debug, configure with > --enable-alloc-checks and compile again. Unbound prints: > info: Memory conditions: %u front=%u back=%u mesh=%u msg=%u rrset=%u > infra=%u iter=%u val=%u alloccache=%u globalalloccache=%u me=%u > But with values. > > It also prints a log line for *every* alloc call. So, you can see where > in the code it allocates all that memory. This will make the > server very slow. If you send me the log file (off-list to > wou...@nlnetlabs.nl , then I can parse it with testcode/memstats.c; > or you can do that yourself: make memstats ; ./memstats < logfile ). > > To answer your question, unbound keeps some memory around, up to a > maximum, because free-ing something that you need later again is slow. > This should top out at 40Mb for the standard config. > > Gentoo? Is it 64bit? Does it use a 'special' memory allocator in libc? > > Best regards, >Wouter > > On 09/11/2009 10:06 AM, Tao Ma wrote: >> What's more,I found that the memory occupied by unbound would not be >> freed when all requets finished,that means,unbound only consume memory >> but never free them. >> What does unbound hold the memory for?It quite like that a memory leak >> happens... >> Anyone else has been in the same trouble? > > >> Best Regards, >> Tao Ma > > >> --- *09t911åh, Tao Ma //* S > > >> Ñöº: Tao Ma >> ;: Re: [Unbound-users] Is there any memory leak in unbound? >> 6öº: "W.C.A. Wijngaards" >> : unbound-users@unbound.net >> å: 2009t911å,h, > H11:50 > >> Hi Wouter,thanks very much for your reply. > >> I tried the method you mentioned,here is the info I got from the log: >> debug: cache memory msg=16528 rrset=33040 infra=672 val=24888 >> This info keeps the same from the start of the unbound(I have only >> about 5 records in unbound). > >> I also found the value of mem.total.sbrk printed by "unbound-control >> stats" keeps growing.The item "mem.total.sbrk" is an estimate of the >> heap size of unbound in bytes,it is close to the memory used by unbound. > >> I have a script keeps recording mem.total.sbrk in a file,and found >> when mem.total.sbrk grew to 2017873920,unbound was killed by >> system(My server has a 2G memory). > >> Unbound runs in a Gentoo Linux with 2.6.23 kernel,does the problem >> relate to the OS? > >> Thanks & Best regards, >> Tao Ma > > > >> --- *09t910åhÛ, W.C.A. Wijngaards //* >> S > > >> Ñöº: W.C.A. Wijngaards >> ;: Re: [Unbound-users] Þ Re: Is there any memory leak in >> unbound? >> 6öº: "Tao Ma" >> : "Paul Wouters" , unbound-users@unbound.net >> å: 2009t910å,hÛ,H10:25 > >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA1 > >> Hi Tao Ma, > >> The memory statistics that unbound makes can help? > >> It prints them into the log file (depending on verbosity >> level and --enable-debug to configure) like this: >> debug: cache memory msg=1620127 rrset=2964258 infra=738306 >> val=128832 > >> Also the memory statistics are printed when you enable extended >> statistics and do unbound-control stats, like this: >> mem.cache.rrset=2980222 >> mem.cache.message=1633031 >> mem.mod.iterator=16460 >> mem.mod.validator=130126 > >> Best regards, >>Wouter > > >> >> }©:aI`Ñ®±:ah° > ¿ >> >> <http://cn.rd.yahoo.com/mail_cn/tagline/card/*http://card.mail.cn.yahoo.com/> > >> -b:Dö ¹- > >> __
Re: [Unbound-users] Is there any memory leak in unbound?
Hi Wouter,I'm very glad to tell you,I patched the diff file you gave me and now unbound works well on the server,thanks very much! I'm sorry that I didn't tell you clearly about the compile problem.When we found we couldn't compile the code with "--enable-alloc-checks",we added another option "--enable-static-exe" and then finished compiling smoothly,that's all. Thanks again for your warmhearted help. Best Regards, Tao Ma --- 09年9月16日,周三, W.C.A. Wijngaards 写道: 发件人: W.C.A. Wijngaards 主题: Re: [Unbound-users] Is there any memory leak in unbound? 收件人: "Tao Ma" 日期: 2009年9月16日,周三,下午8:47 Hi Tao Ma, Does this solve your memory leak problem? Index: libunbound/libworker.c === --- libunbound/libworker.c (revision 1831) +++ libunbound/libworker.c (working copy) @@ -514,6 +514,7 @@ ldns_buffer_write_u16_at(w->back->udp_buff, 2, qflags); if(local_zones_answer(ctx->local_zones, &qinfo, &edns, w->back->udp_buff, w->env->scratch)) { + regional_free_all(w->env->scratch); libworker_fillup_fg(q, LDNS_RCODE_NOERROR, w->back->udp_buff, sec_status_insecure); libworker_delete(w); @@ -630,6 +631,7 @@ ldns_buffer_write_u16_at(w->back->udp_buff, 2, qflags); if(local_zones_answer(w->ctx->local_zones, &qinfo, &edns, w->back->udp_buff, w->env->scratch)) { + regional_free_all(w->env->scratch); q->msg_security = sec_status_insecure; add_bg_result(w, q, w->back->udp_buff, UB_NOERROR); free(qinfo.qname); Index: daemon/worker.c === --- daemon/worker.c (revision 1831) +++ daemon/worker.c (working copy) @@ -806,6 +806,7 @@ } if(local_zones_answer(worker->daemon->local_zones, &qinfo, &edns, c->buffer, worker->scratchpad)) { + regional_free_all(worker->scratchpad); if(ldns_buffer_limit(c->buffer) == 0) { comm_point_drop_reply(repinfo); return 0; Sorry about the compile problem with unbound-host and alloc-checks, I'll look at it later. Best regards, Wouter On 09/16/2009 10:05 AM, Tao Ma wrote: > Hi Wouter,thanks for your reply. > > We use a 32-bit Gentoo and I don't think there's a special memory > allocator in libc. > I can always find the memory consumed by unbound keep growing from ouput > of 'ps' and 'top',the result of sbrk estimate seems to be in accord with > this. > > Following your suggestion,we configure with --enable-alloc-checks and > compile unbound again,now we can find the "Memory conditions" value keep > growing in the log. > > We use memstats to parse the log,but found nothing but the following ouput: > Reading /var/unbound/log_test of size 38767531 > 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16% 17% 18% 19% 20% > 21% 22% 23% 24% 25% 26% 27% 28% 29% 30% 31% 32% 33% 34% 35% 36% 37% 38% > 39% 40% 41% 42% 43% 44% 45% 46% 47% 48% 49% 50% 51% 52% 53% 54% 55% 56% > 57% 58% 59% 60% 61% 62% 63% 64% 65% 66% 67% 68% 69% 70% 71% 72% 73% 74% > 75% 76% 77% 78% 79% 80% 81% 82% 83% 84% 85% 86% 87% 88% 89% 90% 91% 92% > 93% 94% 95% 96% 97% 98% 99% 100% done > > 0 / 0 total in 0 code lines > > We got a complete record in the log from the start of unbound to the > exiting of unbound,and the log file has been attatched to the mail,maybe > it will help you to analyse. > > Let me know if you find any mistakes in my operation. > > By the way,when we configure unbound-1.3.3 with "./configure > --prefix=/usr/local/unbound --enable-alloc-checks" and compile,we got > the following error: > build/smallapp/.libs/unbound-host.o: In function `print_rd': > /home/mt/tools/unbound-1.3.3/smallapp/unbound-host.c:233: undefined > reference to `unbound_stat_malloc_log' > /home/mt/tools/unbound-1.3.3/smallapp/unbound-host.c:255: undefined > reference to `unbound_stat_free_log' > build/smallapp/.libs/unbound-host.o: In function `pretty_rdata': > /home/mt/tools/unbound-1.3.3/smallapp/unbound-host.c:247: undefined > reference to `unbound_stat_free_log' > build/smallapp/.libs/unbound-host.o: In function `lookup': > /home/mt/tools/unbound-1.3.3/smallapp/unbound-host.c:384: undefined > reference to `unbound_stat_free_log' > collect2: ld returned 1 exit status > make: *** [unbound-host] Error 1 > > We did some modification to the code and finished compiling.Is there > anything wrong in the code? > > Thanks & Best regards, > Tao Ma > > > > --- *09年9月15日,周二, W.C.A. Wijngaards //* 写道: