Re: Unbound swapping
Wouter, Something I just noticed: root@dns:/opt# tail -f unbound.log [1505432173] unbound[53221:4] error: internal error: looping module stopped [1505432173] unbound[53221:4] error: internal error: looping module stopped [1505432173] unbound[53221:4] error: internal error: looping module stopped [1505432252] unbound[53221:2] error: internal error: looping module stopped [1505432252] unbound[53221:2] error: internal error: looping module stopped [1505432252] unbound[53221:2] error: internal error: looping module stopped Maybe it's a problem? Regards, 2017-09-14 20:21 GMT-03:00 Eduardo Schoedler : > 2017-09-14 5:08 GMT-03:00 W.C.A. Wijngaards via Unbound-users > : >> Hi Eduardo, > > Hi Wouter! > Thank you for the answer. > >> I have no real good idea. But looking at your numbers, I see that you >> are running a network heavy application, unbound, and it uses about 10G >> on 12G memory. The buff/cache is 2G. Adds up to 12G. And it is >> swapping. Sounds reasonable, it is maxed out on memory, this is where >> swap is supposed to make space, right? > > Yes, our application is really heavy. > > Indeed about swap, but when unbound is swapping, performance degrades a lot. > If possible, we expect to not swap at all. > > >> These options change the buffer space allocated by unbound: so-rcvbuf, >> so-sndbuf, maybe also so-reuseport, or more tcp connections (that use >> buffer space) or more network interfaces, or simply a different kernel >> version that uses (slightly) more memory or something along those lines. > > root@dns:~# cat /etc/unbound/unbound.conf | grep 'so-' > so-rcvbuf: 16m > so-sndbuf: 16m > so-reuseport: yes > > root@dns:~# cat /etc/unbound/unbound.conf | grep 'tcp' > do-tcp: yes > incoming-num-tcp: 1024 > outgoing-num-tcp: 1024 > > root@dns:~# uname -a > Linux dns 4.9.26-040926-generic #201705031231 SMP Wed May 3 16:34:12 > UTC 2017 x86_64 x86_64 x86_64 GNU/Linux > > > >> What is the limit on the subnet cache size? What does the memory max >> out on when the subnetmodule is not enabled? > > It's a public DNS resolver project in Brasil (like Google DNS), so... > That's why we need to use subnetcache module. > > # cat /etc/unbound/unbound.conf | grep 'subnet' > module-config: "subnetcache validator iterator" > send-client-subnet: 0.0.0.0/0 > send-client-subnet: 2000::/3 > client-subnet-always-forward: no > max-client-subnet-ipv6: 48 > max-client-subnet-ipv4: 24 > > In subnetcache module manual says: > > "The maximum size of the ECS cache is controlled by 'msg-cache-size' in >the configuration file. On top of that, for each query only 100 differ- >ent subnets are allowed to be stored for each address family. Exceeding >that number, older entries will be purged from cache." > > This section "100-different subnets ... and then purged", how to know > if it's doing exactly this? > My guessing there is a memory leak, because the large memory it uses. > > > Best regards, > > -- > Eduardo Schoedler -- Eduardo Schoedler
Re: Unbound swapping
2017-09-14 5:08 GMT-03:00 W.C.A. Wijngaards via Unbound-users : > Hi Eduardo, Hi Wouter! Thank you for the answer. > I have no real good idea. But looking at your numbers, I see that you > are running a network heavy application, unbound, and it uses about 10G > on 12G memory. The buff/cache is 2G. Adds up to 12G. And it is > swapping. Sounds reasonable, it is maxed out on memory, this is where > swap is supposed to make space, right? Yes, our application is really heavy. Indeed about swap, but when unbound is swapping, performance degrades a lot. If possible, we expect to not swap at all. > These options change the buffer space allocated by unbound: so-rcvbuf, > so-sndbuf, maybe also so-reuseport, or more tcp connections (that use > buffer space) or more network interfaces, or simply a different kernel > version that uses (slightly) more memory or something along those lines. root@dns:~# cat /etc/unbound/unbound.conf | grep 'so-' so-rcvbuf: 16m so-sndbuf: 16m so-reuseport: yes root@dns:~# cat /etc/unbound/unbound.conf | grep 'tcp' do-tcp: yes incoming-num-tcp: 1024 outgoing-num-tcp: 1024 root@dns:~# uname -a Linux dns 4.9.26-040926-generic #201705031231 SMP Wed May 3 16:34:12 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux > What is the limit on the subnet cache size? What does the memory max > out on when the subnetmodule is not enabled? It's a public DNS resolver project in Brasil (like Google DNS), so... That's why we need to use subnetcache module. # cat /etc/unbound/unbound.conf | grep 'subnet' module-config: "subnetcache validator iterator" send-client-subnet: 0.0.0.0/0 send-client-subnet: 2000::/3 client-subnet-always-forward: no max-client-subnet-ipv6: 48 max-client-subnet-ipv4: 24 In subnetcache module manual says: "The maximum size of the ECS cache is controlled by 'msg-cache-size' in the configuration file. On top of that, for each query only 100 differ- ent subnets are allowed to be stored for each address family. Exceeding that number, older entries will be purged from cache." This section "100-different subnets ... and then purged", how to know if it's doing exactly this? My guessing there is a memory leak, because the large memory it uses. Best regards, -- Eduardo Schoedler
Re: Unbound swapping
Hi Eduardo, I have no real good idea. But looking at your numbers, I see that you are running a network heavy application, unbound, and it uses about 10G on 12G memory. The buff/cache is 2G. Adds up to 12G. And it is swapping. Sounds reasonable, it is maxed out on memory, this is where swap is supposed to make space, right? These options change the buffer space allocated by unbound: so-rcvbuf, so-sndbuf, maybe also so-reuseport, or more tcp connections (that use buffer space) or more network interfaces, or simply a different kernel version that uses (slightly) more memory or something along those lines. What is the limit on the subnet cache size? What does the memory max out on when the subnetmodule is not enabled? Best regards, Wouter On 12/09/17 01:28, Eduardo Schoedler via Unbound-users wrote: > I'm trying to understand why unbound is swapping ~900MB: > > root@dns:~# top -b -n 1 | grep "SWAP\|unbound" > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ > SWAP COMMAND > 11991 unbound 20 0 9751036 8.189g 3824 S 81.2 69.9 3192:03 > 840364 unbound > > > root@dns:~# cat /etc/unbound/unbound.conf | grep '\-size' > ratelimit-size: 16m > rrset-cache-size: 1536m > msg-cache-size: 768m > > This configuration doesn't hurt the "2.5x" tip from documentation. > > root@dns:~# free -m > totalusedfree shared buff/cache > available > Mem: 1200488141296 2371893 > 1449 > Swap: 3395 9072488 > > root@dns:~# unbound-control stats_noreset | grep mem > mem.cache.rrset=110262793 > mem.cache.message=92861937 > mem.mod.iterator=16548 > mem.mod.validator=5767649 > mem.mod.respip=0 > mem.mod.subnet=872423102 > > root@dns:~# unbound-control stats_noreset | grep time > time.now=1505172298.707575 > time.up=265711.841724 > time.elapsed=279.009376 > > root@dns:~# sysctl vm.swappiness > vm.swappiness = 10 > > Some clue? > > Thank you. > > > Regards, > signature.asc Description: OpenPGP digital signature
Unbound swapping
I'm trying to understand why unbound is swapping ~900MB: root@dns:~# top -b -n 1 | grep "SWAP\|unbound" PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ SWAP COMMAND 11991 unbound 20 0 9751036 8.189g 3824 S 81.2 69.9 3192:03 840364 unbound root@dns:~# cat /etc/unbound/unbound.conf | grep '\-size' ratelimit-size: 16m rrset-cache-size: 1536m msg-cache-size: 768m This configuration doesn't hurt the "2.5x" tip from documentation. root@dns:~# free -m totalusedfree shared buff/cache available Mem: 1200488141296 23718931449 Swap: 3395 9072488 root@dns:~# unbound-control stats_noreset | grep mem mem.cache.rrset=110262793 mem.cache.message=92861937 mem.mod.iterator=16548 mem.mod.validator=5767649 mem.mod.respip=0 mem.mod.subnet=872423102 root@dns:~# unbound-control stats_noreset | grep time time.now=1505172298.707575 time.up=265711.841724 time.elapsed=279.009376 root@dns:~# sysctl vm.swappiness vm.swappiness = 10 Some clue? Thank you. Regards, -- Eduardo Schoedler
