Re: BIND-9.16.1 memory leak?
On Friday, 17 April 2020 08:45:16 CDT Steinar Haug wrote: > We have what appears to be a significant memory leak in BIND-9.16.1. > > Environment: > FreeBSD 12.1-STABLE. > BIND-9.16.1 installed from packages. > Also uses libuv-1.35.0 installed from packages. > Authoritative only. > Around 800 zones of varying sizes. DNSSEC in use. > https://gitlab.isc.org/isc-projects/bind9/-/issues/1893 -- Greg ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: BIND-9.16.1 memory leak?
On 2020-04-17 06:45, sth...@nethelp.no wrote: > We have what appears to be a significant memory leak in BIND-9.16.1. > > Environment: > FreeBSD 12.1-STABLE. > BIND-9.16.1 installed from packages. > Also uses libuv-1.35.0 installed from packages. > Authoritative only. > Around 800 zones of varying sizes. DNSSEC in use. Additional datum, as I am seeing the same thing: - FreeBSD 12.1-RELEASE-p3 - BIND-9.16.1 compiled from ports/poudriere via a local package build server (no options changed, though, so it likely could have been installed from the FreeBSD package repo). - Authoritative only - `rndc status` reports 1058 zones (69 automatic) - Host is a VM with 16GiB allocated and 4 CPU cores - named running for approx 2.5 weeks (wall-clock) Current BIND status (from `top`): PID USERNAMETHR PRI NICE SIZERES STATEC TIMEWCPU COMMAND 1707 bind 14 520 5312M 5260M sigwai 2 34.4H 5.79% named A recursive-only server, running the same versions of all software, on an identically-provisioned VM, running for the same amount of wall-clock time (approximately 2.5 weeks) looks like this: PID USERNAMETHR PRI NICE SIZERES STATEC TIMEWCPU COMMAND 1485 bind 14 520 927M 890M sigwai 3 89.6H 32.86% named The recursive memory footprint looks normal. Contrast that with a separate server: - FreeBSD 11.3-RELEASE-p7 - BIND 9.14.11 compiled from ports/poudriere via a local package build server (no options changed, though, so it likely could have been installed from the FreeBSD package repo). - Authoritative only + recursive only running in a separate jail - Same configuration as above, only a bit busier - Host is standalone with 96GiB RAM and 8 cores In the `top` output below, both the jailed named processes are shown. The busier one is the authoritative-only: PID USERNAME THR PRI NICE SIZERES STATE C TIMEWCPU COMMAND 896 bind 18 520 956M 927M sigwai 0 99.2H 30.03% named 1584 bind 18 520 1171M 1080M sigwai 2 166.2H 13.47% named It definitely looks like a memory leak in 9.16.1 when configured as authoritative-only. The leak seems slow enough as to be manageable, but the footprint does appear to growing monotonically (and is still growing--by another 4M as I wrote this email). michael ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: BIND-9.16.1 memory leak?
On Sun, Apr 19, 2020 at 02:55:53PM +0200, sth...@nethelp.no wrote: > I now have a pcap file of queries that I can replay with the "drool" > application, and I'm consistently seeing similar memory leak problems > (i.e. the problems are reproducible). The memory leak rate seems to be > very approximately linear with the query rate - so replaying at 10 > times the original speed means we also leak around 10 times as much per > minute. > > Upgrading to 9.16.2 (and also libuv 1.36.0) makes no difference - the > same memory leak is observed. Is there anything unusual in your server configuration? -- Evan Hunt -- e...@isc.org Internet Systems Consortium, Inc. ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: BIND-9.16.1 memory leak?
Followup: > We have what appears to be a significant memory leak in BIND-9.16.1. > > Environment: > FreeBSD 12.1-STABLE. > BIND-9.16.1 installed from packages. > Also uses libuv-1.35.0 installed from packages. > Authoritative only. > Around 800 zones of varying sizes. DNSSEC in use. > > Running a ps command for the named process every minute and logging > the result, I see the named virtual memory size (VSZ) increasing at > around 1.2 Mbyte/minute, and the resident size (RSS) increasing at > around 0.85 Mbyte/minute. No problems due to this so far, but pretty > obviously it's not viable in the long run. > > I tried reading the CHANGES from 9.16.2, and didn't see anything which > suggested a fix for a memory leak problem. I now have a pcap file of queries that I can replay with the "drool" application, and I'm consistently seeing similar memory leak problems (i.e. the problems are reproducible). The memory leak rate seems to be very approximately linear with the query rate - so replaying at 10 times the original speed means we also leak around 10 times as much per minute. Upgrading to 9.16.2 (and also libuv 1.36.0) makes no difference - the same memory leak is observed. Steinar Haug, Nethelp consulting, sth...@nethelp.no ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: BIND-9.16.1 memory leak?
On 17/04/2020 17:02, Karl Pielorz wrote: Hi Karl, > I seem to remember we got 'bitten' by large memory use when moving from > a previous version of bind - do you have 'max-cache-size' set in your > config? It's an authoritative-only server, so there is (almost) no caching involved. Anand ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: BIND-9.16.1 memory leak?
>> We have what appears to be a significant memory leak in BIND-9.16.1. ... > I seem to remember we got 'bitten' by large memory use when moving > from a previous version of bind - do you have 'max-cache-size' set in > your config? Yes. Set to 1G. In reality it shouldn't need a cache at all, since this is a purely authoritative server (recursion no). So this doesn't appear to be the problem. But thanks for the suggestion! Steinar Haug, Nethelp consulting, sth...@nethelp.no ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: BIND-9.16.1 memory leak?
--On 17 April 2020 at 15:45:16 +0200 sth...@nethelp.no wrote: We have what appears to be a significant memory leak in BIND-9.16.1. ... Running a ps command for the named process every minute and logging the result, I see the named virtual memory size (VSZ) increasing at around 1.2 Mbyte/minute, and the resident size (RSS) increasing at around 0.85 Mbyte/minute. No problems due to this so far, but pretty obviously it's not viable in the long run. I tried reading the CHANGES from 9.16.2, and didn't see anything which suggested a fix for a memory leak problem. Any suggestions? Hi, I seem to remember we got 'bitten' by large memory use when moving from a previous version of bind - do you have 'max-cache-size' set in your config? As far as I can remember, the 'default' is to take 90% of the memory on the machine. This is great, unless the machine has lots of other stuff going on etc. I think we noticed this during bind startup (i.e. from syslog output). On our boxes we set it to "something sensible" - rather than using the default. Might not be your problem - but thought it was worth mentioning / checking. -Karl ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
BIND-9.16.1 memory leak?
We have what appears to be a significant memory leak in BIND-9.16.1. Environment: FreeBSD 12.1-STABLE. BIND-9.16.1 installed from packages. Also uses libuv-1.35.0 installed from packages. Authoritative only. Around 800 zones of varying sizes. DNSSEC in use. Running a ps command for the named process every minute and logging the result, I see the named virtual memory size (VSZ) increasing at around 1.2 Mbyte/minute, and the resident size (RSS) increasing at around 0.85 Mbyte/minute. No problems due to this so far, but pretty obviously it's not viable in the long run. I tried reading the CHANGES from 9.16.2, and didn't see anything which suggested a fix for a memory leak problem. Any suggestions? Steinar Haug, Nethelp consulting, sth...@nethelp.no ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
