We have several varnishd instances that have been running fine for years at constant memory use. We've recently decided to upgrade our linux distribution. After some troubleshooting, we were unable to find a configuration of Varnish on the new distro version that did not have dramatically higher memory usage. After that, we tried upgrading to a modern, supported version of Varnish, but were also unable to find a configuration that doesn't use dramatically more memory. We have tried the following configurations:
* Varnish 3.0.3, Ubuntu 10.04, Linux 3.2, 24G malloc storage, glibc malloc, constant memory use at 29G rss * Varnish 3.0.5, Ubuntu 14.04, Linux 3.19, 24G malloc storage, jemalloc, memory usage grows until it passes 64G and starts swapping * Varnish 4.1.0, Ubuntu 14.04, Linux 3.19, 24G malloc storage, jemalloc, memory usage grows until it passes 64G and starts swapping * Varnish 4.1.0, Ubuntu 14.04, Linux 3.19, 24G malloc storage, glibc malloc, memory usage grows faster than with jemalloc until it passes 64G and starts swapping I've reviewed varnishstat output, but I've been unable to account for the increased memory usage. I asked the IRC channel for support a few weeks ago, and nobody else there was able to account for this in varnishstat, and the only idea that we were able to come up with was to try glibc malloc with Varnish 4.1, which as you can see above turned out to be slightly worse. I'm really out of ideas here; any help in trying to find a configuration that has constrained memory usage on a more-modern platform would be highly appreciated. Details on the functional Varnish 3 instances with a steady 29G memory usage: Command line: /usr/sbin/varnishd -P /var/run/varnishd.pid -a :3000 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -p sess_timeout 60 -p thread_pool_min 500 -p thread_pool_max 2000 -p nuke_limit 1000 -s malloc,24G varnishstat -1: http://sprunge.us/iKUC vcl config: http://sprunge.us/LOUE libraries linked: http://sprunge.us/FOSL Details on the Varnish 3 instances with much higher memory usage: Command line: /usr/sbin/varnishd -P /var/run/varnishd.pid -a :3000 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -p sess_timeout=60 -p thread_pool_min=500 -p thread_pool_max=2000 -p nuke_limit=1000 -s malloc,24G varnishstat -1: http://sprunge.us/KGIa vcl config: Same config as above, http://sprunge.us/LOUE Details on the varnish 4.1 instances with much higher memory usage: Command line: /usr/sbin/varnishd -P /run/varnishd.pid -a :3000 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -p timeout_idle=60 -p thread_pool_min=500 -p thread_pool_max=2000 -p nuke_limit=1000 -s malloc,24G varnishstat -1: http://sprunge.us/gCOG vcl config: http://sprunge.us/LQRM libraries linked: http://sprunge.us/WCbj Graph of memory usage for two varnish instances. AF002245 is Varnish 3.0.3, AF002256 is Varnish 4.1.0: http://i.imgur.com/Fxsms48.png The closest I've been able to find on Google is people talking about accidentally using too much Transient space, but in varnishstat I see <800k used in transient: SMA.Transient.g_bytes 766728 . Bytes outstanding They both have comparable numbers of objects cached: n_object 419597 . N struct object MAIN.n_object 420371 . object structs made I'm pretty sure I've got to have something bad in my config, because I'm seeing the same behaviour on both 3.0.5 and 4.1.0, and a memory leak this bad for so long seems very unlikely. I've considered trying to narrow this down by porting forward exactly the same 3.0.3 version to a slightly more modern platform, or trying to backport 4.1.0 back to the ancient Ubuntu 10.04, but I'm not really sure what I'd do differently if either of those worked, as I would rather not stay on such an old platform if I can avoid it, in either case. Are there any other metrics I can gather here? Can anyone give me any recommendations on how I can configure Varnish to constrain its memory usage? Any ideas about what I'm missing? _______________________________________________ varnish-misc mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
