Hi Guillaume, This time it is I who needs to apologize for a late reply (7 months later).
We are strictly on the LTS diet, I'm afraid. So we can't upgrade the major version until there is a new LTS series of Varnish. So it's 6.0.x for a while still I guess. We are on Ubuntu 20 though, and I guess we could update to the latest LTS version there. ________________________________ From: Guillaume Quintard <[email protected]> Sent: Thursday, June 13, 2024 10:07 PM To: Batanun B <[email protected]> Cc: [email protected] <[email protected]> Subject: Re: Varnish suddenly started using much more memory Sorry Batanun, this thread got lost in my inbox. Would you be able to upgrade to 7.5 and see if you get the same results? I'm pretty sure it's a jemalloc issue, but upgrading should make it clear. You are on Ubuntu, right? Which version? -- Guillaume Quintard On Mon, May 20, 2024 at 1:50 AM Batanun B <[email protected]<mailto:[email protected]>> wrote: > Sorry, I should have been clearer, I meant: where are the varnish packages > coming from? Are they from the official repositories, from > https://packagecloud.io/varnishcache/ or built from source maybe? Ah, I see. They come from varnishcache packagecloud. More specifically, we use: https://packagecloud.io/install/repositories/varnishcache/varnish60lts/script.deb.sh > you should really invest some time in something like prometheus, it would > probably have made the issue obvious Yes, in hindsight we definitely should have done that. I will discuss this with my coworkers going forward. > Is there any chance you can run the old version on the server to explore the > differences? Possibly, for a limited time. If so, what types of tests would I do? And how long time would I need to run the old version? Note that with our setup, we wouldn't be able to run two different images at the same time, in the same environment, with both recieving traffic. So all traffic would be routed to this version (multiple servers, but all running the same image). An alternative approach that I'm considering, is to switch to the old image, but manually update the VCL to the new version. If the problem remains, then the issue is almost certainly with the VLC. But if the problem disapears, then it's more likely something else. > what's the output of: varnishstat -1 -f '*g_bytes' SMA.default.g_bytes 10951750929 . Bytes outstanding SMA.large.g_bytes 8587329728 . Bytes outstanding SMA.Transient.g_bytes 3177920 . Bytes outstanding So, the default storage usage has gone up with 2GB since my first message here, while the others have remained the same. Meanwhile, the total memory usage of Varnish has gone up to 26 GB, an increase of 3 GB. So now the overhead has gone up with 1GB to a total of 6 GB. Going forward, it will be interesting to see how the memory consumption changes after the default storage has reached its max (2 GB from where it is now). If we're lucky, it will stabilize, and then I'm not sure if it's worth it to troubleshoot any further. Otherwise, the free memory would get a bit too close to zero for our comfort, with no indication of stopping. Does Varnish keep track of total available OS memory, and start releasing memory by throwing out objects from the cache? Or will it continue to eat memory until something fails? > have you tweaked any workspaces/thread parameters? Nope. As I said, we haven't changed any OS or Varnish configuration. _______________________________________________ varnish-misc mailing list [email protected]<mailto:[email protected]> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
_______________________________________________ varnish-misc mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
