Re: Strategy for large cache sets
Darryl Dixon - Winterhouse Consulting wrote: >> Our cache files reside on a SAN where we can get around 500MB/s >> over iSCSI. We went that route because we could not get Linux to >> swap efficiently enough to make malloc feasible. > > I know this is off-topic, but, I'm interested in the 500MB/s > figure... Are you on 10gbit ethernet? No, the boxes only have GigE interfaces. I think our data center uses 10GigE switches from the rack to the SAN, however. -Drew ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Strategy for large cache sets
> Our cache files reside on a SAN where > we can get around 500MB/s over iSCSI. We went that route because we > could not get Linux to swap efficiently enough to make malloc > feasible. I know this is off-topic, but, I'm interested in the 500MB/s figure... Are you on 10gbit ethernet? Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Strategy for large cache sets
Am Dienstag 01 Juli 2008 20:16:15 schrieb Skye Poier Nott: > I want to deploy Varnish with very large cache sizes (200GB or more) > for large, long lived file sets. Is it more efficient to use large > swap or large mmap in this scenario? > > According to the FreeBSD lists, even 20GB of swap requires 200MB of > kern.maxswzone just to keep track of it, so it doesn't seem like that > will scale too well. Is one or the other method better for many > small files vs less many big files? > > Thanks again... when I'm a Varnish expert I'll help the newbs :) > > Skye I'm administrating three varnish instances for static images; cache file is setup as 517 GB (pre allocated with "dd"); currently, after 35 days uptime, it's filled more than half: 305543155712 bytes allocated 248759787520 bytes free It's running on Debian etch, 2x dual-core amd opteron, 32 GB RAM, 30 GB swap: Mem: 32969244k total, 32870944k used,98300k free, 120672k buffers Swap: 29045480k total, 6096612k used, 22948868k free, 25752752k cached the varnish process looks like this in "top": PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ P COMMAND 3221 username 15 0 522g 27g 21g S1 87.0 539:33.89 2 varnishd hitrate is 98% for recent short term, overall average is 95%. load of the machines is between 1 and 2 on medium traffic times, and goes up to 3 only seldom. but, the request rate is may be relatively low, compared to what others reported on the list. I don't have hard numbers unfortunately, average is 80.49 according to varnishstat. At peaks the reate may be in the order of may be 400 req/sec for a single instance. We had issues with 1.1.2 crashing, but since running on trunk (r2640), everything runs smooth. response time according to a nagios http response check is between 0.5 - 1 seconds, almost never over 1 second, even at peak times. Hope it's useful for someone, let me know if you need more details. Cheers, Sascha ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Strategy for large cache sets
Simon Lyall wrote: > I was wondering how large a cache people are managing to use without any > problems? Maybe a survey: Version: trunk-r2646 Config: file backend Traffic: 90 req/s avg, 1.5-2K peak RAM: 12GB, 1GB swap Cache Size: 36GB reserved, 24GB allocated OS: Linux 2.6.18-6-amd64 (Debian 4.0r3) This represents one of our image cache boxes, with a 99.6% hitrate, in a pool fronted with nginx. Our cache files reside on a SAN where we can get around 500MB/s over iSCSI. We went that route because we could not get Linux to swap efficiently enough to make malloc feasible. Except for the occasional flood of connections which will cause varnishd to consume more RSS than we've allotted, it's been extremely stable and performant. -Drew ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
RE: Strategy for large cache sets
We are running varnish with a huge dataset (+5TB) on our backends, we've been trying a lot of different setups but we've also ran into the problem with load spikes with every configuration that cahches more than the amount of ram we have, both mmap and malloc with larger swap. Right now we are running trunk with tcmalloc and 55gb cachestore (we have 64gb ram in the frontends now) we have about 70% cachehits and it's very stable, but we're still looking for a way to use mmap or malloc+swap to be able to have a cachestore over 100gb. So I'm also very interested in your results. Regards Calle -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Barry Abrahamson Sent: den 2 juli 2008 17:59 To: varnish-misc@projects.linpro.no Subject: Re: Strategy for large cache sets On Jul 1, 2008, at 2:16 PM, Skye Poier Nott wrote: > I want to deploy Varnish with very large cache sizes (200GB or more) > for large, long lived file sets. We are doing this also and running into performance problems. We have tried files and swap Running on Linux (Debian). When the cache starts to get large, we start to see huge load spikes (loads of 200+) caused by IO wait but no corresponding spikes in request rates or any of the varnish metrics (except threads running which I am pretty sure is a result of the load spike and not the cause). We are currently running 1.1.2 but have tried with trunk and the same thing happens. If you find anything useful in your testing, I would love you hear about it. -- Barry Abrahamson | Systems Wrangler | Automattic Blog: http://barry.wordpress.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Strategy for large cache sets
Hi, On Tue, Jul 01, 2008 at 11:16:15AM -0700, Skye Poier Nott wrote: > I want to deploy Varnish with very large cache sizes (200GB or more) > for large, long lived file sets. Is it more efficient to use large > swap or large mmap in this scenario? > > According to the FreeBSD lists, even 20GB of swap requires 200MB of > kern.maxswzone just to keep track of it, so it doesn't seem like that > will scale too well. Is one or the other method better for many small > files vs less many big files? My experience with Varnish on FreeBSD with long lived (~1 week) large data sets tells me that using the file storage backend easily gives you 60-70 second hangs. The malloc backend works smoother. I've been using 256 MB maxswzone on a few servers with upto 80 GB of data in the swap and did not have any problems with maxswzone beeing too small. That said, I do get large peaks in number of threads and vm faults with peak/high traffic, which makes it difficult to scale further. I don't know if this is due to bottlenecks in the VM subsystem, Varnish or if I have too little RAM. But I hope to find out more about it. I suspect there is more work needed in this area to be done by the developers. PS: FreeBSD supports swap devices upto only 32 GB, so you may need to split your disks/volumes up in many partitions. Bye, -- Anders. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Strategy for large cache sets
On Wed, 2 Jul 2008, Barry Abrahamson wrote: > On Jul 1, 2008, at 2:16 PM, Skye Poier Nott wrote: >> I want to deploy Varnish with very large cache sizes (200GB or more) >> for large, long lived file sets. > > We are doing this also and running into performance problems. > about it. Personally I'm keeping with the "cache" bit in the title and only trying to keep a subset of may data in Varnish. Then again my usage patterns are more concentrated. I was wondering how large a cache people are managing to use without any problems? Maybe a survey: Version: Special Features/Config: Hits/Second: RAM: Total Cache Size: OS: Comments: BTW: I'm still on the testing stage for various non-technical reasons. -- Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Strategy for large cache sets
On Jul 1, 2008, at 2:16 PM, Skye Poier Nott wrote: > I want to deploy Varnish with very large cache sizes (200GB or more) > for large, long lived file sets. We are doing this also and running into performance problems. We have tried files and swap Running on Linux (Debian). When the cache starts to get large, we start to see huge load spikes (loads of 200+) caused by IO wait but no corresponding spikes in request rates or any of the varnish metrics (except threads running which I am pretty sure is a result of the load spike and not the cause). We are currently running 1.1.2 but have tried with trunk and the same thing happens. If you find anything useful in your testing, I would love you hear about it. -- Barry Abrahamson | Systems Wrangler | Automattic Blog: http://barry.wordpress.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Strategy for large cache sets
In message <[EMAIL PROTECTED]>, Skye Poier Nott writes: >Thanks, that was my hunch. I'll let you know how it goes when I have >some performance metrics. Input is very much appreciated. I can also recommend setting up Munin or similar to plot all the varnishstat and systat variables, that has proven to be quite helpful to figuring out what dynamics caused what problems. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 [EMAIL PROTECTED] | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Strategy for large cache sets
Thanks, that was my hunch. I'll let you know how it goes when I have some performance metrics. Skye On 1-Jul-08, at 11:24 AM, Poul-Henning Kamp wrote: > In message <[EMAIL PROTECTED]>, Skye Poier > Nott writes > : >> I want to deploy Varnish with very large cache sizes (200GB or more) >> for large, long lived file sets. Is it more efficient to use large >> swap or large mmap in this scenario? > > We have no real-world experience with content of that size, so > the answer is: we don't know. > > Off the bat, I would think files would be better, for exactly the > reson you cite: the swap management is pretty thirsty after metadata. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > [EMAIL PROTECTED] | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by > incompetence. > ___ > varnish-misc mailing list > varnish-misc@projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Strategy for large cache sets
In message <[EMAIL PROTECTED]>, Skye Poier Nott writes : >I want to deploy Varnish with very large cache sizes (200GB or more) >for large, long lived file sets. Is it more efficient to use large >swap or large mmap in this scenario? We have no real-world experience with content of that size, so the answer is: we don't know. Off the bat, I would think files would be better, for exactly the reson you cite: the swap management is pretty thirsty after metadata. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 [EMAIL PROTECTED] | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
