The usage of varnish, revisited
Dump from my blog posting at http://ingvar.blog.linpro.no/ Ingvar The usage of varnish, revisited This is more or less a repost, with updated numbers. Some months have passed, and it is time to run my poking scripts again, looking for sites that run Varnish. There is no deep magic here. I just parse the available top lists that I know of, and peek at the HTML headers of the sites that are listed. If there are subsites linked from the front page of the site, I scan them too. This means that twitter.com shows up, though Twitter only runs Varnish on its search site. Subsites with a Varnish match are shown in parenthesis in the results. For the Nordic countries, I have found quite good lists, that is, upload result lists from the probably most visited media sites in the respective countries. Remember of course, that these are generally pay-to-be-included lists, and there may exist sites with far more hits than the ones listed. For a global overview, I have used Alexa. Now for the results. Varnish is sponsored by large Norwegian sites, so it is no big surprise that there are a lot of hits in Norway. Of the TNS Gallup top list, Varnish runs at 36 of the top 100 sites. That’s 3 up since my last probe. For Denmark, I use FDIM’s list. From May, we are up from 3 to 11 sites in the top 100. For Finland, I use TNS’ numbers again. No changes there. For Sweden, I use the KIA Index list. I now probe the 200 top sites, so there are several more varnish sites on the list. In the top 100, we are up from 8 to 9. For the Alexa’s World top 500 list, I have tweaked my filters a bit, and the list is up from 7 to 17 sites since my last probe in May. World Domination, here we come! The whole World: Alexa's global top 500 list Place 12 Varnish running on twitter.com (http://integratedsearch.twitter.com/search.html) Place 47 Varnish running on photobucket.com (http://blog.photobucket.com/) Place 90 Varnish running on mixi.jp (http://img.mixi.jp/static/css/basic/logout_quirks) (and others) Place 101 Varnish running on weather.com (http://www.weather.com/) (weather.com) Place 107 Varnish running on globo.com (globo.com) (http://www.globo.com/) Place 111 Varnish running on ifeng.com (http://big5.ifeng.com/gate/big5/www) Place 138 Varnish running on answers.com (http://www.answers.com/) (answers.com) Place 170 Varnish running on orange.fr (http://r.orange.fr/r/Eorangepublicite) Place 179 Varnish running on hulu.com (hulu.com) Place 199 Varnish running on wikia.com (wikia.com) Place 213 Varnish running on xinhuanet.com (http://big5.xinhuanet.com/gate/big5/www) Place 290 Varnish running on people.com.cn (http://bbs1.people.com.cn/boardList) (and others) Place 344 Varnish running on tuenti.com (http://estaticos1.tuenti.com/layout/web2/images/favicon) Place 428 Varnish running on chinanews.com.cn (http://www.chinanews.com.cn/test/index_back_d.html) (and others) Place 433 Varnish running on mercadolivre.com.br (http://veiculos.mercadolivre.com.br/) (and others) Place 447 Varnish running on mercadolibre.com.mx (http://tendencias.mercadolibre.com.mx/) (and others) Place 456 Varnish running on break.com (break.com) In my last probe, I poked sites all over Europe. With a few exceptions, that was a bit less interesting. Global .com and .net sites tend to cover most of the top 100 entries, as I had only toolbar lists, and it’s not that spectacular that for example people in Serbia and Monte Negro are browsing Twitter, like the rest of the World does. So I have skipped other countries. If you know of any good top list for you country that is not toolbar based, please let me know. All the gory details are available here: http://users.linpro.no/ingvar/varnish/stats-2010-01-18.txt Other more or less worth mentioned sites that is reported to use Varnish but does not show up in my lists, may be Slashdot, WAT TV, The Pirate Bay, JDownloader, e.Republik, WOWwiki, Globo.com, PCWelt.de, BlackPlanet, funnyordie.com and n-tv.de, to name a few. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
RPM-packages of varnish-2.0.5 for RHEL and Fedora available
I have submitted varnish-2.0.5 for Fedora and Fedora EPEL, and updates to the stable releases will be requested, so they will trickle down to the stable repos in a few weeks. For RHEL, both el4 and el5 packages are now in the EPEL testing repo. For those who are too impatient to wait for stable, or want to participate in testing, you can download the package with yum: rhel5# yum --enablerepo=epel-testing update varnish ... or download the package from RedHat: http://download.fedora.redhat.com/pub/epel/testing/ Fedora packages are still pending for testing, but will be visible in a few days, I guess. If you need packages for Fedora now, try http://kojipkgs.fedoraproject.org/packages/varnish/ Bugs in the package can be reported in Red Hat's Bugzilla: http://bugzilla.redhat.com/ or to varnish-d...@projects.linpro.no. Ingvar ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: varnish-misc Digest, Vol 23, Issue 25
* Fredrik Nygren > Alright. Wonder then if somebody has any rpm's to share for Redhat EL4 > i386 with the patch included? Post a request with a short recap of the discussion to the varnish-dist list, and I'll consider it. Ingvar -- Buddha wears an iPod ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Setting default_ttl in vcl file
I take this off varnish-misc as it is quite distro specific. * Ingvar Hagelund >> The RedHat and Debian init scripts reads default values from their >> respective defaults file, that is, for RedHat, /etc/sysconfig/varnish, >> and for Debian, /etc/defaults/varnish. * Dag-Erling Smørgrav > The problem is that if you define DAEMON_OPTS, none of the other knobs > work, right? FreeBSD at least has that issue: > > if you define varnishd_flags, you have to specify everything manually, > since the other knobs only serve to populate the default value for > varnishd_flags. At least for RedHat, there are no other knobs. All the startup configuration is in the file. There are no hard coded values in the init script. The init script won't start varnishd if DAEMON_OPTS is empty, or the config file does not exist. if [ "$DAEMON_OPTS" = "" ]; then echo "\$DAEMON_OPTS empty." echo -n "Please put configuration options in /etc/sysconfig/varnish" echo_failure The configuration file is fairly documented with the most used options, and points to the manpage for more advanced users. I would not know if this is suitable for FreeBSD ports, but I think it's quite a clean method to ensure that the user knows what she wants to change before changing it. As for myself, I dislike magic "invisible" default values. Ingvar -- Buddha wears an iPod ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Setting default_ttl in vcl file
* Steve Scally >>> Now my question, is there a more direct way to change the default_ttl >>> either through configuration files or .vcl file? If these are pretty >>> much the only three ways that is fine, I just wanted to make sure I >>> wasn't skipping over any options. Thank you in advance for your help. >>> * Dag-Erling Smørgrav > The default TTL can be configured on the command line with -t ttl or > (as any other run-time parameter) with -p default_ttl=ttl. The > FreeBSD rc script does not currently have a variable for setting the > default TTL directly, nor a mechanism for setting run-time parameters > in general. I could either add code to the rc script to read in a > list of run-time parameters and pass them on the command line, or we > could add code to varnishd to read initial values from a file. I > prefer the former, but we need to add similar code to the Debian and > RedHat init scripts as well. > The RedHat and Debian init scripts reads default values from their respective defaults file, that is, for RedHat, /etc/sysconfig/varnish, and for Debian, /etc/defaults/varnish. In that file, you can add any command line parameter to varnishd, not only those that are commented in the file. Do we need any other mechanism for adding runtime parameters? A full-fledged configuration file parser in varnishd? Ingvar redhat$ tail -4 /etc/sysconfig/varnish ## Alternative 4, Do It Yourself. See varnishd(1) for more information. # # DAEMON_OPTS="" ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Setting default_ttl in vcl file
This mail has stayed unanswered on varnish-dist, but I think someone on
varnish-misc may be able to answer.
Ingvar
Steve Scally <[EMAIL PROTECTED]> said on [EMAIL PROTECTED]
> Hello,
>
> I have just started to use varnish and was looking into the default
> cache time of an object / item, etc. I know the default_ttl is 120
> seconds and I was looking to change that to 3600 seconds. From the
> documentation and previous mail archives I have found three ways to
> set this. Two ways are through the rc.d and rc.conf files and the
> last is through the vcl.
>
> A. /usr/local/etc/rc.d/varnishd file
>
> Create a variable called varnishd_ttl and set it to 3600.
> : ${varnishd_ttl="3600"}
>
> Edit varnishd_flags and add my new variable
> : ${varnishd_flags="-P ${pidfile} -a ${varnishd_listen} -f $
> {varnishd_config} -T ${varnishd_telnet} -s ${varnishd_storage} -t $
> {varnishd_ttl}"}
>
> B. /etc/rc.conf
>
> Create a variable called varnishd_ttl and set it to 3600.
> : ${varnishd_ttl="3600"}
>
> Edit varnishd_flags and add my new variable
>
> varnishd_enable="YES"
> varnishd_listen="mb-prv.kcilink.com:80"
> varnishd_storage="file,/var/tmp,512M"
> varnishd_config="/usr/local/etc/varnish/morebiz.vcl"
> varnishd_ttl="3600"
> varnishd_flags="-P ${pidfile} -a ${varnishd_listen} -f $
> {varnishd_config} -T ${varnishd_telnet} -s ${varnishd_storage} -t $
> {varnishd_ttl}"
>
> c. /usr/local/etc/varnish/default.vcl
>
> *if (obj.ttl < 3600s) {
> set obj.ttl = 3600s;
> }
>
> *not exact just scratch code.
>
> Now my question, is there a more direct way to change the default_ttl
> either through configuration files or .vcl file? If these are pretty
> much the only three ways that is fine, I just wanted to make sure I
> wasn't skipping over any options. Thank you in advance for your help.
>
> Steve
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: docs, howto and best practices
* Sébastien FOUTREL > Unfortunately, I already read that articles and it has only a bit of meat to > feed my hunger. > > Where can I found vcl languages docs ? > The website and my googling did not gave me good results. I don't think there exists more, at least not in publishable form. Dag-Erling? I agree that documentation of the vcl language, and some best-practice notes and advice is poorly needed for the project. On the other hand, at least, we now have the man pages. We all have to prioritize how we use our time, and that of course includes the main developers, who probably wisely use most of their varnish time on squashing bugs and programming for varnish-2. Dag-Erling, is it possible to push some documentation production up the prio list? > I know basics but wants to go further. > Varnish is already production used on a 7M hits/D website and it works > perfectly. Post your config here and ask about comments for it. Then tell us what else you want to do with varnish, and if you are lucky, some on the list may tell you more. Ingvar -- ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: docs, howto and best practices
* Sébastien FOUTREL > I'am relatively new to varnish, but the more I use it, the more I like it. > There is only a black point on my pleasure. > There is not enough docs available. > I browsed the web site and google and did not find enough meat to feed my > beast. > Where can I find vcl languages docs (apart man pages) ? > Could you please give me your ressources on Varnish. > I'am ready to use it and, if developers are ok, to feed the wiki to help > other people like me. > Thank You, > Hello, Sébastien Unfortunately, there is still quite a lack of more fleshy documentation than the man pages. John Buswell wrote a nice article in issue #6 of o3magazine. It's available online at http://www.emea.o3magazine.com/pastissues/issue6/ . It includes some starting point vcl code and some general advice. You should start with that article and the man pages, and then be welcome to return to the mailing list for more help. Regards, Ingvar ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Redhat
* jdouglas > Are there instructions on installing this on Redhat systems? > > I am able to get it working on FreeBSD 6.2 but not RHEL 4 with a 2.6 Kernel. > > When I try to build an rpm from spec i get: > (...) > > RPM build errors: > Installed (but unpackaged) file(s) found: >/usr/include/varnish/shmlog.h >/usr/include/varnish/shmlog_tags.h >/usr/include/varnish/stat_field.h >/usr/include/varnish/stats.h >/usr/include/varnish/varnishapi.h >/usr/lib/pkgconfig/varnishapi.pc This is fixed in trunk. The fix will also be included in the upcoming 1.1.1 release. Ingvar -- At barnebokfigurer er snille er bare ljug Mummipappa dater Barbamamma i smug ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: varnish(log|nscsa|....) > varnishlog,> Cannot open NONE
* Jean-Marc Pouchoulon > just build varnish 1.1.1 rpm from svn on linux fedora 7 > #129 is fixed thanks to the developpers :-) > But I have this strange message on varnish utility: > >> varnishlog >> Cannot open NONE/var/varnish/localhost.localdomain/_.vsl: No such file >> or directory > I built varnish from source , I get the same "error" > Using varnishlog on varnish.log works fine. > > NB : > To build varnish rpm I must add this line in .SPEC >> %define _unpackaged_files_terminate_build 0 These are related. The file /var/varnish does not exist on your system, as it's not in the build (and by doing that define, you force the packge to be built anyway). The rpm specfile is not updated to 1.1.1. It's on my todolist. Ingvar -- ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: log files
* Gaute Amundsen >> Did you ever get around to writing that init script? If not, how >> would you recommend going about it? >> >> Adding it to /etc/init.d/varnish, or perhaps a separate initscript? >> >> This would be the bare necessities I guess? >> >> /usr/bin/nohup \ >> /usr/bin/varnishncsa -c -a -w /var/log/httpd/varnish_combined-access_log & * Stig Sandbeck Mathisen > One separate init script (for Debian) is available at > http://varnish.projects.linpro.no/browser/trunk/varnish-cache/debian/varnish.varnishlog.init Just for the record, the RedHat subdirectory has a similar script. It is also included in the RHEL packages at sourceforge and in the Fedora package. Ingvar -- At barnebokfigurer er snille er bare ljug Mummipappa dater Barbamamma i smug ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: "Whitescreen" issue with varnish 1.0.2-2 (debian etch)
* Kenneth Rørvik >> I saw the same problems with 1.0.2 on redhat - empty documents >> returned after a while, possibly related to filling up the disk backend file. * Denis Brækhus >> 1.0.3 fixed it here, I didn't look deeper into it. > > 1.0.3 worked here too. I always intended to run 1.0.3, but I wanted to test > the etch included package just for the fun of it. > It's a bit scary that the package in etch can give a newcomer to Varnish the > impression that it is not really stable/production ready though, I am afraid > new users could get a bad start. Rumors says 1.0.4 is on the stairs, banging at the door. It will probably include an update to the Debian package. I don't know if changes/updates will trickle into etch over the time. Stig, Lars? Ingvar -- Når alt annet feiler: Symlink ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: branches/1.0 updated
* Dag-Erling Smørgrav > I've merged trunk into branches/1.0. > > I actually only intended to merge up to r1288, but SVK ignored the > range I specified and merged everything. > > This does not mean I'm preparing a release (especially now that I > can't trust SVK to DTRT), just that I consider some of the changes in > trunk to be sufficiently stable and sufficiently useful that it might > make sense for people to track branches/1.0 to get something a little > newer than the latest release, but not quite bleeding edge. > > Specifically, I consider trunk up to and including r1288 to be both a > significant improvement over 1.0.3 and relatively low-risk. Changes > in this range include improved backend handling, a fix for a > (non-exploitable) buffer overflow in the management interface, and > improved documentation. If these are not changes enough to roll 1.0.4, when (if ever) can we expect a new 1.0 release? > > The corresponding revision on branches/1.0 is r1331, which you can > check out with the following Subversion command: > > % svn co -r1331 http://varnish.projects.linpro.no/svn/branches/1.0 varnish With these improvements, would you say it's still appropriate to run 1.0.3 in production environments? Ingvar -- ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Packaging: A need for a devel package?
* Matthias Saou > From what Dag-Erling answered, it seems like the devel package might > make sense. Since the documentation mentioned is missing, I'm going to push the devel package for later/request, as proposed by Poul-Henning. > If you do decide to have one, your example above isn't good > (at least for Red Hat and Fedora, SuSE, Mandriva and others do things > differently), as you would need to have for instance : > > varnish (the main package with the daemon) > varnish-libs > varnish-devel (and not "varnish-libs-devel") It's kind of strange, as I get different answers every time I ask anybody about this :-) At the moment, I have a package for review for Fedora. Matthias, could you post comments in RedHat Bugzilla, Bug #230275, please? > The "libs" only make sense to split out if some programs could require > them without requiring the main daemon. Again, from what Dag-Erling > wrote, maybe this would make sense if someone writes a varnishlog file > parser. I guess I will to keep the libs package for future use. It's complete, and thus easier to cope with than a non-existing list of header files and documentation. > Attached are the files I used to build the latest varnish package I > used, in case they can be of any help. Yes, the changes to the initscript and the configuration file are absolutely interesting, though I might insist on putting the config file in /etc/sysconfig. Ingvar -- ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Packaging: A need for a devel package?
I am putting some effort in the (RedHat) rpm package again. I got a question about a devel package. Usually, on RedHat based systems, one typically puts things like static libraries and header files in a devel package, like "varnish-libs-devel-1.0.3-5.i386.rpm" for instance. Now, I wonder: Would it be appropriate with a devel package at all? Is it thinkable that anyone would use varnish technology to build things outside varnish itself? If so, I could use a list of actual header files and a suggestion on where to put them (/usr/include/varnish?), and maybe some starting point hacking documentation, if that's feasible. If this seems nonsense, please tell, and I'll just skip the devel package. Ingvar -- ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Packaging: A need for a devel package?
Repost, just to make sure it comes through after the greylisting problem. I am putting some effort in the (RedHat) rpm package again. I got a question about a devel package. Usually, on RedHat based systems, one typically puts things like static libraries and header files in a devel package, like "varnish-libs-devel-1.0.3-5.i386.rpm" for instance. Now, I wonder: Would it be appropriate with a devel package at all? Is it thinkable that anyone would use varnish technology to build things outside varnish itself? If so, I could use a list of actual header files and a suggestion on where to put them (/usr/include/varnish?), and maybe some starting point hacking documentation, if that's feasible. If this seems nonsense, please tell, and I'll just skip the devel package. Ingvar -- ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
RPM packages of varnish-1.0.3 available
I have just uploaded rpm packages of varnish-1.0.3 to sourceforge. x86_64, i386 and src flavours available. They should spread to worldwide mirrors in a few hours. I'll see if I get around to build some ppc64 packages too. Most relevant changes in the rpm package: It's now three packages, varnish, libvarnish1 and libvarnish1-devel. This way it conforms more to the RedHat/Fedora standards. There are also some other small changes to make it pass the needle eye of rpmlint. Most intrusive, I converted the manpages to utf8. Dag-Erling, you might want to consider this for the upstream package too. Ingvar -- When everything else fail: Symlink ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Missing files in the varnish 1.0.3 release
* Gaute Amundsen > Just today, I got asked by our server guy, if he should upgrade from 1.0.1 to > 1.0.3, to se if that would fix some problems we have had reported. > > I am not yet convinced that those glitches are related to varnish and not the > customers IE7, but if you have some RPM's right around the corner, then I > guess we wil wait for that :) > > Wil you be building for rhel4 as well? RHEL4/x86_64 is my main target. I may build i386, ppc and ppc64 too, all for RHEL4. I don't know if I ever will get the package into Fedora. I haven't even started the process. It should be quite easy to make a SuSE package too, though I probably won't have the time (unless some customer should want to pay for it). Ingvar -- When everything else fail: Symlink ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Missing files in the varnish 1.0.3 release
* Matthias Saou > It seems that 1.0.3 has finally been made with "make dist", > unfortunately, both "etc" and "redhat" directories are missing from the > EXTRA_DIST line of Makefile.am, thus weren't included in the released > tarball. > > Was this intentional? The "etc" directory contained the example > vcl.conf which was quite useful to have. I can't speak for the main release, but I'll roll RPMS of 1.0.3 "real soon now". The files should be available from svn anyhow. By the way, does this mean that varnish is going into the rpmforge repos? Ingvar -- Når alt annet feiler: Symlink ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: SV: [RFC] Changes to pass mode
* Poul-Henning > Does anybody even read this mail-list anymore ? :-) *hand* Ingvar -- Når alt annet feiler: Symlink ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
