Re: varnish crashes

2010-01-23 Thread Poul-Henning Kamp
In message 4b5ad8b0.6090...@netmatch.nl, =?ISO-8859-1?Q?Angelo_H=F6ngens?= wr
ites:

By the way: the balancers do a total of 2000 req/sec now, but when
stresstesting I can easily get 9000 cache/hits persec. So I don't think
it's hanging on the upper limits of its performance.

At that level of load, make sure to kldload the http accept filter.


Your varnish-stat looks pretty OK.

Have you configured health-polling of all those backends ?


-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


The usage of varnish, revisited

2010-01-23 Thread Ingvar Hagelund
Dump from my blog posting at http://ingvar.blog.linpro.no/

Ingvar



The usage of varnish, revisited

This is more or less a repost, with updated numbers.

Some months have passed, and it is time to run my poking scripts again, looking 
for sites that run Varnish. There is no deep magic here. I just parse the 
available top lists that I know of, and peek at the HTML headers of the sites 
that are listed. If there are subsites linked from the front page of the site, 
I scan them too. This means that twitter.com shows up, though Twitter only runs 
Varnish on its search site. Subsites with a Varnish match are shown in 
parenthesis in the results.

For the Nordic countries, I have found quite good lists, that is, upload result 
lists from the probably most visited media sites in the respective countries. 
Remember of course, that these are generally pay-to-be-included lists, and 
there may exist sites with far more hits than the ones listed.

For a global overview, I have used Alexa.

Now for the results. Varnish is sponsored by large Norwegian sites, so it is no 
big surprise that there are a lot of hits in Norway. Of the TNS Gallup top 
list, Varnish runs at 36 of the top 100 sites. That’s 3 up since my last probe.

For Denmark, I use FDIM’s list. From May, we are up from 3 to 11 sites in the 
top 100. For Finland, I use TNS’ numbers again. No changes there. For Sweden, I 
use the KIA Index list. I now probe the 200 top sites, so there are several 
more varnish sites on the list. In the top 100, we are up from 8 to 9.

For the Alexa’s World top 500 list, I have tweaked my filters a bit, and the 
list is up from 7 to 17 sites since my last probe in May. World Domination, 
here we come!

The whole World: Alexa's global top 500 list
Place  12 Varnish running on twitter.com 
(http://integratedsearch.twitter.com/search.html)
Place  47 Varnish running on photobucket.com (http://blog.photobucket.com/)
Place  90 Varnish running on mixi.jp 
(http://img.mixi.jp/static/css/basic/logout_quirks) (and others)
Place 101 Varnish running on weather.com (http://www.weather.com/) (weather.com)
Place 107 Varnish running on globo.com (globo.com) (http://www.globo.com/)
Place 111 Varnish running on ifeng.com (http://big5.ifeng.com/gate/big5/www)
Place 138 Varnish running on answers.com (http://www.answers.com/) (answers.com)
Place 170 Varnish running on orange.fr (http://r.orange.fr/r/Eorangepublicite)
Place 179 Varnish running on hulu.com (hulu.com)
Place 199 Varnish running on wikia.com (wikia.com)
Place 213 Varnish running on xinhuanet.com 
(http://big5.xinhuanet.com/gate/big5/www)
Place 290 Varnish running on people.com.cn 
(http://bbs1.people.com.cn/boardList) (and others)
Place 344 Varnish running on tuenti.com 
(http://estaticos1.tuenti.com/layout/web2/images/favicon)
Place 428 Varnish running on chinanews.com.cn 
(http://www.chinanews.com.cn/test/index_back_d.html) (and others)
Place 433 Varnish running on mercadolivre.com.br 
(http://veiculos.mercadolivre.com.br/) (and others)
Place 447 Varnish running on mercadolibre.com.mx 
(http://tendencias.mercadolibre.com.mx/) (and others)
Place 456 Varnish running on break.com (break.com)

In my last probe, I poked sites all over Europe. With a few exceptions, that 
was a bit less interesting. Global .com and .net sites tend to cover most of 
the top 100 entries, as I had only toolbar lists, and it’s not that spectacular 
that for example people in Serbia and Monte Negro are browsing Twitter, like 
the rest of the World does. So I have skipped other countries. If you know of 
any good top list for you country that is not toolbar based, please let me know.

All the gory details are available here: 
http://users.linpro.no/ingvar/varnish/stats-2010-01-18.txt

Other more or less worth mentioned sites that is reported to use Varnish but 
does not show up in my lists, may be Slashdot, WAT TV, The Pirate Bay, 
JDownloader, e.Republik, WOWwiki, Globo.com, PCWelt.de, BlackPlanet, 
funnyordie.com and n-tv.de, to name a few.

___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Varnish extensions for SSO support

2010-01-23 Thread Sam Crawford
Evening all,

I've been an avid Varnish user both personally and at work for a
couple of years now. At work we use it to cache content across our
global intranet, handling a few million requests per day. At present,
we have the following logical setup...

F5 GTM (GSLB device)  F5 load balancer  Varnish  In-house Java
Reverse Proxy  Backend applications (hundreds)

Varnish and the in-house reverse proxy reside on the same servers,
with varnish having a single backend pointing at the in-house reverse
proxy (the F5s handle failover between instances).

The in-house Java reverse proxy performs a range of functions,
including (but certainly not limited to):

* Authenticating/authorising users via our Single Sign On service
* Header injection to help backend applications identify users
* Catching cookies from backend applications and delivering a single
pointer cookie back to clients

I've been wondering if we could write some C extensions to Varnish to
remove the need for the Java reverse proxy. This would help flatten
the infrastructure and save on latency (which is pretty important for
us). The standard Varnish VCL capabilities would meet many of our
requirements, but for some functions we'd certainly need to write
extensions (such as making an out-of-band HTTP request to an SSO
server in order to validate an SSO cookie (which we'd also need to
cache!)).

Whilst I know it's technically feasible for us to do this, I was
wondering (a) if anyone is already doing something similar and (b) if
the community thinks I'm completely mad for evening thinking about
doing it :-)

Thanks,

Sam
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc