Re: Breaking Varnish
On Jan 28, 2009, at 1:54 AM, Poul-Henning Kamp wrote: > In message <20090123222947.gb28...@digdug.corp.631h.metaweb.com>, > Niall O'Higgi > ns writes: Can I get you to take -trunk for a spin ? At least the second of the problems you pasted I'm pretty sure I have nailed recently and the first one could easily be the same one in a different disguise. >> >> I've re-run the load test against varnish-trunk. Trunk is better >> behaved, but I now get output like this over and over: >> >> child (19731) Started >> Child (19731) said Closed fds: 4 7 8 10 11 >> Child (19731) said Child starts >> Child (19731) said managed to mmap 49929912320 bytes of 49929912320 >> Child (19731) said Ready >> Child (19731) not responding to ping, killing it. > > This is a typical indication of raw overload, what levels of traffic > are you hitting it with ? Pretty heavy. We put together a test workload that saturated Squid at around 1500 req/s on a dual-core dev systems. The symptoms above appeared somewhere above 6000 req/s on the same hardware and workload. The test has two goals: 1) To try to find bugs in Varnish that might prevent us from switching to Varnish from Squid. 2) To understand how Varnish behaves when it becomes saturated. When testing Squid in this fashion, we found no bugs. Under heavy load, Squid did become very slow but recovered cleanly and went back into normal operation as soon as the load was removed. Varnish didn't fare quite so well. We did find bugs, as you know. Fortunately, those seem to be fixed in trunk. (When do you expect the next point release?) It also appears that Varnish eventually exits completely if placed under high load. I'm okay with that as long as it's intentional behavior; we have a standard nanny that we use in production to restart crashed services anyway. Of course, I understand that killing the child and starting a new one will also lose the cache, which is obviously not particularly desirable under heavy load. Cheers, Tim ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: What is Age header in varnish
> Abhishek Singh writes: >> Can anyone please explain that why Age header is changing on every >> refresh >> while i set "set obj.ttl" for 1 week and is the use of Age header. >> Is there >> anything that i have to modify in my varnish config file to cache >> index.php. >> please suggest something. If I understand correctly, the "Age" header is set by Varnish to indicate how long the file has been in the cache. It is supposed to change on each request. Why do you think index.php is not being cached? Have you checked your webserver logs? Tim Kientzle ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Breaking Varnish
Dual-core AMD processor using the x86_64 kernel. Uname shows:
Linux 2.6.21.5 #9 SMP Thu Aug 16 17:21:29 UTC 2007 x86_64 AMD
Opteron(tm) Processor 248 AuthenticAMD GNU/Linux
On Jan 21, 2009, at 2:01 PM, Iliya Sharov wrote:
> amd64 or i386 architecture?
>
> Tim Kientzle пишет:
>> We're evaluating Varnish as a possible replacement for our
>> installed Squid servers. Performance-wise, Varnish is very
>> impressive, and we're pretty pleased with the configuration
>> flexibility.
>>
>> But...
>>
>> Under heavy load, we're seeing a lot of segfaults and
>> assertion failures. I've pasted an excerpt below of
>> two of the issues we've seen using Varnish 2.0.2 on Linux
>> 2.6.21 kernel with the default VCL (using command-line options
>> to set the listen address and the addresses of the two back-end
>> servers).
>>
>> We're going to repeat these tests and see if we can get
>> more detail, possibly including core dumps. What other
>> information would be useful in diagnosing and fixing
>> these issues?
>>
>> Cheers,
>>
>> Tim Kientzle
>>
>> ==
>>
>> 1) Varnish repeatedly died due to SIGSEGV:
>>
>> child (2816) Started
>> Child (2816) said Closed fds: 4 7 8 10 11
>> Child (2816) said Child starts
>> Child (2816) said managed to mmap 49392648192 bytes of 49392648192
>> Child (2816) said Ready
>> Child (2816) died signal=11
>> Child cleanup complete
>>
>> 2) Varnish repeatedly died due to SIGABRT:
>>
>> child (3017) Started
>> Child (3017) said Closed fds: 4 7 8 10 11
>> Child (3017) said Child starts
>> Child (3017) said managed to mmap 49392648192 bytes of 49392648192
>> Child (3017) said Ready
>> Child (3017) died signal=6
>> Child (3017) Panic message: Assert error in cnt_lookup(),
>> cache_center.c line 625:
>> Condition(sp->objhead != NULL) not true. thread = (cache-worker)sp
>> = 0x2afee0fb3008 {
>> fd = -1, id = 15, xid = 0,
>> client = 10.2.8.27:45430,
>> step = STP_DONE,
>> handling = DELIVER,
>> ws = 0x2afee0fb3078 {
>> id = "sess",
>> {s,f,r,e} = {0x2afee0fb37b0,,+587,(nil),+8192},
>> },
>> },
>> ___
>> varnish-misc mailing list
>> varnish-misc@projects.linpro.no
>> http://projects.linpro.no/mailman/listinfo/varnish-misc
>>
>
> ___
> varnish-misc mailing list
> varnish-misc@projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Breaking Varnish
We're evaluating Varnish as a possible replacement for our
installed Squid servers. Performance-wise, Varnish is very
impressive, and we're pretty pleased with the configuration
flexibility.
But...
Under heavy load, we're seeing a lot of segfaults and
assertion failures. I've pasted an excerpt below of
two of the issues we've seen using Varnish 2.0.2 on Linux
2.6.21 kernel with the default VCL (using command-line options
to set the listen address and the addresses of the two back-end
servers).
We're going to repeat these tests and see if we can get
more detail, possibly including core dumps. What other
information would be useful in diagnosing and fixing
these issues?
Cheers,
Tim Kientzle
==
1) Varnish repeatedly died due to SIGSEGV:
child (2816) Started
Child (2816) said Closed fds: 4 7 8 10 11
Child (2816) said Child starts
Child (2816) said managed to mmap 49392648192 bytes of 49392648192
Child (2816) said Ready
Child (2816) died signal=11
Child cleanup complete
2) Varnish repeatedly died due to SIGABRT:
child (3017) Started
Child (3017) said Closed fds: 4 7 8 10 11
Child (3017) said Child starts
Child (3017) said managed to mmap 49392648192 bytes of 49392648192
Child (3017) said Ready
Child (3017) died signal=6
Child (3017) Panic message: Assert error in cnt_lookup(),
cache_center.c line 625:
Condition(sp->objhead != NULL) not true. thread = (cache-worker)sp
= 0x2afee0fb3008 {
fd = -1, id = 15, xid = 0,
client = 10.2.8.27:45430,
step = STP_DONE,
handling = DELIVER,
ws = 0x2afee0fb3078 {
id = "sess",
{s,f,r,e} = {0x2afee0fb37b0,,+587,(nil),+8192},
},
},
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: 2.0.3 planning
This is a very strange comment. If Varnish requires a particular sequence, it should implement its own. If it requires particular statistical properties, it should test for those, not test for a specific sequence. Tim On Jan 8, 2009, at 2:12 AM, Tollef Fog Heen wrote: > > r3367 | phk | 2008-11-10 10:37:21 +0100 (ma., 10 nov. 2008) | 14 lines > > Add a toplevel word which examines the sequence returned by > srandom(1) and stops the test if we do not get the same sequence > as we expect. > > The Open Group does not define which deterministic sequence srandom(1) > should result in, on that it be deterministic, but I have high hopes > in the general sanity and expect that UNIX people across the board > have realized that for portability the same sequence should be > returned on all platforms. > > At the very least FreeBSD and Linux/GLIBC, as seen on > projects.linpro.no, > agree. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: default.vcl insufficient
On Dec 5, 2008, at 3:45 AM, Dag-Erling Smørgrav wrote: > Tim Kientzle <[EMAIL PROTECTED]> writes: >> According to the "Getting Started" guide on the website, I should be >> able to just run varnish with the default VCL, but that doesn't >> actually work. > > Yes, it does. Varnish requires that you specify *either* a VCL script > *or* a backend on the command line. What you did wrong was that you > specified an empty VCL script. The default VCL script is a copy of > the > build-in default script that Varnish uses if you don't specify one, > and > is provided for documentation purposes only; you should never use it. I think you just managed to say both "Yes, it does work to just run varnish with the default VCL" and "you should never use the default VCL script." ;-) In either case, the online documentation could use an update. http://varnish.projects.linpro.no/static/getting-started.html Under "Installing Varnish from source", it claims you should configure, make, make install, then: "# Start Varnish by typing: varnishd -f /etc/varnish/default.vcl" Which clearly doesn't actually work. It's good to have an easy "make sure it works" step here. I would suggest providing a separate "varnish.vcl" file specifically for people to customize that by default is just enough to get varnish running, installing this along side default.vcl, and editing the documentation here to refer to "varnish.vcl". Your comment above should also be added to the varnishd.1 man page for the -b option to make clear that it's required if there is no backend defined in the specified VCL file. Tim ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: default.vcl insufficient
Varnish's startup error messaging seems slightly broken.
It took me a while to track down why varnish wasn't starting
for me, since it just silently exited with nothing to the
console and nothing to the varnishlog.
By trial and error, I figured out that I needed an explicit -a
option (why can't this be set in the VCL?). I then found this
in the source code:
In mgt_child.c, around line 245:
if (open_sockets() != 0) {
...
REPORT0(LOG_ERR,
"Child start failed: could not open sockets");
return;
}
And REPORT0 calls both fprintf(stderr) and syslog().
So now I know to look in syslog. But I'm not sure why
the fprintf(stderr) went nowhere. This seems to be before
the child is forked and stdio redirected.
Tim
On Dec 3, 2008, at 3:51 PM, Tim Kientzle wrote:
> According to the "Getting Started" guide on the website, I should be
> able to just run varnish with the default VCL, but that doesn't
> actually work. I get this error:
>
> "No backends or directors found in VCL program, at least one is
> necessary.
> VCL compilation failed"
>
> I recommend that the "backend default" definition in default.vcl be
> uncommented in order to make it easier for people to get started.
>
> Tim
>
>
> ___
> varnish-misc mailing list
> varnish-misc@projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
default.vcl insufficient
According to the "Getting Started" guide on the website, I should be able to just run varnish with the default VCL, but that doesn't actually work. I get this error: "No backends or directors found in VCL program, at least one is necessary. VCL compilation failed" I recommend that the "backend default" definition in default.vcl be uncommented in order to make it easier for people to get started. Tim ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Logged-in users
Another approach is to simply use a small bit of Javascript. It's
easy to test for the existence of the cookie in Javascript and
set that text conditionally.
Then you have only one copy of the page to be cached.
The problem with the approach you've outlined here is
that other downstream caches won't understand the difference
(although most will simply refuse to cache any responses
if the request had a cookie header). Whereas the Javascript
approach also allows downstream caches to cache everything
efficiently.
Tim
On Nov 26, 2008, at 12:31 PM, Miles wrote:
> Miles wrote:
>> Hi,
>>
>> I have a site where users can log in. This sets a cookie with their
>> encrypted login details, so they can be authenticated. There are a
>> small number of pages which are user-specific ("change your details"
>> forms, etc), and these are set not to cache.
>>
>> When a user is logged in, a message is shown at the top of the page
>> "You
>> are now logged in". However, nothing on the page depends on the
>> individual user.
>>
>> My question is, how can I organise the cache to have the most cache
>> hits, given that there are effectively two versions of each page -
>> one
>> for logged in users, and one for anonymous users. I want to
>> specifically avoid each user having their own version of the page
>> stored
>> in the cache.
>>
>> Thanks in advance for any wisdom anyone can share!
>>
>> Miles
>
> Thanks to everyone who suggested using ESI - I may have to use this,
> but
> would quite like to avoid it, as it's useful to be able to run the app
> without varnish in front for development/testing.
>
> I wondered whether it was possible to use vcl_hash for my purposes, as
> follows:
>
> sub vcl_hash {
>
>//hash the object with url+host
>set req.hash += req.url;
>set req.hash += req.http.host;
>
># see if the user has a cookie to indicate they are logged in
>if req.http.cookie ~ '__ac=':
> set req.hash += 'authenticated';
>else:
> set req.hash += 'anonymous'
>hash;
>
> }
>
> Would this give me the two representations that I require for each
> page
> - or am I going down a route that will turn out bad?! I couldn't find
> much information about vcl_hash, so I'm not sure if I'm barking up the
> wrong tree or not...
>
> Regards,
>
> Miles
>
> ___
> varnish-misc mailing list
> varnish-misc@projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Inspect Request bodies?
On Nov 5, 2008, at 11:37 AM, Poul-Henning Kamp wrote: In message <[EMAIL PROTECTED]>, Tim Kientzle wri tes: * I'll need code to actually read and store the POST body in memory (including updates to the PASS handler and other places to use the in-memory data when it's available) We sort of have this as point 15 on our shoppinglist: (http://varnish.projects.linpro.no/wiki/PostTwoShoppingList) The first part looks trickier. Has anyone here tried anything similar? Any pointers (particular source files I should pay attention to or memory-management issues I should keep in mind)? It's pretty straightforward really: allocate an (non-hashed) object, add storage to it and store the contents there. You can see pretty much all the code you need in cache_fetch.c and for it to go into the tree as a patch, I would insist that the code gets generalized so we use the same code in both directions, rather than have two copies. The attached patch is a first step in that direction. It generalizes the existing fetch_straight and fetch_chunked so that the same code is used in both directions. This is mostly just code refactoring, although it does add support for chunked upload. "make check" still succeeds after this change; I haven't tried to add any new tests for this yet. My biggest concern is that I might have changed some of the return values here. I'm not yet clear on what return conventions varnish is using internally. varnish2.0.1-generalize-fetch.patch Description: Binary data ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Getting started...
Ah... It seems to work if I omit the -d option. Tim On Nov 12, 2008, at 11:11 AM, Tim Kientzle wrote: > I'm trying to just run a plain-vanilla varnish so I can see it running > before I start mucking with configuration. > > But I'm not having much luck: > > $ uname -a > Darwin tbkk.local 9.5.0 Darwin Kernel Version 9.5.0: Wed Sep 3 > 11:29:43 PDT 2008; root:xnu-1228.7.58~1/RELEASE_I386 i386 > $ sbin/varnishd -a 127.0.0.1:3128 -b 127.0.0.1:80 -d > storage_file: filename: ./varnish.2wA0fp (unlinked) size 669 MB. > Using old SHMFILE > Debugging mode, enter "start" to start child > $ echo $? > 2 > $ > > So, varnishd simply exits with no explanation at all. > > After the above, bin/varnishlog just hangs with no output. > > I finally resorted to running varnishd under GDB, which shows that > vev_schedule_one() is getting NULL from binheap_root(), which leads it > to return zero, which causes vev_schedule() to return, which causes > mgt_schedule() to log "manager dies" and exit(2). > > What have I missed? > > Are there any good examples just showing how to run varnish? > > Tim > > ___ > varnish-misc mailing list > varnish-misc@projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Getting started...
I'm trying to just run a plain-vanilla varnish so I can see it running before I start mucking with configuration. But I'm not having much luck: $ uname -a Darwin tbkk.local 9.5.0 Darwin Kernel Version 9.5.0: Wed Sep 3 11:29:43 PDT 2008; root:xnu-1228.7.58~1/RELEASE_I386 i386 $ sbin/varnishd -a 127.0.0.1:3128 -b 127.0.0.1:80 -d storage_file: filename: ./varnish.2wA0fp (unlinked) size 669 MB. Using old SHMFILE Debugging mode, enter "start" to start child $ echo $? 2 $ So, varnishd simply exits with no explanation at all. After the above, bin/varnishlog just hangs with no output. I finally resorted to running varnishd under GDB, which shows that vev_schedule_one() is getting NULL from binheap_root(), which leads it to return zero, which causes vev_schedule() to return, which causes mgt_schedule() to log "manager dies" and exit(2). What have I missed? Are there any good examples just showing how to run varnish? Tim ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Inspect Request bodies?
Thanks, Poul-Henning! These are exactly the hints I needed. Agree completely about it being controllable in VCL; my own environment has a mix of requests of widely-varying sizes and I certainly don't want this for large uploads. Tim On Nov 5, 2008, at 11:37 AM, Poul-Henning Kamp wrote: > In message <[EMAIL PROTECTED]>, Tim > Kientzle wri > tes: > >> * I'll need code to actually read and store the POST body in memory >> (including updates to the PASS handler and other places to >> use the in-memory data when it's available) > > We sort of have this as point 15 on our shoppinglist: > > (http://varnish.projects.linpro.no/wiki/PostTwoShoppingList) > > The crucial point here, is that we want it to be controllable in > VCL, so that people can disable it for GB sized uploads and enable > it for short stuff (or vice versa) if they want. > >> * I'll need to add VCL functions to actually analyze the POST body. > > To be honest, I would would probably just use the inline C facility > and do it there, than trying to generalize it into a VCL extension. > >> The first part looks trickier. Has anyone here tried anything >> similar? Any pointers (particular source files I should pay >> attention >> to or memory-management issues I should keep in mind)? > > It's pretty straightforward really: allocate an (non-hashed) > object, add storage to it and store the contents there. > > You can see pretty much all the code you need in cache_fetch.c and > for it to go into the tree as a patch, I would insist that the > code gets generalized so we use the same code in both directions, > rather than have two copies. > > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > [EMAIL PROTECTED] | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by > incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Inspect Request bodies?
Under certain circumstances, I want to inspect the body of a POST request at the proxy cache. It don't see any hooks for this in the current Varnish 2.0.1, but I've skimmed the source and it looks feasible: * I'll need code to actually read and store the POST body in memory (including updates to the PASS handler and other places to use the in-memory data when it's available) * I'll need to add VCL functions to actually analyze the POST body. The second part looks pretty straightforward. The VCL engine seems quite modular and extensible. Because VCL routines run in per-request threads, it should be feasible to do more time-consuming operations using straightforward sequential code. (I've also looked at extending Squid or Nginx, but breaking down some of these operations into the necessary state machines would be rather tedious.) The first part looks trickier. Has anyone here tried anything similar? Any pointers (particular source files I should pay attention to or memory-management issues I should keep in mind)? Finally, has anyone else encountered similar requirements that might benefit from this? (I.e., if I do get this to work, is it worth cleaning up the code to contribute back?) Of course, if Varnish already provides some of this and I've simply missed it, then that's even better. ;-) Cheers, Tim P.S. For the curious, there are two specific issues I'm exploring: First, I have an API which prefers GET but supports POST if the arguments are too long; I'd like to accurately cache responses to these larger requests. Second, I've been exploring request-signing techniques borrowed from OAuth. Both of these boil down to computing a hash over all query arguments, including those in the POST body. So far, I've been handling these issues at the app server, but I've got a growing suite of applications running in that layer and I'd like to move the redundant code into a common proxy layer, so I've been surveying existing proxy implementations to see which ones are most amenable to this kind of extension. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
