Re: Varnish and sticky sessions
ao...@mosso.com writes: > Hash by client source port number and you'll get a nice even > distribution. Won't work, Opera opens multiple parallel connections to each server (IIRC: 4 by default, configurable up to 16) DES -- Dag-Erling Smørgrav - d...@des.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: [PATCH] Add PCRE support
"Jos I. Boumans" writes: > It indeed adds an external dependency, but I'm not quite understanding > why you may think that's a bad thing. Like I said: "it should be possible to compile [Varnish] with nothing more than a C compiler and headers for a more or less POSIX-compatible C library". Don't assume everyone will use a precompiled Varnish package provided by their distro. In fact, assume the opposite: assume the average user is running an LTS release for which there is no up-to-date Varnish package, and it's two in the morning, and they need it up and running NOW. Download, configure, make install. An experienced Debian / Ubuntu admin knows that he needs to apt-get install build-essential, but will not enjoy seeing the build fail and having to read and understand compiler messages and (hopefully) figure out that he also needs to apt-get install libpcre-dev. It's even worse on the BSDs, where prior to this change you can build and install Varnish on a fresh stock install with no other ports / packages installed. This is why some of the tools won't be built if you don't have ncurses (or to be precise the ncurses headers) installed. DES -- Dag-Erling Smørgrav - d...@des.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: [PATCH] Add PCRE support
Tollef Fog Heen writes: > We no longer use POSIX style regexes, but rather PCRE regexes. This > introduces a hard dependency on libpcre. This is a radical departure from one of the basic design objectives for Varnish, which was that it should be possible to compile it with nothing more than a C compiler and headers for a more or less POSIX-compatible C library. DES -- Dag-Erling Smørgrav - d...@des.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: r4097 - trunk/varnish-cache/lib/libvarnish
Tollef Fog Heen writes: > Dag-Erling Smørgrav writes: > > There is a better solution: the RCS ID of every source file is embedded > > in a string in the object, "just like FreeBSD does it". This is a > > better solution not only because it does not break dependency tracking, > > but because it is more detailed, and covers more than just libvarnish. > Take a look at sys/conf/newvers.sh, particularly line 102 and onwards in > the FreeBSD source tree. What I'm trying to do is something along the > lines of that bit. If you have suggestions on improving how my > implementation works, I'm happy to hear those (or have patches applied). I already committed a patch - two months ago - which implements the solution described above. For the record, I objected to FreeBSD's svnversion hack when phk committed it, for [some of] the same reasons I object to this one. DES -- Dag-Erling Smørgrav - d...@des.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: r4097 - trunk/varnish-cache/lib/libvarnish
"Poul-Henning Kamp" writes: > This is nice, and we should have that, _as well_, for people like > sky@ who muck about with their own branch. > > But it is no substitute for a tree-wide version number, and the > two schemes are in no way conflicting or in each others way. There is no such thing. It's a sham. What Tollef's patch does is fill in a string with a fictitious value which may or may not be related in some way to the actual version of one or more of the files in libvarnish in some sort of RCS which may or may not be the official repo. Git hashes, for instance, have meaning only in the context of the specific git tree from which Varnish was built, and the way git works, every user has their own tree. DES -- Dag-Erling Smørgrav - d...@des.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: r4097 - trunk/varnish-cache/lib/libvarnish
"Poul-Henning Kamp" writes: > "Dag-Erling Smørgrav" writes: > > There is a better solution: the RCS ID of every source file is embedded > > in a string in the object, "just like FreeBSD does it". This is a > > better solution not only because it does not break dependency tracking, > > but because it is more detailed, and covers more than just libvarnish. > Provide a patch ? I committed it two months ago (r4093 on 2009-06-06). DES -- Dag-Erling Smørgrav - d...@des.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: r4097 - trunk/varnish-cache/lib/libvarnish
"Poul-Henning Kamp" writes: > I originally asked to have the SVN version in there, and if not exactly > in those words, then by intention of "just like FreeBSD does it". > > I certainly don't run a GNU platform and I have not had problems with > the code Take a closer look - unless you use GNU make, you'll see that it breaks dependency tracking for libvarnish, which, with the svnversion hack, is always considered dirty. > so I would like it back, until a better solution, which will be very > welcome, is offered. There is a better solution: the RCS ID of every source file is embedded in a string in the object, "just like FreeBSD does it". This is a better solution not only because it does not break dependency tracking, but because it is more detailed, and covers more than just libvarnish. DES -- Dag-Erling Smørgrav - d...@des.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: r4097 - trunk/varnish-cache/lib/libvarnish
Dag-Erling Smørgrav writes: > Tollef Fog Heen writes: > > Dag-Erling Smørgrav writes: > > > I sent you an email a long time ago explaining why this was wrong, and > > > gave you adequate time to respond before I committed. Amongst other > > > things, it breaks the build on non-GNU platforms, > > This was fixed in r4014. > No, it's still broken (dependency tracking doesn't work properly). And I should have added: it provides incomplete information for builds from mixed trees (e.g. if one file has been rolled back to work around a bug). DES -- Dag-Erling Smørgrav - d...@des.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: r4097 - trunk/varnish-cache/lib/libvarnish
Tollef Fog Heen writes: > Dag-Erling Smørgrav writes: > > I sent you an email a long time ago explaining why this was wrong, and > > gave you adequate time to respond before I committed. Amongst other > > things, it breaks the build on non-GNU platforms, > This was fixed in r4014. No, it's still broken (dependency tracking doesn't work properly). DES -- Dag-Erling Smørgrav - d...@des.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: r4097 - trunk/varnish-cache/lib/libvarnish
Tollef Fog Heen writes: > which is not particularly useful. Previously, it would report the svn > revision number as well. Can we revert 4097 and 4096, or would you > rather implement it in another way? I sent you an email a long time ago explaining why this was wrong, and gave you adequate time to respond before I committed. Amongst other things, it breaks the build on non-GNU platforms, and it doesn't work with releases, source tarballs, or trees checked out with 'svn export' instead of 'svn co'. DES -- Dag-Erling Smørgrav - d...@des.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: munin plugin
Norman Leutner writes: > Now I stuck cause the munin-node doesn’t deliver data… > > Trying manually using munin-run I’m getting the expected results. > > # munin-run varnish_hit_rate > client_req.value 30714 > cache_miss.value 9349 > cache_hitpass.value 97 > cache_hit.value 20207 > > Using the munin telnet interface I get no data… > > # telnet localhost 4949 > Trying 127.0.0.1... > Connected to localhost.localdomain. > Escape character is '^]'. > # munin node at .com > fetch varnish_hit_rate > . > > Any hints ? Adding new plugins to Munin is a nightmare - you need to have all the correct symlinks in all the correct places, then sacrifice a black goat under the full moon. Check the Munin docs and / or ask the Munin devs. DES -- Dag-Erling Smørgrav - d...@des.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
projects.linpro.no accounts
People with projects.linpro.no accounts are strongly encouraged to either check their email there, or install a .forward file to have it forwarded somewhere so it's actually possible for someone who disagrees with a commit to reach you. DES -- Dag-Erling Smørgrav - d...@des.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: r4096 r4097 removal of svn_version in the version string
Artur Bergman writes: > What is the rationale for removing the revision number from the > versoin string in varnishd? It didn't work, and the way it was implemented broke dependency tracking on non-Linux platforms. However, each library and executable now contains version strings for all its source files (but unfortunately not for headers). To pick a couple of short examples: % ident /opt/varnish/lib/libvarnishapi.so /opt/varnish/lib/libvarnishapi.so: $FreeBSD: head/lib/csu/amd64/crti.S 127252 2004-03-21 01:39:01Z peter $ $Id: base64.c 4093 2009-06-06 15:56:17Z des $ $Id: instance.c 4093 2009-06-06 15:56:17Z des $ $Id: shmlog.c 4093 2009-06-06 15:56:17Z des $ $FreeBSD: head/lib/csu/amd64/crtn.S 127252 2004-03-21 01:39:01Z peter $ % ident /opt/varnish/bin/varnishstat /opt/varnish/bin/varnishstat: $FreeBSD: head/lib/csu/amd64/crti.S 127252 2004-03-21 01:39:01Z peter $ $Id: varnishstat.c 4093 2009-06-06 15:56:17Z des $ $FreeBSD: head/lib/csu/amd64/crtn.S 127252 2004-03-21 01:39:01Z peter $ $FreeBSD: head/lib/csu/common/crtbrand.c 174251 2007-12-04 12:18:43Z kib $ $FreeBSD: head/lib/csu/amd64/crt1.c 151072 2005-10-07 22:13:17Z bde $ DES -- Dag-Erling Smørgrav - d...@des.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Bug? Barage of hits leads to failure creating worker threads / stats tracking
Ray Barnes writes: > Rafael Umann writes: > > 32 bits restrict you to use more than ~2.5gb of ram. > I'm sure you know about PAE kernels, so I'm assuming there is some other > artificial limit at 2.5GB, like SHM space maybe? PAE is irrelevant. The address space of each process is still limited to anywhere between 2 to 3 GB depending on the OS. DES -- Dag-Erling Smørgrav - d...@des.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: ./configure error for sys/mount.h
Olivier Nicole writes: > configure: WARNING: sys/mount.h: present but cannot be compiled Fixed, thanks. DES -- Dag-Erling Smørgrav - d...@des.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Debugging / nuked objects spike
Cloude Porteus writes: > Any ideas what would cause Varnish to nuke ~100k objects all at once? Just a guess: these objects are your hot set, so they're all loaded within the first seconds of operation, and they all have the same expiry time, so they all expire within a few seconds of each other. DES -- Dag-Erling Smørgrav - d...@des.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: sendfilev vs writev
Nils Goroll writes: > Which wouldn't surprise me as (at least on Solaris) sendfilev and > writev both do scatter-gather writes when used on memory buffers. My > understanding is that sendfilev would only be advantageous if Varnish > actually read from file descriptors rather than from cache buffers. It does. That's the whole point. But empirical testing shows that sendfile() does not improve performance, most likely because of the overhead of setting up the VM structures for the transfer. DES -- Dag-Erling Smørgrav - d...@des.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Any known issues with Solaris event ports?
Theo Schlossnagle writes: > Also, as a note, the configure.in with 2.0.3 managed to disable > sendfile (which works on Solaris). 1) Are you absolutely sure that Solaris's sendfile() works, i.e. that it does not return until all the data has been transmitted? 2) I disabled the sendfile() test for Solaris in r3338 (between 2.0.1 and 2.0.2), but even before that, it didn't work, because the #ifdefs in *your* code were incorrect. 3) Experience has shown that even when it works, sendfile() does not provide any significant performance improvement unless your working set consists mostly of large (multi-megabyte) objects. DES -- Dag-Erling Smørgrav - d...@des.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: cache_acceptor_poll.c oddities (Solaris)
"Jyri J. Virkki" <[EMAIL PROTECTED]> writes: > The other immediate problem I saw I've filed as ticket 222[1] and > included the patch there (related to IOV_MAX limit on Solaris, which I > see from the archives was discussed earlier and the define was changed, > but didn't quite work). Thanks DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Caching issue
Dag-Erling Smørgrav <[EMAIL PROTECTED]> writes: > "Poul-Henning Kamp" <[EMAIL PROTECTED]> writes: > > Is this FAQ fodder ? > Yes, I'll stick it in the wiki. I added a FAQ entry which links to a separate page on the subject: http://varnish.projects.linpro.no/wiki/FAQ/Compression DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: SSL support
Audun Ytterdal <[EMAIL PROTECTED]> writes: > Dag-Erling Smørgrav wrote: > > That was a wishlist item, not a promised feature. However, Varnish 2.0 > > supports Vary, so if can enable compression on your backend, it will > > "just work" - but DO NOT UNDER ANY CIRCUMSTANCES use the sample > > configuration at the top of Apache's mod_deflate documentation, as it > > will effectively make every compressable document uncacheable. > > You mean > > "AddOutputFilterByType DEFLATE text/html text/plain text/xml" > > Why will that make documents uncacheable? No, that part is fine. I was referring to the "Compress everything except images" example right below it. > Would be sad for non-gzipable browser, but that would be fixed by > > "Header append Vary User-Agent" Absolutely not. This will force Varnish to cache a separate copy of the document for every single User-Agent string it encounters. Browsers that do not support gzip or deflate do not need special treatment: they do not send Accept-Encoding: and will therefore get an uncompressed version of the page. The only browsers that advertise gzip / deflate support but do not actually support it (and therefore need the BrowserMatch rules) are early versions of Netscape 4. According to thecounter.com's December 2007 statistics, Netscape 4's market share is 0.00067 - less than one in one thousand. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: cache_acceptor_poll.c oddities (Solaris)
"Jyri J. Virkki" <[EMAIL PROTECTED]> writes: > Starting varnish (current code from svn) on Solaris, without any > requests I see it sits there with one thread using up all of 1 cpu, > stuck in a poll loop (per truss). The poll code has seen very little maintenance or testing lately, simply because there are better alternatives on all the platforms we officially support. Theo Schlossnagle has an acceptor implementation that uses Solaris event ports; you may have more luck with that than with poll. In any case, thank you for the patch; I will commit it as soon as possible. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Caching issue
"Poul-Henning Kamp" <[EMAIL PROTECTED]> writes: > Is this FAQ fodder ? Yes, I'll stick it in the wiki. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Caching issue
"Poul-Henning Kamp" <[EMAIL PROTECTED]> writes:
> You could try the following in vcl_recv:
>
> if (req.http.accept-encoding) {
> set req.http.accept-encoding = regsub(
> req.http.accept-encoding,
> "gzip, *", "gzip,");
> }
>
> to get rid of the space(s), but it is not guaranteed to get all cases.
>
> Alternatively, the more brutal:
>
> if (req.http.accept-encoding ~ "gzip") {
> set req.http.accept-encoding = "gzip";
> } else {
> unset req.http.accept-encoding;
> }
>
> Will get the desired effect in all cases, provided your backend does
> not attempt deflate as fallback.
A slightly more complex solution, to cover all bases without losing
functionality:
set req.http.accept-encoding = regsub(req.http.accept-encoding,
"^ *gzip, *deflate *$", "gzip,deflate");
set req.http.accept-encoding = regsub(req.http.accept-encoding,
"^ *deflate, *gzip *$", "gzip,deflate");
This should take care of >99% of cases; I don't know of any browser that
supports only one of the two, or supports anything *but* those two.
However, Apache only supports gzip, so Poul-Henning's solution is
sufficient in this case.
DES
--
Dag-Erling Smørgrav
Senior Software Developer
Linpro AS - www.linpro.no
___
varnish-dev mailing list
varnish-dev@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Caching issue
Shain Miley <[EMAIL PROTECTED]> writes: > Here is the output from an original request...nothing is cached at > this point..I hope it has the backend info you need. Well, I don't need anything since I've already found the problem :) But yes, the backend information is there (lines that have 'b' instead of 'c' in the third column) DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Caching issue
"Shain Miley" <[EMAIL PROTECTED]> writes: > The output from below is a result of 'varnishlog > log.file'. > According to the man page if I don't use '-b' or '-c' both are > assumed. Best practice for Varnish logging is to store the raw log data in a file which can later be read back by varnishlog and varnishncsa. > I don't know why any of the logging would be missing? Am I missing > somthing? The backend traffic is missing; I can't tell why since I don't know precisely what you did. > In terms of the 'different Accept-Encoding header ' being set by the > other browser..do you mean the 'Vary' header? No. The client sends Accept-*, the server sends Vary. The Vary header tells Varnish which of the Accept-* headers are relevant. > The reason I ask is because both set Accept-Encoding to 'gzip:deflate' > so they are the same there. No, they aren't. Look closer. > If it is the result of 'Vary' can I simply remove that header and > achive what I am looking for? Not unless you also strip any incoming Accept* headers. > I would also like to know if anyone knows how I can better debug the > hash that is being created...ie...I would like you know exaclty what > information is being used in it's creation, so I can get to a plcae > really where only the url is being used as the hash... The way your vcl_hash is set up only the URL is being used in the hash. Vary support cannot be implemented correctly by just including it in the hash string, so it isn't. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Caching issue
cl_recv {
>
> # Remove the "Cookie:" header from the request.
> remove req.http.Set-Cookie;
> remove req.http.Cache-Control;
>
>
> if (req.request == "GET" && req.url ~
> "\.(html|php|gif|jpg|swf|css|js|png|jpg|jpeg|gif|png|tiff|tif|svg|swf|ico|css|js|vsd|doc)$")
> {
> lookup;
> }
>
> if (req.http.Cache-Control ~ "no-cache") {
>lookup;
> }
> if (req.http.Cache-Control == "max-age=0") {
>lookup;
> }
This is useless, because a) you've already removed that header and b)
Varnish doesn't obey it anyway.
> # Do a "lookup" in the cache. This goes to "vcl_hit", or to
> # "vcl_miss" and then "vcl_fetch"
> lookup;
If you're going to do a lookup anyway, why bother with all of the above?
This will break badly with POST, btw.
> }
>
> # Do the PURGE thing
> sub vcl_hit {
> }
> sub vcl_miss {
> }
>
> sub vcl_fetch {
> if (obj.ttl < 7200s) {
> set obj.ttl = 7200s;
> }
> insert;
>
> if (obj.http.Pragma ~ "no-cache" || obj.http.Cache-Control ~ "no-cache" ||
> obj.http.Cache-Control ~ "max-age=0") {
> insert;
> }
This is useless because a) the "insert" above terminates vcl_fetch and
b) Varnish does not obey Pragma or Cache-Control anyway.
>
> if (obj.ttl < 3600s) {
> set obj.ttl = 3600s;
> }
> insert;
Redundant.
> }
>
> sub vcl_hash {
>
> set req.hash += req.url;
> hash;
> }
Why not host? Why define vcl_hash at all? The default should suffice.
DES
--
Dag-Erling Smørgrav
Senior Software Developer
Linpro AS - www.linpro.no
___
varnish-dev mailing list
varnish-dev@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Caching issue
Shain Miley <[EMAIL PROTECTED]> writes: > Thanks for the info...here is the output from my request (ip's > changed)..in case this can shed some light on things: This is useless if you don't also show us a request that worked. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Caching issue
Adrian Otto <[EMAIL PROTECTED]> writes: > I can't say for certain without looking at it myself, but I wonder if > when you request the URL from the second machine using firefox you > are actually doing a "refresh" that's setting a Cache-Control header > that's forcing varnish to consult the origin server. For the millionth time, Varnish *intentionally* *does* *not* *obey* these headers. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Caching issue
Shain Miley <[EMAIL PROTECTED]> writes:
> What I cannot do...is request a url using GET on one machine..then stop
> the webserver and request that same url from firefox (different machine)
> and get a valid page...I get a 503 error...
Most likely because Firefox has a cookie that triggers the following
code:
> # We only care about the "__ac.*" cookies, used for authentication
> if (req.http.Cookie && req.http.Cookie ~
> "__ac(|_(name|password|persistent))=") {
> pass;
> }
DES
--
Dag-Erling Smørgrav
Senior Software Developer
Linpro AS - www.linpro.no
___
varnish-dev mailing list
varnish-dev@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: SSL support
Louis Luo <[EMAIL PROTECTED]> writes: > Thanks for your info. But do you mean Varnish 2.0 won't support > compression by itself, but rely on the backend to compress HTML pages? > If so, then can it cache compressed pages? Yes, and yes. > I need to offload the compression work from backend to the proxy Why? A properly configured cache should reduce the backend load to a point where you can afford compression. > so probably cannot take advantage of this Vary feature. I was > wondering how much work there would be to add this compression > feature. If this is not that challenging, our team might do some work > to port lighty's mod_deflate to Varnish, if that won't conflict with > your plan. Compression itself is easy. The hard part is figuring out when and how to do it; ideally, we should compress a document only if and only if we get a request from a client that supports compression, and when we do, we should cache it with the same ttl as the original object. I imagine that some of the mechanisms that were introduced to support Vary can also be used to support native compression. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: SSL support
Louis Luo <[EMAIL PROTECTED]> writes: > BTW, some ticket mentioned that compression would be added to 2.0. > How is it now? I need this feature badly. Do you guys have some > early work that I can try? That was a wishlist item, not a promised feature. However, Varnish 2.0 supports Vary, so if can enable compression on your backend, it will "just work" - but DO NOT UNDER ANY CIRCUMSTANCES use the sample configuration at the top of Apache's mod_deflate documentation, as it will effectively make every compressable document uncacheable. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Load Balancing Algorithms in VCL
Shiraz Kanga <[EMAIL PROTECTED]> writes: > Thanks for your reply. Would you mind explaining why load balancing is > "best done in C" especially since VCL should have the same performance > as C as it goes through the C compiler. I don't see why I have to explain our decision. This is the way things are. If it doesn't work for you, tell us what you're trying to do (not how you're trying to do it) and we'll think about it. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: SSL support
Per Andreas Buer <[EMAIL PROTECTED]> writes: > Dag-Erling Smørgrav <[EMAIL PROTECTED]> writes: > > Actually, it would be very complicated... > Oh. I guess I'm wrong, then. I though it was the simple matter of > linking in some SSL library and let it do its magic. Whats the > complication? We would need to add a translation layer between the code that talks to the client and the code that handles requests. The former is not entirely centralized, so some cleanup would be required. Performance would suffer, possibly even in the non-SSL case. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Load Balancing Algorithms in VCL
Shiraz Kanga <[EMAIL PROTECTED]> writes: > In order to support these VCL will need access to various statistics like > * Number of active requests per server > * Number of requests per second per server > * Avg response time per server > * Bandwidth per server > > Are these stats available in the VCL environment? Not presently, and our current take on backend load balancing is that it is best done in C, probably with pluggable algorithms (or directors as they are called in the code). DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: SSL support
Per Andreas Buer <[EMAIL PROTECTED]> writes: > As far as I know there are no concrete plans to include HTTPS > support. I guess it would be a rather simple task to include HTTPS > support so it might be done on some point. Actually, it would be very complicated... DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: development efforts on the Solaris side.
Theo Schlossnagle <[EMAIL PROTECTED]> writes: > [re: stv = stevedores = stevedores->next;] > > [...] Was worried that the assumption that both assignments were > atomic. One is, then the next is. Perhaps that doesn't matter. It > does seem like a hard to trigger race condition to me because while > the assignment is atomic, the access to stevedores->next is not > guaranteed to be view consistent in that assignment: > > T1: get stevedores->next in R(T1,a) > T2: get stevedores->next in R(T2,a) > T1: set stevedores to R(T1,a) > T2: set stevedores to T(T2,a) > > Now T1 and T2 both "advanced the pointer" but they did the same work > and the stv they have is the same. Is that not a problem? Perhaps > I don't understand the impact of that scenario correctly. There is a race, but it's irrelevant. The point of this code is to spread the load between the stevedores. It doesn't matter if two simultaneours allocations go to the same stevedore as long as *all* allocations don't go to the same stevedore. It will balance out in the long run. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: development efforts on the Solaris side.
Theo Schlossnagle <[EMAIL PROTECTED]> writes: > Solaris guarantees that the address ranges (and file data) > references as the source for data in calls to both sendfile() and > sendfilev() will not be touched after control returns to the caller. Thanks, that means we can enable sendfile() on Solaris. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: race condition in ticket 144 - 1.1 backport fix? [patch]
Daniel Papasian <[EMAIL PROTECTED]> writes: > I see that in 1.2 the backend code is significantly different and more > flexible, and I suspect the bug does not exist there. Are there plans > to backport this backend code to 1.1? Sorry, no... > If not, I have a patch that I believe fixes the issue on the 1.1 > branch by acquiring a lock before the addr structure is used. I suspect that would severely impact performance, but I'll have a look at it. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: how to build the nagios-plugin?
Jodok Batlogg <[EMAIL PROTECTED]> writes: > i'm trying to build the nagios-plugin for varnish but i fail already > at the ./configure step... > > checking for VARNISHAPI... configure: error: Package requirements > (varnishapi) were not met: > > No package 'varnishapi' found Sorry for the late response. If you install a sufficiantly new version of Varnish, it will install the necessary pkg-config file. If you installed Varnish from .rpm or .deb, you need to install the devel package as well (IIRC, varnish-libs-devel for .rpm and libvarnish0-dev for .deb) DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Solaris support
Theo Schlossnagle <[EMAIL PROTECTED]> writes:
> Dag-Erling Smørgrav <[EMAIL PROTECTED]> writes:
> > Theo Schlossnagle <[EMAIL PROTECTED]> writes:
> > > Dag-Erling Smørgrav <[EMAIL PROTECTED]> writes:
> > > > [...] session timeout is broken (commented out, actually) in
> > > > your patch, so broken backends and / or clients will bog you
> > > > down.
> > > Sure. When I said "works well," I meant "as well as on Linux.
> > Uh, no, Linux actually supports SO_{RCV,SND}TIMEO, so Varnish does
> > *not* work as well on Solaris as on Linux, with or without your
> > patch.
> And Solaris supports portfs which is better than epoll.
This is not a valid parallel: event ports and epoll are two ways of
implementing the same functionality, but your patch completely
disables the idle session timeout instead of replacing it with
something that works in Solaris.
Implementing session timeouts portably (without SO_{RCV,SND}TIMEO)
would not be impossible, but it would certainly be hard. Removing
sendfile support entirely might help, by simplifying the code.
> I'd note that the performance from the umem stevedore implementation
> is pretty nice.
What umem stevedore implementation? I've never seen this mentioned
anywhere on or off the lists.
> And that works on FreeBSD and Linux now that libumem is ported
> there. Obviously, every implementation has its advantages and
> disadvantages, but umem stuff is an excellent alternative for the
> malloc based stevedore under similar usage.
The malloc based stevedore is not inteded to be used at all, except
for debugging - and it hasn't been used for even that in ages, so it's
really pretty much just a proof of concept, or a sample implementation
if you will.
DES
--
Dag-Erling Smørgrav
Senior Software Developer
Linpro AS - www.linpro.no
___
varnish-dev mailing list
varnish-dev@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Solaris support
Theo Schlossnagle <[EMAIL PROTECTED]> writes:
> Dag-Erling Smørgrav <[EMAIL PROTECTED]> writes:
> > Theo Schlossnagle <[EMAIL PROTECTED]> writes:
> > > We've been running this for a while on Solaris. Works really well.
> > Only because you haven't noticed the bugs yet... for instance,
> > session timeout is broken (commented out, actually) in your patch, so
> > broken backends and / or clients will bog you down.
> Sure. When I said "works well," I meant "as well as on Linux.
Uh, no, Linux actually supports SO_{RCV,SND}TIMEO, so Varnish does
*not* work as well on Solaris as on Linux, with or without your patch.
> > > sendfile and sendfilev on solaris
> > Probably not a good idea unless sendfile() semantics are significantly
> > better on Solaris than on FreeBSD and Linux.
> It's sendfile, it has all the advantages of sendfile. To support
> them, you have to conform to their APIs. I just added support so it
> could say "oh, look, I know how to use that sendfile..." and then
> actually use it (just as linux and freebsd now). And I think
> sendfilev on Solaris is pretty slick.
So you've missed the numerous threads on sendfile() bugs affecting
Varnish, and the more recent threads on sendfile() in FreeBSD and
Linux being broken by design so that Varnish cannot reliably use it,
and Poul-Henning's commit disabling the sendfile() detection in
configure.ac to stop the whining.
> > > using fcntl() when flock() is unavailable
> > There are issues here as well; the semantics are subtly different from
> > OS to OS. For instance, what happens if separate threads in the same
> > process try to lock the same file? It's even less fun if you take
> > into consideration systems that support both.
> As I see it you only supported flock().
You've got it exactly backwards - Varnish has used fcntl() locks
exclusively for... what... five months now? ever since I determined
that in addition to being more portable, fcntl() tends to be the least
broken on platforms that support both (though not on FreeBSD, where
flock() is slightly better, but I didn't consider it "better enough"
to warrant an #ifdef). I even credited you in the commit log.
DES
--
Dag-Erling Smørgrav
Senior Software Developer
Linpro AS - www.linpro.no
___
varnish-dev mailing list
varnish-dev@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Solaris support
Theo Schlossnagle <[EMAIL PROTECTED]> writes: > We've been running this for a while on Solaris. Works really well. Only because you haven't noticed the bugs yet... for instance, session timeout is broken (commented out, actually) in your patch, so broken backends and / or clients will bog you down. > What we need is the function in cache_acceptor.c: > [...] > I see this as a cleaner mechanism regardless as there is no reason for > the generic cache_acceptor to care about int vca_pipes[2]; -- it's an > implementation detail. How's that sound? That sounds like a good idea. Even better if you can submit an isolated patch :) >sendfile and sendfilev on solaris Probably not a good idea unless sendfile() semantics are significantly better on Solaris than on FreeBSD and Linux. >using fcntl() when flock() is unavailable There are issues here as well; the semantics are subtly different from OS to OS. For instance, what happens if separate threads in the same process try to lock the same file? It's even less fun if you take into consideration systems that support both. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Support Cache-Control no-cache/private as per RFC2616 ?
"BUSTARRET, Jean-francois" <[EMAIL PROTECTED]> writes: > Why not use "HTTP accelerator" ? It is pretty much self-explanatory, > and does not imply anything about caching. That's what I've been trying to use consistently since the beginning (cf. the front page, the SourceForge public profile, the Freshmeat profile, the FreeBSD / Debian / RedHat package descriptions etc.) I still think we should define it, and include a mention of the EAS somewhere. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Support Cache-Control no-cache/private as per RFC2616 ?
"Poul-Henning Kamp" <[EMAIL PROTECTED]> writes: > Dag-Erling Smørgrav <[EMAIL PROTECTED]> writes: > > The closest there is to a formal description of what Varnish is > > and how it should behave is the Edge Architecture Specification, > > which unfortunately is far less impressive than its title. > ESI is indeed not impressive, but it is also aimed at a very narrow > market which Varnish is not (3rd party but non-adverserial caching) I was not referring to ESI. The Edge Architecture Specification (EAS) is the document that defines the term "HTTP Surrogate" and the Surrogate-Control header. It is the closest we get to a formal definition of Varnish's niche in the HTTP ecology, and it isn't much. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Support Cache-Control no-cache/private as per RFC2616 ?
"Manuel Amador (Rudd-O)" <[EMAIL PROTECTED]> writes: > If you build a description/table of how an RFC2616 cache (I > understand that to be an HTTP accelerator, but google's kind of slow > to come up in this slow machine) should act, I promise I will build > at least a flowchart. There is no such thing as an HTTP accelerator in RFC2616. There are only private and shared caches. RFC2616 does not come anywhere near anticipating the need for something like Varnish (or Squid in reverse mode for that matter) or describing how it should work. We already have a table of RFC2616 requirements for shared caches (lifted from the Squid web site) in the wiki, but it is not directly applicable to Varnish. The closest there is to a formal description of what Varnish is and how it should behave is the Edge Architecture Specification, which unfortunately is far less impressive than its title. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Support Cache-Control no-cache/private as per RFC2616 ?
"BUSTARRET, Jean-francois" <[EMAIL PROTECTED]> writes: > Dag-Erling Smørgrav <[EMAIL PROTECTED]> writes: > > BTW, could I suggest that you subscribe to varnish-misc or > > varnish-dev (or both) so I don't have to manually approve > > everything you send to the list? > Sorry for that. I'll never get used to what Exchange does to > emails... This should be ok. No, you are posting from a different address than that with which you subscribed. The easiest fix is probably to move your subscription over to the address you are posting from. Alternatively, you can subscribe with both addresses and disable one of them so you don't get everything twice; you will then be able to post to the lists using either address. Let me know which you prefer, and I'll take care of it for you. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Support Cache-Control no-cache/private as per RFC2616 ?
"BUSTARRET, Jean-francois" <[EMAIL PROTECTED]> writes: > IMHO, this is mainly a documentation problem. Why not : > - remove the term "reverse proxy" from the FAQ and replace it by "HTTP > Accelerator", > - describe exactly what/when varnish caches by default, > - describe how to build a RFC2616 reverse proxy, and bundle a sample vcl with > varnish. > > As Poul-Henning told, someone should contribute/sponsor this. Having > more time than money (and feeling not 100% welcome here), I can > write some documentation if you agree. Documentation is always welcome. The wiki needs a lot of work, and both Poul-Henning and I have far too much to do (both with Varnish and other projects) to be able to spend much time on it. BTW, could I suggest that you subscribe to varnish-misc or varnish-dev (or both) so I don't have to manually approve everything you send to the list? DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Support Cache-Control no-cache/private as per RFC2616 ?
"Manuel Amador (Rudd-O)" <[EMAIL PROTECTED]> writes: > Do you really need to snap at contributors, Dag-Erling? Which contributors? All I see in this discussion are people trying to tell me what to do with my (and my employer's) time. It is one thing to ask questions about something you do not understand, and quite another to summarily decide that what you do not understand is wrong, and must be changed, at somebody else's expense. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Support Cache-Control no-cache/private as per RFC2616 ?
"BUSTARRET, Jean-francois" <[EMAIL PROTECTED]> writes: > Yet http://varnish.projects.linpro.no/wiki/FAQ says "Varnish was > written from the ground up to be a high performance caching reverse > proxy." Varnish is a cache, and should follow HTTP/1.1 RFCs. Excuse me, but who are you to tell us what Varnish is or is not? Do you realize how arrogant that is? That aside, you are trying to fit Varnish into an RFC2616 pigeonhole, but there is no pigeonhole that fits - RFC2616 did not anticipate anything like Varnish. There is a draft W3 specification, the Edge Architecture Specification, which attempts to fill that hole, but it is not widely known, so I'm not sure it would help much to write that Varnish is an HTTP surrogate rather than an HTTP accelerator (I try to avoid the term "reverse proxy"). DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Support Cache-Control no-cache/private as per RFC2616 ?
"Poul-Henning Kamp" <[EMAIL PROTECTED]> writes: > There is a draft floating around which defines a "Surrogate-Control" > along the lines of "Cache-Control" but it seems to have little > backing and even less use. It has just as much backing and use as ESI; in fact, it is a prerequisite for ESI. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Support Cache-Control no-cache/private as per RFC2616 ?
"BUSTARRET, Jean-francois" <[EMAIL PROTECTED]> writes: > varnish currently only uses max-age to check the cacheability of > objects. This can be done with some vcl code, but IMHO vcl is not > the right place. Why not? Actually, our intention is to move as much policy as possible (including the interpretation of the Cache-Control header) into VCL as soon as VCL is able to handle it. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Varnish LXR
"Luke Macpherson" <[EMAIL PROTECTED]> writes: > Does anyone have the varnish source loaded into a publicly available > LXR (Linux Cross Referencer)? Not at the moment. I'm not sure it would be worth the effort, as the code base is fairly small. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: W3C Extented Log Format
"Caleb Anthony" <[EMAIL PROTECTED]> writes: > I have created an account on Trac. My username is "Caleb". Give me > access, and I'll create a ticket about the s-ip s-port issue. There is no need to give individual users access. All authenticated users have TICKET_CREATE access. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
SES_Delete (#162)
Here's a typical backtrace for that issue:
#0 0x000800d1048c in thr_kill () from /lib/libc.so.7
#1 0x000800d9a63b in abort () from /lib/libc.so.7
#2 0x00080066dc7f in lbv_assert (func=Could not find the frame base for
"lbv_assert".) at assert.c:58
#3 0x0041afb1 in SES_Delete (sp=0x2104b1a008) at cache_session.c:338
#4 0x00408588 in vca_kev (kp=0x7f5fb340)
at cache_acceptor_kqueue.c:112
#5 0x004087e4 in vca_kqueue_main (arg=0x0)
at cache_acceptor_kqueue.c:151
#6 0x000800a979a8 in pthread_getprio () from /lib/libthr.so.3
#7 0x in ?? ()
Looking at the code, the assert in question is right at the top of
SES_Delete(), so we look at its caller instead. The relevant code in
vca_kev() is:
109 } else if (kp->flags == EV_EOF) {
110 VTAILQ_REMOVE(&sesshead, sp, list);
111 vca_close_session(sp, "EOF");
112 SES_Delete(sp);
113 return;
114 }
We get to this part of the code when the remote end closes the
connection, triggering an EV_EOF kqueue event.
So what does vca_close_session() do? Not much, really:
void
vca_close_session(struct sess *sp, const char *why)
{
int i;
VSL(SLT_SessionClose, sp->id, "%s", why);
if (sp->fd >= 0) {
i = close(sp->fd);
assert(i == 0 || errno != EBADF); /* XXX EINVAL seen */
}
sp->fd = -1;
}
What it certainly *doesn't* do in any way, shape or form is guarantee
that the session no longer has an object or VCL associated with it.
In other words, this assertion will fail every time the client closes
the connection before the completion of the current request.
Please tell me I've overlooked something...
DES
--
Dag-Erling Smørgrav
Senior Software Developer
Linpro AS - www.linpro.no
___
varnish-dev mailing list
varnish-dev@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Source Code paranoia in Varnish
Poul-Henning Kamp <[EMAIL PROTECTED]> writes: > So what is the correct way to deal with the return value of close(2) > system calls ? > > You assert(3) that it succeeds and hope to never see that coredump. #168 says you're wrong. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: release 1.1.2
Andrew McRae <[EMAIL PROTECTED]> writes: > I was curious if there is an ETA for release 1.1.2 - the web site > says it has been delayed from an original date of 9/24, and > I was wondering if a vague date is in mind for the release. I'm back and will try to have it out tomorrow... DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Varnish statistics
"Cryer,Phil" <[EMAIL PROTECTED]> writes: > Is there a tool, other than varnishstat, that I can use to show hits, > misses, etc of a session? When we demo'd Squid we used the Cache > Manager CGI, and while it was clumsy, we could show results from one IP > and know how it did, caching wise. I'm looking for something in Varnish > I can do that with, so far I've been doing a fresh run, then running > 'varnishstat -w 1000' and then just copying the output from the > terminal. The output of 'varnishstat -1' should be much more manageable. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: plans to daemonize varnishncsa?
Jeff Nichols <[EMAIL PROTECTED]> writes: > To continue my questioning of varnishncsa... are there any plans to > add a '-D' option to varnishncsa (as found in varnishlog)? Not really, no... DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Error
"surendar p" <[EMAIL PROTECTED]> writes: > When i compile the code,i got this error > "Expected positive indentation". That's a Flexelint warning, you shouldn't get it while compiling. A little more information (such as copy&paste from your terminal) wouldn't hurt. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Degraded operation
"Poul-Henning Kamp" <[EMAIL PROTECTED]> writes: > I'm not entirely happy with this proposal for a large number of > reasons, but instead of picking it apart, here are my thoughts > on the subject. > > The first observation is that the entire concept of degraded mode > is a minefield of decisions and trouble once you have multiple > backends. I know that you're allergic to the word "mode", and to a certain extent I both understand and agree. However, you are fighting a strawman, because I didn't use that word. There is nothing modal about what I propose. You're right that we can't keep objects around forever, because it will cause swapping in situations where it otherwise wouldn't occur - but don't throw out the baby with the bath water. The rest of my proposal is still valid: vcl_hit() still needs to be able to make a decision about using the existing object or replacing it, vcl_miss() still needs to be split in two, etc. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Degraded operation
What follows is my design proposal for degraded operation, i.e. the ability to serve expired content when a backend is not responding. - The first step is to stop discarding objects when they expire. Instead of placing the object on death row, the misnamed exp_prefetch() simply marks it as expired. If the expired object is never needed again, it will eventually be discarded by LRU_DiscardLocked() (or not, if the cache never fills up) - As a consequence, Varnish will start getting cache hits for expired objects. Therefore, vcl_hit() must be able to inspect the object's expired flag and decide whether to serve the expired object or fetch a new copy. - In order to make that decision, vcl_hit() must be able to figure out which backend the object came from in the first place and inspect that backend's health metric. This opens up a number of interesting possibilities - things like "if the backend is starting to get bogged down but not quite dead yet, only refresh objects that are more than N seconds past their expiry time". - Now that vcl_miss() is no longer the only path to a backend request, we need to take a closer look at it. It really performs two distinct functions: one of them is to react to a cache miss, the other is to prepare a backend request. We need to separate these two functions, so if vcl_hit() decides to fetch a new copy, control can pass straight to the prepare-a-backend-request bit without passing through the react-to-a-cache-miss bit. I would like to name that last part vcl_fetch(), but that name is already taken by the verify-the-result-of-the-backend-request bit. We should rename vcl_fetch() to something else, though I'm not sure what. - It would be nice to be able to define global variables that can be manipulated in the management interface as well as in VCL; for instance, an admin might want a "no-modify" flag that when set, causes VCL to never refresh objects which are already in cache. On the other hand, this can be done by loading a different VCL config. There are still a few gaps: - If we try to refetch an expired object, not knowing (yet) that the backend is dead, what happens to the old copy of the object? Preferably, it should remain in the cache so we can serve it instead of the non-existent fresh copy. - This does not address prefetching at all - I will have to think about how to implement it once the above is in place. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Varnish crashes, and Cache child dies
Andreas Røsdal <[EMAIL PROTECTED]> writes: >>> > Child said (2, 10351): < 977: >Condition((hp->hd[HTTP_HDR_STATUS].b) != 0) not true. >errno = 0 (Success) >>> This is a known bug, see http://varnish.projects.linpro.no/ticket/128 for an explanation and a patch. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Assert for error handling sucks
Marcus Rueckert <[EMAIL PROTECTED]> writes: > Can this not be handled more gracefully? Sure it can, please send patches :) > now i wonder if 524288 is a good default value? It is an arbitrary minimum. I can't imagine a setup where it would make sense to run with that little cache, though. > what would you suggest for a package default config? I suggest reading the release engineering documentation in the wiki, and subscribing to [EMAIL PROTECTED] as you were invited to do a couple of months ago. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: utma utmz
[moved from -dev to -misc] "Manuel Amador (Rudd-O)" <[EMAIL PROTECTED]> writes: > My site always feeds the utma and utmz (google analytics) cookies to > requests. How can I specify in the varnish configuration file > default.vcl that if *only* those two cookies are present, it should > supply files from cache, but if any other cookies are present, it should > contact the backend server (or use the appropriate caching policy)? There's no easy way to do that with the current code base, but we expect to have the required functionality in place within a couple of weeks. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Problem with varnish and caching
[moved from -dev to -misc] "Manuel Amador (Rudd-O)" <[EMAIL PROTECTED]> writes: > As you might already have noted, I reported a bug on varnish caching > files indiscriminately. This is not a bug, it is a misunderstanding. It appears you expect Varnish to act like an RFC 2616 "shared cache" whereas it is in fact a "surrogate" (see the "Edge Architecture Specification" by Oracle and Akamai, although Varnish does not yet fully implement that specification either) > My page sets a few cookies. That'd be okay and it should produce > dynamic pages, which Varnish is furnishing through my backend. The > thing is, these cookies are sent along requests for CSS and PNG and JPG > and JS files, which causes varnish to contact the backend. I don't want > that to happen (I'm happy with them being cached by Varnish 120 > seconds). > > How can I tell Varnish that requests with a response that includes ETag > (a discriminant for static files) should be forcibly cached? This is basically the same issue as in your previous email, and the answer is the same. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Status of ticket #105?
"Vilhelm K. Vardøy" <[EMAIL PROTECTED]> writes:
> I'm wondering if there is any progress regarding the issue described in
> http://varnish.projects.linpro.no/ticket/105 ?
Sorry, we haven't had time to look into it.
The first stack trace shows an assertion failure in EXP_TTLchange() and
is probably related to url.purge; the second shows an assertion failure
in smf_alloc() and is most likely a result of Varnish running out of
storage space.
The attached patch should prevent the first case from reoccurring. The
second is being worked on, and should be fixed in a couple of weeks.
DES
--
Dag-Erling Smørgrav
Senior Software Developer
Linpro AS - www.linpro.no
Index: bin/varnishd/cache_hash.c
===
--- bin/varnishd/cache_hash.c (revision 1533)
+++ bin/varnishd/cache_hash.c (working copy)
@@ -157,7 +157,8 @@
} else if (BAN_CheckObject(o, h->hd[HTTP_HDR_URL].b)) {
o->ttl = 0;
VSL(SLT_ExpBan, 0, "%u was banned", o->xid);
- EXP_TTLchange(o);
+ if (o->heap_idx != 0)
+EXP_TTLchange(o);
} else if (VRY_Match(sp, o->vary))
break;
o->refcnt--;
___
varnish-dev mailing list
varnish-dev@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: quick LRU question
john allspaw <[EMAIL PROTECTED]> writes: > I wonder: will varnish be able to handle a cache that is constantly > full and overflowing ? We currently use squid for reverse-proxy > caching for a working set of objects that is constantly growing, and > for sure much larger than the caches will ever be. Efficient object > eviction in this case is pretty important. :) We have to focus first of all on getting *some* kind of eviction working, but rest assured that we will not settle for the simplest implementation, at least not in the long run. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: question marks in the url
"Brian Ott" <[EMAIL PROTECTED]> writes: > We have urls: http://.yyy.com/test.jpg?tsp=11301981 > > the tsp gets updated when the picture is updated, its a timestamp. > this way the user doesnt have to refresh their browser to see new > pictures. This is a very strange way of doing things. Far better to drop the timestamp, set a long expiry time on the image and invalidate the URL in Varnish when the picture is updated. > varnish doesnt cache urls with ? in them, it appears. Yes, it does. There must be something else at play - most likely cookies. Since you didn't provide logs, there is no way to tell exactly what is going on. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Request for review: synthetic objects
Dag-Erling Smørgrav <[EMAIL PROTECTED]> writes: > The attached patch adds the beginning of an API for synthetic objects. Poul-Henning replied off-list, the patch has been committed. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Request for review: synthetic objects
The attached patch adds the beginning of an API for synthetic objects.
Brief overview:
- moved http_msg array from cache_response.c to cache_http.c,
introduced http_StatusMessage() lookup function
- introduced http_Put{Protocol,Status,Response} to complement
http_PrintfHeader().
- introduced SYN_ErrorPage() in a new file, cache_synthetic.c.
SYN_ErrorPage() populates the session's current object with the
specified error code and a corresponding HTML error page; it is the
caller's responsibility to ensure that the session has a suitable
object (i.e. one that doesn't already have headers or a body)
- rewrote RES_Error() to simply call SYN_ErrorPage() (with ttl = 0) and
RES_WriteObj().
- rewrote cnt_fetch() to use SYN_ErrorPage() to create a 503 page with
a TTL of 30 seconds when Fetch() fails.
- removed the call to RES_Error() in cache_backend.c; the error
trickles back up to cnt_fetch() anyway.
DES
--
Dag-Erling Smørgrav
Senior Software Developer
Linpro AS - www.linpro.no
Index: bin/varnishd/cache_http.c
===
--- bin/varnishd/cache_http.c (revision 1469)
+++ bin/varnishd/cache_http.c (working copy)
@@ -43,6 +43,10 @@
#include "shmlog.h"
#include "cache.h"
+#ifndef HAVE_STRLCPY
+#include
+#endif
+
#define HTTPH(a, b, c, d, e, f, g) char b[] = "*" a ":";
#include "http_headers.h"
#undef HTTPH
@@ -94,7 +98,69 @@
}
/**/
+/* List of canonical HTTP response code names from RFC2616 */
+static struct http_msg {
+ unsigned nbr;
+ const char *txt;
+} http_msg[] = {
+ { 101, "Switching Protocols" },
+ { 200, "OK" },
+ { 201, "Created" },
+ { 202, "Accepted" },
+ { 203, "Non-Authoritative Information" },
+ { 204, "No Content" },
+ { 205, "Reset Content" },
+ { 206, "Partial Content" },
+ { 300, "Multiple Choices" },
+ { 301, "Moved Permanently" },
+ { 302, "Found" },
+ { 303, "See Other" },
+ { 304, "Not Modified" },
+ { 305, "Use Proxy" },
+ { 306, "(Unused)" },
+ { 307, "Temporary Redirect" },
+ { 400, "Bad Request" },
+ { 401, "Unauthorized" },
+ { 402, "Payment Required" },
+ { 403, "Forbidden" },
+ { 404, "Not Found" },
+ { 405, "Method Not Allowed" },
+ { 406, "Not Acceptable" },
+ { 407, "Proxy Authentication Required" },
+ { 408, "Request Timeout" },
+ { 409, "Conflict" },
+ { 410, "Gone" },
+ { 411, "Length Required" },
+ { 412, "Precondition Failed" },
+ { 413, "Request Entity Too Large" },
+ { 414, "Request-URI Too Long" },
+ { 415, "Unsupported Media Type" },
+ { 416, "Requested Range Not Satisfiable" },
+ { 417, "Expectation Failed" },
+ { 500, "Internal Server Error" },
+ { 501, "Not Implemented" },
+ { 502, "Bad Gateway" },
+ { 503, "Service Unavailable" },
+ { 504, "Gateway Timeout" },
+ { 505, "HTTP Version Not Supported" },
+ { 0, NULL }
+};
+
+const char *
+http_StatusMessage(int status)
+{
+ struct http_msg *mp;
+
+ assert(status >= 100 && status <= 999);
+ for (mp = http_msg; mp->nbr != 0 && mp->nbr <= status; mp++)
+ if (mp->nbr == status)
+ return (mp->txt);
+ return ("Unknown Error");
+}
+
+/**/
+
void
http_Setup(struct http *hp, void *space, unsigned len)
{
@@ -817,16 +883,60 @@
/**/
void
+http_PutProtocol(struct worker *w, int fd, struct http *to, const char *protocol)
+{
+ int l;
+
+ CHECK_OBJ_NOTNULL(to, HTTP_MAGIC);
+ l = strlcpy(to->f, protocol, to->e - to->f);
+ xxxassert(to->f + l < to->e);
+ to->hd[HTTP_HDR_PROTO].b = to->f;
+ to->hd[HTTP_HDR_PROTO].e = to->f + l;
+ to->f += l + 1;
+ WSLH(w, HTTP_T_Protocol, fd, to, HTTP_HDR_PROTO);
+}
+
+void
+http_PutStatus(struct worker *w, int fd, struct http *to, int status)
+{
+ int l;
+
+ CHECK_OBJ_NOTNULL(to, HTTP_MAGIC);
+ assert(status >= 100 && status <= 999);
+ l = snprintf(to->f, to->e - to->f, "%d", status);
+ xxxassert(to->f + l < to->e);
+ to->hd[HTTP_HDR_STATUS].b = to->f;
+ to->hd[HTTP_HDR_STATUS].e = to->f + l;
+ to->f += l + 1;
+ WSLH(w, HTTP_T_Status, fd, to, HTTP_HDR_STATUS);
+}
+
+void
+http_PutResponse(struct worker *w, int fd, struct http *to, const char *response)
+{
+ int l;
+
+ CHECK_OBJ_NOTNULL(to, HTTP_MAGIC);
+ l = strlcpy(to->f, response, to->e - to->f);
+ xxxassert(to->f + l < to
cnt_deliver()
from cnt_deliver():
if (sp->obj->objhead != NULL && sp->obj->pass) {
/* we will no longer need the storage */
HSH_Freestore(sp->obj);
}
HSH_Deref(sp->obj);
sp->obj = NULL;
isn't this redundant? In the PASS case, there are no other references
to the object, and HSH_Deref() will call HSH_Freestore().
DES
--
Dag-Erling Smørgrav
Senior Software Developer
Linpro AS - www.linpro.no
___
varnish-dev mailing list
varnish-dev@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Binding to "any"
Marcus Rueckert <[EMAIL PROTECTED]> writes: > On 2007-05-15 10:50:02 +0200, Dag-Erling Smørgrav wrote: > > OK, so the only practical difference is that IPV6_V6ONLY is the default > > on Linux? Don't know why I never thought of that. > no. ipv6_only is default on *bsd. linux defaults to ipv6_only off. That's what I meant, even if it isn't what I wrote :) DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Binding to "any"
Marcus Rueckert <[EMAIL PROTECTED]> writes:
> On 2007-05-14 16:50:58 +0200, Dag-Erling Smørgrav wrote:
> > On Linux, the current behaviour is good enough - binding to either
> > INADDR_ANY or IN6ADDR_ANY will give you both.
> bind the ipv6 binds with IPV6_ONLY. than you get the same behavior as
> on *BSD. postfix does that e.g. you just need to detect the socket type.
OK, so the only practical difference is that IPV6_V6ONLY is the default
on Linux? Don't know why I never thought of that.
This still means we need to modify the listening socket setup code to
handle multiple sockets resulting from a single specification, for
instance ":80" -> { 0.0.0.0:80, [::]:80 }.
DES
--
Dag-Erling Smørgrav
Senior Software Developer
Linpro AS - www.linpro.no
___
varnish-dev mailing list
varnish-dev@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-dev
Binding to "any"
Currently, there is a known issue with varnishd on FreeBSD, where if you run it with "-a :80" it will only bind to IN6ADDR_ANY instead of both INADDR_ANY and IN6ADDR_ANY. This is not easy to solve. The obvious fix, which is to bind to *all* addresses returned by getaddrinfo() instead of just the first one, will work on FreeBSD, but not on Linux, where you can't bind to the same port on both INADDR_ANY and IN6ADDR_ANY. On Linux, the current behaviour is good enough - binding to either INADDR_ANY or IN6ADDR_ANY will give you both. On FreeBSD, if we want both *and* the kernel has IPv6 support compiled in, we can bind to IN6ADDR_ANY and clear the IPV6_V6ONLY socket option (it defaults to on). If the kernel does *not* have IPv6 support compiled in, getaddrinfo() will still return both INADDR_ANY and IN6ADDR_ANY, but we binding to IN6ADDR_ANY will fail. I don't see any good way out of this mess - we're pretty much forced to special-case the "any" case. I've looked at other network servers, such as Apache and OpenSSH, and they pretty much universally solve (or rather avoid) this issue by explicitly specifying either 0.0.0.0 (INADDR_ANY) or :: (IN6ADDR_ANY) as the default listening address. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: sendfile changes
Andreas Røsdal <[EMAIL PROTECTED]> writes: > Poul-Henning Kamp <[EMAIL PROTECTED]> writes: > > Andreas Røsdal <[EMAIL PROTECTED]> writes: > > > What does "not doing it's job" mean here? Sending corrupt or > > > incomplete data? > > One, the other or even Both, depending on your kernel. > Yes, I've seen both. What is the best workaround for this problem? > Should I install the varnish from SVN trunk? telnet to the management port and type param.set sendfile_threshold -1 and add "-p sendfile_threshold=-1" to your startup flags (VARNISHD_PARAMS in /etc/sysconfig/varnish) DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Varnish crashing in production
Per Andreas Buer <[EMAIL PROTECTED]> writes: > Dag-Erling Smørgrav wrote: > > Within reason - there is no point in using a cache file that is many > > times the size of your data set. I'm not sure how the allocator works > > in detail, but somewhere between 1x and 2x the size of your data set > > should be sufficient. > Is an "out of memory" situation handled properly now? No, but it won't occur if your cache is larger than your data set (and I mean "data set", not "working set" - thanks to the likes of Google, most of your data set will be in your working set at some point or other) DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Varnish crashing in production
Barry Abrahamson <[EMAIL PROTECTED]> writes: > On May 11, 2007, at 12:59 AM, Per Andreas Buer wrote: > > Use should use a much bigger cache. The cache should never fill up - > > so use 36-150GB is you can. > Ok, I will make the cache bigger. Within reason - there is no point in using a cache file that is many times the size of your data set. I'm not sure how the allocator works in detail, but somewhere between 1x and 2x the size of your data set should be sufficient. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Varnish crashing in production
Barry Abrahamson <[EMAIL PROTECTED]> writes: > We are currently testing Varnish as a reverse proxy to serve images. > We are running into problems where the varnishd child (and sometimes > parent), will die unexpectedly. When the child process dies, the > parent restarts it and when the parent process dies, it is restarted > by our monitoring systems. Sometimes this happens every 5 minutes, > sometimes it can stay running up to 2 hours. I don't see any pattern > as to when it is failing. In our development environment, with no > "real" traffic, but the same configuration, we are unable to > replicate the crash. The vcl file we are using is basically the > default one, except in the fetch subroutine, the we only insert > objects if they have a certain header and pass everything else. The > core dumps created as a result of the crash all show this: 1.0.3 has a known bug which will cause the child process to crash if something unexpected happens while retrieving an object from the backend. Please try again with the tip of the 1.0 branch. > Program terminated with signal 6, Aborted. That's an assertion failure. If you run the management process in the foreground (start varnishd manually with -d -d, then type "start" to kick off the child process) you should see an error message when the child dies. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Child crashes when the backend is down
"Poul-Henning Kamp" <[EMAIL PROTECTED]> writes:
> Obviously nobody else will ever check the busy flag in that case,
> so the locking isn't necessary.
HSH_Ref() and HSH_Deref() already check that o->objhead != NULL before
trying to lock the mutex, so I applied the same logic to HSH_Unbusy().
Is this a suitable fix?
DES
--
Dag-Erling Smørgrav
Senior Software Developer
Linpro AS - www.linpro.no
Index: bin/varnishd/cache_hash.c
===
--- bin/varnishd/cache_hash.c (revision 1373)
+++ bin/varnishd/cache_hash.c (working copy)
@@ -185,6 +185,7 @@
void
HSH_Unbusy(struct object *o)
{
+ struct objhead *oh;
struct sess *sp;
CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
@@ -192,9 +193,14 @@
assert(o->refcnt > 0);
if (o->cacheable)
EXP_Insert(o);
- LOCK(&o->objhead->mtx);
+ oh = o->objhead;
+ if (oh != NULL) {
+ CHECK_OBJ(oh, OBJHEAD_MAGIC);
+ LOCK(&oh->mtx);
+ }
o->busy = 0;
- UNLOCK(&o->objhead->mtx);
+ if (oh != NULL)
+ UNLOCK(&oh->mtx);
while (1) {
sp = TAILQ_FIRST(&o->waitinglist);
if (sp == NULL)
___
varnish-dev mailing list
varnish-dev@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Child crashes when the backend is down
"Poul-Henning Kamp" <[EMAIL PROTECTED]> writes: > o->busy is protected by the objhead mutex because otherwise we would have > to grab all the objects mutexes in the hash lookup. > > The busy flag is very seldomly manipulated, and this is much more efficient. Fine, but what is the locking protocol when o has no objhead? DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: vcl and cache-control header
[EMAIL PROTECTED] (Dag-Erling Smørgrav) writes: > OK, I'm not sure we've ever tested going from hit to pass, although > it's meant to work. Could you please open a ticket about this? Call > it "returning pass from vcl_hit crashes the child" or something. Looking at the code, I can definitely say that it doesn't work, although it was clearly intended (and promptly forgotten) at some point. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Child crashes when the backend is down
This is probably what's killing my customer up north:
> % varnishd -a localhost:8080 -b localhost:80 -sfile,/tmp,4m -d -d
[note that there is no server at localhost:80]
> file /tmp/varnish.TIGksM (unlinked) size 4194304 bytes (1024 fs-blocks, 1024
> pages)
> Using old SHMFILE
> rolling(1)...
> rolling(2)...
> start
> CLI
> start child pid 7023
> 200 0
>
> Child said (2, 7023): < managed to mmap 4194304 bytes of 4194304
> Ready
> CLI ready
> >>
[now point firefox at http://localhost:8080/]
> Child said (2, 7023): < >>
> Child not responding to ping
[because it is busy dumping core]
> Cache child died pid=7023 status=0x8b
[0x8b = SIGSEGV]
> Clean child
> Child cleaned
> start child pid 7039
> Child said (2, 7039): < managed to mmap 4194304 bytes of 4194304
> Ready
> CLI ready
> >>
> Manager got SIGINT
> Clean child
> Child stopping
> unlink /tmp/vcl.XX9MHO5n
[let's look at that core file]
> % gdb =varnishd ~/core
> GNU gdb 6.6-debian
> Copyright (C) 2006 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu"...
> Using host libthread_db library "/lib/libthread_db.so.1".
> Error while mapping shared library sections:
> /tmp/vcl.XX9MHO5n: No such file or directory.
> Reading symbols from /opt/varnish/lib/libvarnish.so.0...done.
> Loaded symbols for /opt/varnish/lib/libvarnish.so.0
> Reading symbols from /opt/varnish/lib/libvcl.so.0...done.
> Loaded symbols for /opt/varnish/lib/libvcl.so.0
> Reading symbols from /lib/libdl.so.2...done.
> Loaded symbols for /lib/libdl.so.2
> Reading symbols from /lib/librt.so.1...done.
> Loaded symbols for /lib/librt.so.1
> Reading symbols from /lib/libpthread.so.0...done.
> Loaded symbols for /lib/libpthread.so.0
> Reading symbols from /lib/libc.so.6...done.
> Loaded symbols for /lib/libc.so.6
> Reading symbols from /lib/ld-linux-x86-64.so.2...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from /lib/libnss_files.so.2...done.
> Loaded symbols for /lib/libnss_files.so.2
> Symbol file not found for /tmp/vcl.XX9MHO5n
> Core was generated by `varnishd -a localhost:8080 -b localhost:80
> -sfile,/tmp,4m -d -d'.
> Program terminated with signal 11, Segmentation fault.
> #0 0x2afef513d3d9 in pthread_mutex_lock () from /lib/libpthread.so.0
> (gdb) where
> #0 0x2afef513d3d9 in pthread_mutex_lock () from /lib/libpthread.so.0
> #1 0x0040cfa3 in HSH_Unbusy (o=0x64e840) at cache_hash.c:195
> #2 0x004095fc in cnt_fetch (sp=0x64c308) at cache_center.c:292
> #3 0x0040a428 in CNT_Session (sp=0x64c308) at steps.h:40
> #4 0x00411c2b in wrk_do_one (w=0x43002c50) at cache_pool.c:191
> #5 0x00411fe7 in wrk_thread (priv=0x632340) at cache_pool.c:231
> #6 0x2afef513b2a5 in start_thread () from /lib/libpthread.so.0
> #7 0x2afef541f61d in clone () from /lib/libc.so.6
> #8 0x in ?? ()
> (gdb) up
> #1 0x0040cfa3 in HSH_Unbusy (o=0x64e840) at cache_hash.c:195
> 195 LOCK(&o->objhead->mtx);
> (gdb) l
> 190 CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
> 191 assert(o->busy);
> 192 assert(o->refcnt > 0);
> 193 if (o->cacheable)
> 194 EXP_Insert(o);
> 195 LOCK(&o->objhead->mtx);
> 196 o->busy = 0;
> 197 UNLOCK(&o->objhead->mtx);
> 198 while (1) {
> 199 sp = TAILQ_FIRST(&o->waitinglist);
> (gdb) p *o
> $1 = {magic = 847584578, refcnt = 1, xid = 780407261, objhead = 0x0,
> heap_idx = 0, ban_seq = 0, pass = 0, response = 0, valid = 0, cacheable = 0,
> busy = 1, len = 0, age = 0, entered = 0, ttl = 0, last_modified = 0, http =
> {
> magic = 1680389577, s = 0x0, t = 0x0, v = 0x0, f = 0x0, e = 0x0,
> conds = 0 '\0', logtag = HTTP_Rx, hd = {{b = 0x0,
> e = 0x0} }, hdf = '\0' , nhd = 0},
> list = {tqe_next = 0x0, tqe_prev = 0x0}, deathrow = {tqe_next = 0x0,
> tqe_prev = 0x0}, store = {tqh_first = 0x0, tqh_last = 0x64eb18},
> waitinglist = {tqh_first = 0x0, tqh_last = 0x64eb28}}
There is no o->objhead because no object was ever retrieved and
Fetch() returned early.
Why is o->busy protected by o->objhead->mtx? Why not have a mutex
directly in struct object instead?
DES
--
Dag-Erling Smørgrav
Senior Software Developer
Linpro AS - www.linpro.no
___
varnish-dev mailing list
varnish-dev@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: vcl and cache-control header
Denis Ahrens <[EMAIL PROTECTED]> writes:
> Poul-Henning Kamp <[EMAIL PROTECTED]> writes:
> > Denis Ahrens <[EMAIL PROTECTED]> writes:
> > > I have problems with the following vcl-script:
> > >
> > > if(req.http.Cache-Control == "max-age=0") {
> > > set obj.ttl = 0s;
> > > pass;
> > > }
> > Guys, when you quote VCL code, please also tell us which
> > function you do this in...
> sorry, this is in vcl_hit()
OK, I'm not sure we've ever tested going from hit to pass, although
it's meant to work. Could you please open a ticket about this? Call
it "returning pass from vcl_hit crashes the child" or something.
DES
--
Dag-Erling Smørgrav
Senior Software Developer
Linpro AS - www.linpro.no
___
varnish-dev mailing list
varnish-dev@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Varnish configuration / (s)maxage
"Alan Runyan" <[EMAIL PROTECTED]> writes: > varnish is amazing! I have a lot of feature requests *grin*. But > what is important that there is a large community, Zope/Plone that > could benefit from this technology. The community is already familiar > with complex caching configurations and most large systems use Squid > in production. Moving to varnish is a no-brainer. Especially once > varnish reaches feature parity with squid. Specifically vary support. Thank you for your feedback. I hope Poul-Henning will have time to look into your cache-control issues. As for Zope/Plone integration, we are planning to develop some sort of "best practices" document offering advice for integrating Varnish with various CMS frameworks. If you wish to contribute your experience, contact me off-list and I will set you up with wiki access. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Getting useful information from a varnishd crash
Steven Murdoch <[EMAIL PROTECTED]> writes: > I've been running varnish-1.0.2 for about a month, and it generally > works well, but it has stopped running twice (I presume a crash). > Since it only keeps in-memory logs, I am a bit in the dark about what > went wrong. > > What is the best way I could use to find out what the problem is, so I > can either fix it or submit a useful bug report? First of all, please upgrade to 1.0.3 before reporting any bugs. Second, you can run 'varnishlog -w filename' to capture complete logs to file. See the varnishlog man page for additional details. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Instability when using management CLI
Kristoffer Gleditsch <[EMAIL PROTECTED]> writes: > I'm not sure I understand every aspect of this function, but I'm unable > to see where the space for that extra newline is allocated, and > re-compiling version 1.0.3 with the attached patch seems to have > stabilized it. Good catch, but there are a couple of snags. First of all, I think the final newline is meant to replace the trailing space, i.e. ["a" "b" "c"\n] rather than ["a" "b" "c" \n]. However, the for loop escapes double quotes, newlines and backslashes, so the string may grow well beyond the strlen() + 3 set aside for each argument. I'll send you an updated patch later today. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Regarding varnish pdf and presentation
[EMAIL PROTECTED] writes: > Read the attached PDF > for a presentation and some stats. > Im trying to find this pdf > > The reason I say so is because Varnish uses some "state > of the art" features in the OS (Even if they have been around a long > time :) Like you say, let the OS do the stuff it knows how to do > best.). Right now I use Varnish in a prod. environment with mixed > content static/dynamic, and it works amazing. Read the attached PDF > for a presentation and some stats. I'm not sure what you are trying to tell us. Are you looking for a PDF about Varnish, or are you trying to send us one? Mailman strips most attachments from incoming mail (with good reason), so if you are trying to send us something, you'd be better off posting it on the web and sending us the URL. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
