Malformed varnishncsa output
I'm getting a lot of lines line this from varnishncsa: 10.151.1.1 - - [25/Nov/2008:19:11:14 +] "GET http:// vectordevhttp://vectordev/devsite/diagrams/tn-rev1.png HTTP/1.1" 200 60834 "-" "curl/7.16.3 (amd64-portbld-freebsd6.3) libcurl/7.16.3 OpenSSL/0.9.7e zlib/1.2.3" Notice the duplicated http://vectordevhttp://vectordev part after GET. Should I just change it to not print out lp->df_Host in varnishncsa.c or something? Thanks, Skye ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
varnishncsa and liblogging
Hello again, I'm currently working on adding liblogging (reliable syslog over BEEP) support to varnishncsa. http://www.liblogging.org/ Is this something that the project would be interested in adding to trunk when it's done? I presume it would need to be wrapped in a configure --with-liblogging option (default off) Thanks, Skye ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: purge.hash in trunk
On 14-Jul-08, at 2:58 PM, Poul-Henning Kamp wrote:
> Because the default vcl_hash says:
>
>sub vcl_hash {
>set req.hash += req.url;
>if (req.http.host) {
>set req.hash += req.http.host;
>} else {
>set req.hash += server.ip;
>}
>hash;
>}
*blush* that would explain it, this whole time I've been trying to
'purge host#path#' when its supposed to be 'purge path#host#'
Thanks,
Skye
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: purge.hash in trunk
Oh I see, right, it matches against the whole concatenated hash string.
Why is it that this purges all files from w0011.example.com
purge.hash w0011.*#.*
But this does nothing?
purge.hash w0011.*#.*#
Flipping the #'s around works:
purge.hash #w0011.*#.*
This also has no effect, is "+" not supported in the regex?
purge.hash w0011.+#.+
purge.hash w0011.+#.+#
purge.hash #w0011.+#.+
But none of these do anything:
purge.hash #w0011.*#.*html
purge.hash w0011.*#.*html
purge.hash w0011.*#.*html#
So I'm somewhat confused again. I haven't messed with vcl_hash at all.
Thanks,
Skye
On 14-Jul-08, at 12:33 PM, Poul-Henning Kamp wrote:
> In message <[EMAIL PROTECTED]>, Skye Poier
> Nott writes
> :
>> Updated to r2945 today, is purge.hash changed or broken?
>>
>> This works:
>>
>> purge.url .
>> purge.hash .
>>
>> But none of these have any effect:
>>
>> purge.hash .#.#
>
> This would only purge single character urls on single character
> host names.
>
> Try:
> purge.hash .*#.*#
>
> for wider matching
>
>> (The last trunk rev I was using had #host#path but now it's
>> host#path# ? It would be nice if 'help purge.hash' mentioned the
>> current expected format)
>
> The expected format is what vcl_hash {} set up, and since the
> user can redefine that function, a constant help message would
> have good chances of being confusing.
>
> --
> Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
> [EMAIL PROTECTED] | TCP/IP since RFC 956
> FreeBSD committer | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by
> incompetence.
> ___
> varnish-misc mailing list
> varnish-misc@projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
purge.hash in trunk
Updated to r2945 today, is purge.hash changed or broken? This works: purge.url . purge.hash . But none of these have any effect: purge.hash .#.# 200 0 purge.hash #.#. 200 0 purge.hash $#$# 200 0 purge.hash #$#$ 200 0 (The last trunk rev I was using had #host#path but now it's host#path# ? It would be nice if 'help purge.hash' mentioned the current expected format) -Skye ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: hash.purge
Answering my own questions again... yes, regex in the host part does seem to work! Cool! However I noticed escaping dots doesn't seem to work as normal: hash.purge #w...6\.example\.com#$ 100 41 Syntax Error: Invalid backslash sequence hash.purge #w...6.example.com#$ 200 27 PURGE #w...6.example.com#$ Bug or feature? Skye On 7-Jul-08, at 5:06 PM, Skye Poier Nott wrote: > Does hash.purge take a regex in the host section as well, or just the > url section? > ie is this valid (delete all jpg's for all *.bar.com hosts) > > hash.purge #.+\.bar\.com#\.jpg$ > > Thanks, > Skye > > ___ > varnish-misc mailing list > varnish-misc@projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
hash.purge
Does hash.purge take a regex in the host section as well, or just the url section? ie is this valid (delete all jpg's for all *.bar.com hosts) hash.purge #.+\.bar\.com#\.jpg$ Thanks, Skye ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Object leak?
I installed Munin to start keeping track of various Varnish stat and in my test configuration which keeps requesting the same ~4,000 small objects over and over, I notice that the Varnish object count keeps climbing steadily, at a rate of about 5,000 more objects every 10 hours. VM alloc stays constant. How can I debug this issue? (using -trunk) Maybe its just a varnishstat bug... Thanks, Skye ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Trunk build errors
Weird, any thoughts? Or am I just asking for trouble with -trunk :)
$ svn update (last night's)
$ make clean && configure && make
mkdir .libs
gcc -DVARNISH_STATE_DIR=\"/usr/local/var/varnish\" -g -O2 -o .libs/
varnishd varnishd-cache_acceptor.o varnishd-cache_acceptor_epoll.o
varnishd-cache_acceptor_poll.o varnishd-cache_acceptor_kqueue.o
varnishd-cache_backend.o varnishd-cache_ban.o varnishd-cache_center.o
varnishd-cache_cli.o varnishd-cache_dir_random.o varnishd-
cache_dir_simple.o varnishd-cache_expire.o varnishd-cache_fetch.o
varnishd-cache_hash.o varnishd-cache_http.o varnishd-cache_httpconn.o
varnishd-cache_main.o varnishd-cache_panic.o varnishd-cache_pool.o
varnishd-cache_pipe.o varnishd-cache_response.o varnishd-
cache_session.o varnishd-cache_synthetic.o varnishd-cache_vary.o
varnishd-cache_vcl.o varnishd-cache_vrt.o varnishd-cache_vrt_acl.o
varnishd-cache_vrt_re.o varnishd-cache_vrt_esi.o varnishd-cache_ws.o
varnishd-hash_simple_list.o varnishd-hash_classic.o varnishd-
instance.o varnishd-mgt_child.o varnishd-mgt_cli.o varnishd-
mgt_event.o varnishd-mgt_param.o varnishd-mgt_vcc.o varnishd-rfc2616.o
varnishd-shmlog.o varnishd-stevedore.o varnishd-storage_file.o
varnishd-storage_malloc.o varnishd-tcp.o varnishd-varnishd.o -Wl,--
export-dynamic ../../lib/libvarnish/.libs/libvarnish.so ../../lib/
libvarnishcompat/.libs/libvarnishcompat.so ../../lib/libvcl/.libs/
libvcl.so -lthr -lm -Wl,--rpath -Wl,/usr/local/lib
varnishd-cache_http.o(.text+0x5a0): In function `http_GetHdr':
../../include/vct.h:47: undefined reference to `vct_typtab'
varnishd-cache_http.o(.text+0x65e): In function `http_GetHdrField':
../../include/vct.h:49: undefined reference to `vct_typtab'
varnishd-cache_http.o(.text+0x679):../../include/vct.h:49: undefined
reference to `vct_typtab'
varnishd-cache_http.o(.text+0x6a5):../../include/vct.h:49: undefined
reference to `vct_typtab'
varnishd-cache_http.o(.text+0x6cd):../../include/vct.h:49: undefined
reference to `vct_typtab'
varnishd-cache_http.o(.text+0x737):../../include/vct.h:47: more
undefined references to `vct_typtab' follow
*** Error code 1
$ uname -a
FreeBSD XX 7.0-RELEASE FreeBSD 7.0-RELEASE #0: Sun Feb 24 10:35:36 UTC
2008 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/GENERIC
amd64
$
$ gcc --version
gcc (GCC) 4.2.1 20070719 [FreeBSD]
Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There
is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
PURPOSE.
$ srcgrep vct_typtab~/varnish/
varnish-cache
include/vct.h:extern unsigned char vct_typtab[256];
include/vct.h:return (vct_typtab[x] & (y));
lib/libvarnish/vct.c:unsigned char vct_typtab[256] = {
Today's svn barfs on configure:
configure: creating ./config.status
config.status: creating Makefile
config.status: creating bin/Makefile
config.status: creating bin/varnishadm/Makefile
config.status: creating bin/varnishd/Makefile
config.status: creating bin/varnishlog/Makefile
config.status: creating bin/varnishhist/Makefile
config.status: creating bin/varnishncsa/Makefile
config.status: creating bin/varnishreplay/Makefile
config.status: creating bin/varnishstat/Makefile
config.status: error: cannot find input file: bin/varnishtest/
Makefile.in
zsh: exit 1 configure
Thanks,
Skye
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Strategy for large cache sets
Thanks, that was my hunch. I'll let you know how it goes when I have some performance metrics. Skye On 1-Jul-08, at 11:24 AM, Poul-Henning Kamp wrote: > In message <[EMAIL PROTECTED]>, Skye Poier > Nott writes > : >> I want to deploy Varnish with very large cache sizes (200GB or more) >> for large, long lived file sets. Is it more efficient to use large >> swap or large mmap in this scenario? > > We have no real-world experience with content of that size, so > the answer is: we don't know. > > Off the bat, I would think files would be better, for exactly the > reson you cite: the swap management is pretty thirsty after metadata. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > [EMAIL PROTECTED] | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by > incompetence. > ___ > varnish-misc mailing list > varnish-misc@projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Strategy for large cache sets
I want to deploy Varnish with very large cache sizes (200GB or more) for large, long lived file sets. Is it more efficient to use large swap or large mmap in this scenario? According to the FreeBSD lists, even 20GB of swap requires 200MB of kern.maxswzone just to keep track of it, so it doesn't seem like that will scale too well. Is one or the other method better for many small files vs less many big files? Thanks again... when I'm a Varnish expert I'll help the newbs :) Skye ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Reloading default.vcl
Is it possible to reload /usr/local/etc/varnish/default.vcl without restarting? From the docs it looks like one way to do this would be through the telnet interface: vcl.load some-unique-name /usr/local/etc/varnish/default.vcl vcl.use some-unique-name each time the default.vcl is changed, unless there's reload option I haven't found. Thanks, Skye ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: A couple of simple questions...
(Whoops, meant to send this to the list) > > | 3. When I want to use Varnish in a WAN transparent accelerator mode, > | if I don't specify any "backend default" is the default to fetch the > | document from the hostname in the URL? > > You have to define a default backend. Hmm, OK. Is the scenario I describe above even possible with Varnish? Can you build a dynamic backend at vcl_recv() time out of req.http.host or something like that? Thanks, Skye ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
A couple of simple questions...
Hello... a few beginner questions:
1. When you restart Varnish the cache is always purged?
2. If I have a section like this in vcl_fetch, is the default_ttl
parameter basically ignored? (assuming default_ttl is less than the
value below). So in other words, this VCL code enforces a minimum TTL?
if (obj.ttl < 36h) {
set obj.ttl = 36h;
}
3. When I want to use Varnish in a WAN transparent accelerator mode,
if I don't specify any "backend default" is the default to fetch the
document from the hostname in the URL?
Thanks,
Skye
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Varnish on FreeBSD 6.3 vs 7.0
Any advantages (performance, stability, resource usage) from running Varnish on FreeBSD 7.0 instead of 6.3? I know they did some work on the threading for 7.0 but maybe it has no relevance to Varnish. Thanks, Skye ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish dumping cache?
Thanks, I'll do as you suggest and see what happens. If it's wedged, then I should do what, attach with gdb and get a backtrace? Thanks, Skye On 20-Apr-08, at 11:41 PM, Poul-Henning Kamp wrote: > In message <[EMAIL PROTECTED]>, Skye Poier > Nott writes > : >> Update: I ran varnishd in foreground with -d and I'm seeing these >> periodically, which would explain the cache invalidation... >> >> Child not responding to ping >> Cache child died pid=23899 status=0x9 > > This is the manager process not getting a reply from the child > process and restarting it, assuming that it is not serving > requests either. > > You need to find out why the child process does not reply to pings. > > The first thing to do is to increase the managers timeout by > increasing > the "cli_timeout" parameter to see if the child process is wedged > or just slow. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > [EMAIL PROTECTED] | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by > incompetence. > ___ > varnish-misc mailing list > varnish-misc@projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish dumping cache?
Update: I ran varnishd in foreground with -d and I'm seeing these periodically, which would explain the cache invalidation... Child not responding to ping Cache child died pid=23899 status=0x9 Clean child Child cleaned start child pid 23914 Child said (2, 23914): <> Cache child died pid=23914 status=0xb Clean child Child cleaned start child pid 23916 Child said (2, 23916): <> Cache child died pid=23916 status=0xb Clean child Child cleaned start child pid 23917 Child said (2, 23917): <> Child said (2, 23917): <> I'm running varnish-1.1.2 built out of ports on amd64...? # uname -a FreeBSD 6.3-RELEASE FreeBSD 6.3-RELEASE #0: Wed Jan 16 01:43:02 UTC 2008 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/SMP amd64 Skye On 14-Apr-08, at 11:55 AM, Skye Poier Nott wrote: > Hello, > > I'm a new Varnish user (coming from Squid - glad to be rid of it) but > I'm seeing some strange behaviour when I test load a Varnish server. > I have a configuration of 4 machines with 250 http client test threads > each requesting a set of 46 files from 2,000 virtual hosts (92,000 > objects total). But while it's running, I see the following: > > # while ((1)); do > date > varnishstat -1 | grep n_object > sleep 5 > done > > n_object will climb steadily and then all of a sudden, it all goes > away: > > n_objecthead86765 . N struct objecthead > Mon Apr 14 11:51:06 PDT 2008 > n_object86780 . N struct object > n_objecthead86782 . N struct objecthead > Mon Apr 14 11:51:11 PDT 2008 > n_object 358 . N struct object > n_objecthead 358 . N struct objecthead > Mon Apr 14 11:51:16 PDT 2008 > n_object 358 . N struct object > n_objecthead 358 . N struct objecthead > > I don't see anything in varnishlog -i Error when this happens but I'm > not sure where to look. > Here's my config: > > param.show > 200 783 > user nobody (65534) > groupnogroup (65533) > default_ttl 86400 [seconds] > thread_pools 1 [pools] > thread_pool_max 1000 [threads] > thread_pool_min 1 [threads] > thread_pool_timeout 120 [seconds] > overflow_max 100 [%] > http_workspace 8192 [bytes] > sess_timeout 5 [seconds] > pipe_timeout 60 [seconds] > send_timeout 600 [seconds] > auto_restart on [bool] > fetch_chunksize 128 [kilobytes] > sendfile_threshold unlimited [bytes] > vcl_traceoff [bool] > listen_address ":80" > listen_depth 1024 [connections] > srcaddr_hash 1049 [buckets] > srcaddr_ttl 30 [seconds] > backend_http11 off [bool] > client_http11off [bool] > ping_interval3 [seconds] > > storage is varnishd_storage="file,/var/cache/varnish,128G" (pre- > alloced) > on FreeBSD 6.3-RELEASE > > Thanks for helping this Varnish newbie. Great daemon! > > Skye > > > ___ > varnish-misc mailing list > varnish-misc@projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Varnish dumping cache?
Hello, I'm a new Varnish user (coming from Squid - glad to be rid of it) but I'm seeing some strange behaviour when I test load a Varnish server. I have a configuration of 4 machines with 250 http client test threads each requesting a set of 46 files from 2,000 virtual hosts (92,000 objects total). But while it's running, I see the following: # while ((1)); do date varnishstat -1 | grep n_object sleep 5 done n_object will climb steadily and then all of a sudden, it all goes away: n_objecthead86765 . N struct objecthead Mon Apr 14 11:51:06 PDT 2008 n_object86780 . N struct object n_objecthead86782 . N struct objecthead Mon Apr 14 11:51:11 PDT 2008 n_object 358 . N struct object n_objecthead 358 . N struct objecthead Mon Apr 14 11:51:16 PDT 2008 n_object 358 . N struct object n_objecthead 358 . N struct objecthead I don't see anything in varnishlog -i Error when this happens but I'm not sure where to look. Here's my config: param.show 200 783 user nobody (65534) groupnogroup (65533) default_ttl 86400 [seconds] thread_pools 1 [pools] thread_pool_max 1000 [threads] thread_pool_min 1 [threads] thread_pool_timeout 120 [seconds] overflow_max 100 [%] http_workspace 8192 [bytes] sess_timeout 5 [seconds] pipe_timeout 60 [seconds] send_timeout 600 [seconds] auto_restart on [bool] fetch_chunksize 128 [kilobytes] sendfile_threshold unlimited [bytes] vcl_traceoff [bool] listen_address ":80" listen_depth 1024 [connections] srcaddr_hash 1049 [buckets] srcaddr_ttl 30 [seconds] backend_http11 off [bool] client_http11off [bool] ping_interval3 [seconds] storage is varnishd_storage="file,/var/cache/varnish,128G" (pre-alloced) on FreeBSD 6.3-RELEASE Thanks for helping this Varnish newbie. Great daemon! Skye ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
