how to...accelarate randon access to millions of images?

2008-03-14 Thread Sascha Ottolski
Hi, I'm relatively new to varnish (I'm having an eye on it since it appeared in public, but so far never really used it). Now the time may have come to give it whirl. And am wondering if someone could give me a little advise to get me going. The challenge is to server 20+ million image files,

Re: how to...accelarate randon access to millions of images?

2008-03-17 Thread Sascha Ottolski
Am Sonntag 16 März 2008 15:54:42 schrieben Sie: > Sascha Ottolski <[EMAIL PROTECTED]> writes: > > Now my question is: what kind of hardware would I need? Lots of RAM > > seems to be obvious, what ever "a lot" may be...What about the disk > > subsystem? Should I

Re: how to...accelarate randon access to millions of images?

2008-03-17 Thread Sascha Ottolski
Michael, thanks a lot for taking the time to give me such a detailed answer. please see my replies below. Am Sonntag 16 März 2008 18:00:42 schrieb Michael S. Fischer: > On Fri, Mar 14, 2008 at 1:37 PM, Sascha Ottolski <[EMAIL PROTECTED]> wrote: > > The challenge is to ser

Re: how to...accelarate randon access to millions of images?

2008-03-19 Thread Sascha Ottolski
Am Mittwoch 19 März 2008 12:31:04 schrieb Dag-Erling Smørgrav: > > and, sorry if this is FAQ, are the storage files persistent, that > > is, will they survive a restart of varnish or reboot of the > > machine, or do you always start with an empty cache? > > Varnish always startes with an empty cach

Re: Miscellaneous questions

2008-03-31 Thread Sascha Ottolski
Am Dienstag 18 März 2008 00:07:59 schrieb Michael S. Fischer: > On Mon, Mar 17, 2008 at 3:32 PM, DHF <[EMAIL PROTECTED]> wrote: > > This is called CARP/"Cache Array Routing Protocol" in squid land. > > Here's a link to some info on it: > > > > http://docs.huihoo.com/gnu_linux/squid/html/x2398.ht

production ready devel snapshot?

2008-03-31 Thread Sascha Ottolski
Hi, probably a stupid question, but if I'd like to use more recent features like the load-balancer, and since the latest official release is a bit dated, is there anything like a snapshot release that is worth giving it a try, especially if my configuration will hopefully stay simple for a whi

the most basic config

2008-04-01 Thread Sascha Ottolski
Hi, I'm a bit puzzled by the examples and the explanation of the "default" vcl config presented in the man page. Now I'm wondering, if I want to make my first steps for creating a reverse proxy for static images only, that basically should cache everything indefinetely (as long as cache space

Re: the most basic config

2008-04-01 Thread Sascha Ottolski
Am Dienstag 01 April 2008 17:42:17 schrieb DHF: > Sascha Ottolski wrote: > > Hi, > > > > I'm a bit puzzled by the examples and the explanation of the > > "default" vcl config presented in the man page. Now I'm wondering, > > if I want to

Re: the most basic config

2008-04-02 Thread Sascha Ottolski
Am Dienstag 01 April 2008 21:15:11 schrieb Sascha Ottolski: > thanks very much, this was very helpful. now, could anyone give me a > hint how to interpret the output of varnishhist? > > I'm seeing > > client_req              15054       123.39 Client reques

cache empties itself?

2008-04-02 Thread Sascha Ottolski
Hi, how can this be? My varnish runs for about 36 hours now. yesterday evening, the resident memory size was like 10 GB, which is still way below the available 32. later that evening, I stopped letting request to the proxy over night. now I came back, let the request back in, and am wondering

Re: cache empties itself?

2008-04-03 Thread Sascha Ottolski
Am Donnerstag 03 April 2008 18:07:53 schrieb DHF: > Sascha Ottolski wrote: > > how can this be? My varnish runs for about 36 hours now. yesterday > > evening, the resident memory size was like 10 GB, which is still > > way below the available 32. later that evening, I stoppe

Re: cache empties itself?

2008-04-03 Thread Sascha Ottolski
Am Donnerstag 03 April 2008 18:07:53 schrieb DHF: > > how can this be? My varnish runs for about 36 hours now. yesterday > > evening, the resident memory size was like 10 GB, which is still > > way below the available 32. later that evening, I stopped letting > > request to the proxy over night. no

Re: cache empties itself?

2008-04-03 Thread Sascha Ottolski
Am Donnerstag 03 April 2008 19:30:25 schrieb Michael S. Fischer: > On Thu, Apr 3, 2008 at 10:26 AM, Sascha Ottolski <[EMAIL PROTECTED]> wrote: > > All this with 1.1.2. It's vital to my setup to cache as many > > objects as possible, for a long time, and that they really

Re: cache empties itself?

2008-04-03 Thread Sascha Ottolski
Am Donnerstag 03 April 2008 21:45:25 schrieb Michael S. Fischer: > On Thu, Apr 3, 2008 at 11:53 AM, Ricardo Newbery <[EMAIL PROTECTED]> wrote: > > On Apr 3, 2008, at 11:04 AM, Michael S. Fischer wrote: > > > On Thu, Apr 3, 2008 at 10:58 AM, Sascha Ottolski &l

Re: cache empties itself?

2008-04-04 Thread Sascha Ottolski
Am Freitag 04 April 2008 01:32:28 schrieb DHF: > Sascha Ottolski wrote: > > however, my main problem is currently that the varnish childs keep > > restarting, and that this empties the cache, which effectively > > renders the whole setup useless for me :-( if the cache has fi

Re: cache empties itself?

2008-04-04 Thread Sascha Ottolski
Am Freitag 04 April 2008 04:37:44 schrieb Ricardo Newbery: >           sub vcl_fetch { >               if (obj.ttl < 120s) { >                   set obj.ttl = 120s; >               } >           } > > Or you can invent your own header... let's call it  X-Varnish-1day > >           sub vcl_fetch { >

make varnish still respond if backend dead

2008-04-04 Thread Sascha Ottolski
Hi, sorry if this is FAQ: what can I do to make varnish respond to request if it's backend is dead. should return cache hits, of course, and a "proxy error" or something for a miss. and how can I prevent varnish to cache "404" for objects it couldn't fetch due to a dead backend? at least I thi

Re: cache empties itself?

2008-04-04 Thread Sascha Ottolski
Am Freitag 04 April 2008 10:11:52 schrieb Stig Sandbeck Mathisen: > On Fri, 4 Apr 2008 09:01:57 +0200, Sascha Ottolski <[EMAIL PROTECTED]> said: > > I definetely did nothing like this, I've observed restarts "out of > > the blue". I'm no giv

unable to compile nagios module from trunk

2008-04-04 Thread Sascha Ottolski
after checking out and running autogen.sh, configure stops with this error: ./configure: line 19308: syntax error near unexpected token `VARNISHAPI,' ./configure: line 19308: `PKG_CHECK_MODULES(VARNISHAPI, varnishapi)' Cheers, Sascha ___ varnish-misc

Re: cache empties itself?

2008-04-04 Thread Sascha Ottolski
Am Freitag 04 April 2008 11:50:51 schrieb Michael S. Fischer: > On Thu, Apr 3, 2008 at 8:59 PM, Ricardo Newbery <[EMAIL PROTECTED]> wrote: > > Well, first of all you're setting up a false dichotomy. Not > > everything fits neatly into your apparent definitions of dynamic > > versus static. Your

Re: cache empties itself?

2008-04-04 Thread Sascha Ottolski
Am Freitag 04 April 2008 18:11:23 schrieb Michael S. Fischer: > On Fri, Apr 4, 2008 at 3:20 AM, Sascha Ottolski <[EMAIL PROTECTED]> wrote: > > you are right, _if_ the working set is small. in my case, we're > > talking 20+ mio. small images (5-50 KB each), 400+ GB i

recommendation for swap space?

2008-04-06 Thread Sascha Ottolski
Hi, now that my varnish processes start to reach the RAM size, I'm wondering what a dimension of swap would be wise? I currently have about 30 GB swap space for 32 GB RAM, but am wondering if it could even make sense to have no swap at all? My cache file is 517 GB in size. BTW, the trunk seems

Re: recommendation for swap space?

2008-04-08 Thread Sascha Ottolski
Am Montag 07 April 2008 18:00:14 schrieb Dag-Erling Smørgrav: > Sascha Ottolski <[EMAIL PROTECTED]> writes: > > now that my varnish processes start to reach the RAM size, I'm > > wondering what a dimension of swap would be wise? I currently have > > about 30 GB

mass purge causes high load?

2008-04-10 Thread Sascha Ottolski
Hi, I just needed to get rid of about 27,000 stale URLs, that were cached as 404 or 302 due to a configuration error on the backends. So I did a url.purge in a loop, sleeping 0.1 seconds after each URL: for i in `cat notfound.txt.sorted` ; do varnishadm -T:81 url.purge $i; sleep 0.1; done How

Re: mass purge causes high load?

2008-04-13 Thread Sascha Ottolski
Am Sonntag 13 April 2008 12:08:55 schrieben Sie: > Sascha Ottolski <[EMAIL PROTECTED]> writes: > > I just needed to get rid of about 27,000 stale URLs, that were > > cached as 404 or 302 due to a configuration error on the backends. > > Why couldn't you just wai

Re: mass purge causes high load?

2008-04-14 Thread Sascha Ottolski
Am Montag 14 April 2008 14:19:11 schrieb Dag-Erling Smørgrav: > Sascha Ottolski <[EMAIL PROTECTED]> writes: > > Dag-Erling Smørgrav <[EMAIL PROTECTED]> writes: > > > No, the semantics are completely different. With HTTP PURGE, you > > > do a direct cache l

Re: make varnish still respond if backend dead

2008-04-24 Thread Sascha Ottolski
status == 404) { set obj.ttl = 7200s; } } so 404 still may happen, but are cached shorter than my default. Cheers, Sascha > > On Fri, Apr 4, 2008 at 12:51 AM, Sascha Ottolski <[EMAIL PROTECTED]> wrote: > > Hi, > > > > sorry if this is FAQ: what can I do to

please help on compiling nagios plugin

2008-05-29 Thread Sascha Ottolski
Hi, asked it some weeks ago, but got no answer :-( when running configure, I see this error: ./configure: line 19292: syntax error near unexpected token `VARNISHAPI,' ./configure: line 19292: `PKG_CHECK_MODULES(VARNISHAPI, varnishapi)' the line is # Checks for libraries. PKG_CHECK_MODULES(VAR

Re: please help on compiling nagios plugin

2008-05-29 Thread Sascha Ottolski
Am Donnerstag 29 Mai 2008 17:06:41 schrieb Sascha Ottolski: > Hi, > > asked it some weeks ago, but got no answer :-( > > when running configure, I see this error: > > ./configure: line 19292: syntax error near unexpected token > `VARNISHAPI,' > ./configure

Re: Performance options for trunk

2008-05-30 Thread Sascha Ottolski
Am Freitag 30 Mai 2008 14:01:35 schrieb Audun Ytterdal: > I run trunk in front of a site. I have 3 varnishservers, all with > 32GB > > of ram serving only small pictures, thumbnails and profile pictures. > The cacheset is pretty large (1.5 TB) and changing much over time. > And before you all ask w

Re: Strategy for large cache sets

2008-07-03 Thread Sascha Ottolski
Am Dienstag 01 Juli 2008 20:16:15 schrieb Skye Poier Nott: > I want to deploy Varnish with very large cache sizes (200GB or more) > for large, long lived file sets. Is it more efficient to use large > swap or large mmap in this scenario? > > According to the FreeBSD lists, even 20GB of swap requir

want to allow or deny an URL based on a timestamp

2008-08-12 Thread Sascha Ottolski
Hi all, I'm certain that it's possible, but am not sure how to do it: I want to let my application create "encrypted" URLs, that are valid only for a specific amount of time. I guess with the help of embedded C and manually constructed hash keys this should be doable. Now I'm wondering if may

Re: want to allow or deny an URL based on a timestamp

2008-08-12 Thread Sascha Ottolski
Am Mittwoch 13 August 2008 00:02:07 schrieb Darryl Dixon - Winterhouse Consulting: > 4) Object expires and Varnish goes to fetch it from the backend, > which of course returns 404 or whatever as the URL has expired. Darryl, thanks for your reply. This would of course be a straight forward appro

on linux: only one cpu in use?

2008-09-08 Thread Sascha Ottolski
on a 4 core machine, running varnish exclusively, all the I/O goes to one core only: # cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 0: 603869381 0 0 0IO-APIC-edge timer 6: 3 0 0 0IO-APIC-edge fl

Re: on linux: only one cpu in use?

2008-09-16 Thread Sascha Ottolski
Am Montag 15 September 2008 16:16:33 schrieb Tollef Fog Heen: > ]] Sascha Ottolski > > | it's debian etch with the stock 2.6.18 kernel. performance is ok, > | beside the fact that the load climbs up when the cache is about > | 40-50 % full (with a 517GB cache file); below

Re: on linux: only one cpu in use?

2008-10-16 Thread Sascha Ottolski
Am Montag 29 September 2008 14:12:16 schrieb Tollef Fog Heen: > ]] Sascha Ottolski > > | Am Montag 15 September 2008 16:16:33 schrieb Tollef Fog Heen: > | > ]] Sascha Ottolski > | > > | > | it's debian etch with the stock 2.6.18 kernel. performance is > | &g

varnish 2.0.1 truncates pre-created cache file?

2008-10-20 Thread Sascha Ottolski
Hi, bug or feature? With the new version, my cache file gets shrunk. I made it with dd with a size of ~259 GB, but after starting varnishd, only about 119 GB are left. I don't give a size parameter when starting, as in -s file,/var/cache/varnish/store.bin which worked as expected with old

Re: varnish 2.0.1 truncates pre-created cache file?

2008-10-20 Thread Sascha Ottolski
Am Montag 20 Oktober 2008 16:59:37 schrieb Sascha Ottolski: > Hi, > > bug or feature? With the new version, my cache file gets shrunk. I > made it with dd with a size of ~259 GB, but after starting varnishd, > only about 119 GB are left. I don't give a size parameter wh

inline C-Code

2008-10-21 Thread Sascha Ottolski
Hi, in one or the other presentation slide is a small example about inline C-Code; and there is a commit that only says "undocmented and unsupported" (http://varnish.projects.linpro.no/changeset/1301). Now I'm wondering, is there anyway at list ab bit of documentation somewhere, or could someo

next release?

2008-12-21 Thread Sascha Ottolski
Hi folks, I'm curious when the next stable (minor) release is planned. I'm especially interested in "Date: 2008-11-14 01:19:33 +0100 (Fri, 14 Nov 2008) New Revision: 3390 Modified: trunk/varnish-cache/lib/libvarnish/binary_heap.c Log: Rework the binary heap, we use for expiry processing, to

Re: AW: caching everything incl. 403?

2008-12-21 Thread Sascha Ottolski
Am Sonntag 21 Dezember 2008 14:18:56 schrieb neil.schol...@scoyo.com: > Hi, > > It doesn't work :/ > > sub vcl_fetch { > > /* Set NON-200 caching */ > if (obj.status != 200) > { > set obj.cacheable = "1"; > set obj.ttl = 300s;

"uptime" not available to nagios plugin?

2008-12-31 Thread Sascha Ottolski
Hi, bug or feature? The nagios plugin seems not to be able to extract the uptime: img-proxy1:~# /usr/local/libexec/check_varnish -p uptime Unknown parameter 'uptime' VARNISH UNKNOWN: (null) (3)|uptime=3 but it probably should, shouldn't it? img-proxy1:~# varnishstat -l 2>&1|grep uptime uptime

Re: flushing cache doesn't seem to make varnish know about less

2009-01-11 Thread Sascha Ottolski
Am Montag 12 Januar 2009 05:35:16 schrieb Timothy Ball: > a programming error caused varnish to think there were billions of > pages it had to know about. bug is quashed but varnish doesn't seem > to know > > # this is a line from top > 3596 0.2 28.3g 28g 3808 256 57m 1916 120 S 20 0

Panic message: Assert error in exp_timer(), cache_expire.c line 303

2009-01-26 Thread Sascha Ottolski
Dear list, after introducing a little change to my VCL, we now see crashes like this frequently: Jan 25 01:49:43 localhost varnishd[6039]: Child (25613) not responding to ping, killing it. Jan 25 01:49:43 localhost varnishd[6039]: Child (25613) died signal=6 Jan 25 01:49:43 localhost varnishd[6

Re: Panic message: Assert error in exp_timer(), cache_expire.c line 303

2009-02-01 Thread Sascha Ottolski
Am Mittwoch 28 Januar 2009 10:52:47 schrieb Poul-Henning Kamp: > In message <200901260917.45223.ottol...@web.de>, Sascha Ottolski writes: > >Assert error in exp_timer(), cache_expire.c line 303: > >Condition(oe2->timer_when >= oe->timer_when) not tru

strange restart, taking 15 minutes

2009-03-07 Thread Sascha Ottolski
Hi, I've just seen a strange restart that heppened this morning, after only about 25 h runtime; apparantly, there was a problem, but the parent process needed serveral kill attempts and more than 15 minutes to finally restart it's child: # grep varnish /var/log/syslog Mar 7 08:52:42 localhost

Re: strange restart, taking 15 minutes

2009-03-09 Thread Sascha Ottolski
Am Samstag 07 März 2009 18:52:04 schrieb Sascha Ottolski: > I've just seen a strange restart that heppened this morning, after > only about 25 h runtime; please ignore the posting, it just came out that the server has a broken hdd. Thanks for listening anyw