"ExpBan: nnn was banned"
Hi folks, We have a newly deployed Varnish installation (v2.0.4 on Debian 5.0 64-bit). Our hitrate is pretty good (80%+) and the load on the server is consistently 0.00. The cache size is 50G and the TTL is set to basically cache things indefinitely (it's set to like 1000 years, I think). I got a complaint from a user that some percentage of his page views seem too slow to be coming from the cache. I brought up our home page and started refreshing it over and over. Sure enough, while most of the page views are in fact getting cached, once every 5 or 6 times, I get a cache miss. When this happens, varnishlog shows something like this: 32 VCL_call c recv lookup 32 VCL_call c hash hash * 32 ExpBan c 1607291667 was banned * 32 VCL_call c miss fetch I apologize if my google-fu is broken today, but I can't seem to find a description of what the message above means. I would certainly appreciate any information (or links) that would help explain this message and how to prevent the problem going forward. Thanks and regards, Martin ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Thread memory allocation question
In message <5c056ae2-7207-42f8-9e4b-0f541dc4b...@slide.com>, Ken Brownfield wri tes: >Would a stack overflow take out the whole child, or just that thread? The kernel would try to extend the stack and provided you are not on a 32 bit system, it shouldn't ever have a problem with that. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Thread memory allocation question
On Jun 19, 2009, at 7:15 AM, Tollef Fog Heen wrote: > | 40111000 8192K rw---[ anon ] > Looks like the default stack size. Ah, of course. Good find, thanks. I'm thinking it might be nice to have a thread track its stack history and emit its approximate largest size when it's reaped (and the workspaces too I suppose). Would a stack overflow take out the whole child, or just that thread? The 1024K blocks roughly add up to the SMA outstanding bytes, so I'm assuming these are jemalloc block allocations, and not related to thread count: lib/jemalloc/malloc.c: #define CHUNK_2POW_DEFAULT 20 Thanks! -- Ken. On Jun 19, 2009, at 7:15 AM, Tollef Fog Heen wrote: > ]] Ken Brownfield > > | When looking at /proc/map info for varnish threads, I'm seeing the > | following allocations in numbers that essentially match the child > count: > | > | 40111000 8192K rw---[ anon ] > > Looks like the default stack size. > > | And this at almost double the child count: > | > | 7f4d5790 1024K rw---[ anon ] > > Unsure what this is. > > | I noticed some pretty intense memory usage when I had a backend > issue > | and the thread count increased into the hundreds. Obviously the > | threads will need memory as they scale, but are these large > | allocations intentional, and are they tunable (beyond the relatively > | small workspaces?) > > You should be able to tune it using ulimit -s. If you turn it too > low, > things will break, though. > > -- > Tollef Fog Heen > Redpill Linpro -- Changing the game! > t: +47 21 54 41 73 ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Apache DoS - is Varnish affected?
In message <20090619200701.0dd70...@fabiankeil.de>, Fabian Keil writes: >Actually I think accf_http(9) would only delay the attack. > >While the man page doesn't mention it, accf_http passes >incomplete requests to the userland if its buffer is full. Yeah, but I'm pretty sure the buffer would contain enough junk to make varnish shut the connection immediately, so the fd starvation would not happen. Anyway, if you are interested in this DoS, you can trivially test it yourselv with a telnet connection and patience in front of the keyboard. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Apache DoS - is Varnish affected?
"Poul-Henning Kamp" wrote: > In message <4a3ba393.3010...@loman.net>, Nick Loman writes: > >I would guess that Varnish isn't affected by this, but does anyone know > >for sure? Does Varnish protect against this attack in all cases if you > >have Apache as your backend? > > > >http://isc.sans.org/diary.html?storyid=6601 > > Varnish will abandon the connection after a fixed number of header > lines. > > This attack is more or less exactly _why_ varnish has a fixed limit > on HTTP headers. > > I won't claim that varnish is imune, but the impact should be manageable. > > Systems using "http accept filters" (FreeBSD possibly others) the Varnish > (or apache) will never even see these connections in the first place. Actually I think accf_http(9) would only delay the attack. While the man page doesn't mention it, accf_http passes incomplete requests to the userland if its buffer is full. Fabian signature.asc Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Apache DoS - is Varnish affected?
In message <4a3bbfe6.7070...@stillbilde.net>, "Svein Skogen (listmail account)" writes: >> Systems using "http accept filters" (FreeBSD possibly others) the Varnish >> (or apache) will never even see these connections in the first place. > >Does this basically mean that in these uncertain times where kiddiots >DoS (Destroying our Sanity) just for the fun of it, using accf_http >should go into the checklist for every webserver, or am I reading this >wrong? Good question, I don't know how accf_http will handle these attacks. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Apache DoS - is Varnish affected?
In message <4a3bb2e1.8090...@loman.net>, Nick Loman writes: >Poul-Henning Kamp wrote: >> Varnish will abandon the connection after a fixed number of header >> lines. > >That's reassuring. Out of interest, what is the limit? 32 - 3 (for the first line fields) >Presumably that limit * the read timeout is the length of time a >connection could be held open by a rogue client? Something like that, I have not tried it. Worst case it would be a timeout for each character. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Apache DoS - is Varnish affected?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Poul-Henning Kamp wrote: > > Systems using "http accept filters" (FreeBSD possibly others) the Varnish > (or apache) will never even see these connections in the first place. > Does this basically mean that in these uncertain times where kiddiots DoS (Destroying our Sanity) just for the fun of it, using accf_http should go into the checklist for every webserver, or am I reading this wrong? //Svein - -- - +---+--- /"\ |Svein Skogen | sv...@d80.iso100.no \ / |Solberg Østli 9| PGP Key: 0xE5E76831 X|2020 Skedsmokorset | sv...@jernhuset.no / \ |Norway | PGP Key: 0xCE96CE13 | | sv...@stillbilde.net ascii | | PGP Key: 0x58CD33B6 ribbon |System Admin | svein-listm...@stillbilde.net Campaign|stillbilde.net | PGP Key: 0x22D494A4 +---+--- |msn messenger: | Mobile Phone: +47 907 03 575 |sv...@jernhuset.no | RIPE handle:SS16503-RIPE - +---+--- Picture Gallery: https://gallery.stillbilde.net/v/svein/ - -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAko7v+YACgkQODUnwSLUlKTYbACeKqc2n1UfTGLhkuq1QdP5g+IC PUIAoJrA4eC13T1JGCt2Y7GYIxdYT/Eo =2dIU -END PGP SIGNATURE- ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Apache DoS - is Varnish affected?
Poul-Henning Kamp wrote: > In message <4a3ba393.3010...@loman.net>, Nick Loman writes: > >> I would guess that Varnish isn't affected by this, but does anyone know >> for sure? Does Varnish protect against this attack in all cases if you >> have Apache as your backend? >> >> http://isc.sans.org/diary.html?storyid=6601 >> > > Varnish will abandon the connection after a fixed number of header > lines. > > This attack is more or less exactly _why_ varnish has a fixed limit > on HTTP headers. > Hi Poul-Henning, That's reassuring. Out of interest, what is the limit? Presumably that limit * the read timeout is the length of time a connection could be held open by a rogue client? I agree that is probably manageable but of course still potentially serious in the context of a significant DoS attempt. Cheers, Nick. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Apache DoS - is Varnish affected?
In message <4a3ba393.3010...@loman.net>, Nick Loman writes: >I would guess that Varnish isn't affected by this, but does anyone know >for sure? Does Varnish protect against this attack in all cases if you >have Apache as your backend? > >http://isc.sans.org/diary.html?storyid=6601 Varnish will abandon the connection after a fixed number of header lines. This attack is more or less exactly _why_ varnish has a fixed limit on HTTP headers. I won't claim that varnish is imune, but the impact should be manageable. Systems using "http accept filters" (FreeBSD possibly others) the Varnish (or apache) will never even see these connections in the first place. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Apache DoS - is Varnish affected?
I would guess that Varnish isn't affected by this, but does anyone know for sure? Does Varnish protect against this attack in all cases if you have Apache as your backend? http://isc.sans.org/diary.html?storyid=6601 Many thanks, Nick. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Thread memory allocation question
]] Ken Brownfield | When looking at /proc/map info for varnish threads, I'm seeing the | following allocations in numbers that essentially match the child count: | | 40111000 8192K rw---[ anon ] Looks like the default stack size. | And this at almost double the child count: | | 7f4d5790 1024K rw---[ anon ] Unsure what this is. | I noticed some pretty intense memory usage when I had a backend issue | and the thread count increased into the hundreds. Obviously the | threads will need memory as they scale, but are these large | allocations intentional, and are they tunable (beyond the relatively | small workspaces?) You should be able to tune it using ulimit -s. If you turn it too low, things will break, though. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Thread memory allocation question
When looking at /proc/map info for varnish threads, I'm seeing the following allocations in numbers that essentially match the child count: 40111000 8192K rw---[ anon ] And this at almost double the child count: 7f4d5790 1024K rw---[ anon ] For example, for 64 worker threads, I see 69 of the 8192K allocations, and 121 of the 1024K allocations. For 32 worker threads I see 37 and 82, respectively. I noticed some pretty intense memory usage when I had a backend issue and the thread count increased into the hundreds. Obviously the threads will need memory as they scale, but are these large allocations intentional, and are they tunable (beyond the relatively small workspaces?) Many thanks, -- Ken. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
