Re: Varnish logging and data merging

2010-01-11 Thread pub crawler
Thanks again Rob!

Tracking pixel it is then :)  Off to cobble some code together and test.


On Mon, Jan 11, 2010 at 3:42 AM, Rob S  wrote:
> pub crawler wrote:
>>
>> We have a lot of logging going on in our applications. Logs pages, IP
>> info, time date, URL parameters, etc.  Since many pages are being
>> served out of Varnish cache,  they don't get logged by our
>> application.
>>
>> How is anyone else out there working around this sort of problem with
>> an existing application?  Considering a 1x1 graphic file inclusion
>> into our pages to facilitate logging and ensuring Varnish doesn't
>> cache it
>
> We've not reported analytics based on raw Apache logs for a long time -
> they're far too polluted by spiders etc.  So, instead, for RSS we use
> Feedburner (who provide stats, and reduce our traffic load), and for general
> web access we either use Google Analytics (or commercial alternatives).
>  Google Analytics allows you to add and report on custom parameters, so it
> is very flexible.  However, as you suggested, an alternative is to use a
> tracking pixel.  Depending on how you've previously processed your
> information, you may find it very useful to encourage Varnish to route all
> hits to your tracking pixel to a specific server (obviously with failover).
>  This'd save you having to aggregate logs across multiple servers.
>
> Rob
>
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish logging and data merging

2010-01-11 Thread Rob S
pub crawler wrote:
> We have a lot of logging going on in our applications. Logs pages, IP
> info, time date, URL parameters, etc.  Since many pages are being
> served out of Varnish cache,  they don't get logged by our
> application.
>
> How is anyone else out there working around this sort of problem with
> an existing application?  Considering a 1x1 graphic file inclusion
> into our pages to facilitate logging and ensuring Varnish doesn't
> cache it
We've not reported analytics based on raw Apache logs for a long time - 
they're far too polluted by spiders etc.  So, instead, for RSS we use 
Feedburner (who provide stats, and reduce our traffic load), and for 
general web access we either use Google Analytics (or commercial 
alternatives).  Google Analytics allows you to add and report on custom 
parameters, so it is very flexible.  However, as you suggested, an 
alternative is to use a tracking pixel.  Depending on how you've 
previously processed your information, you may find it very useful to 
encourage Varnish to route all hits to your tracking pixel to a specific 
server (obviously with failover).  This'd save you having to aggregate 
logs across multiple servers.

Rob
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish logging and data merging

2010-01-11 Thread Poul-Henning Kamp
In message <4c3149fb1001102237xaf0d732hb22418df4cf21...@mail.gmail.com>, pub cr
awler writes:

>We have a lot of logging going on in our applications. Logs pages, IP
>info, time date, URL parameters, etc.  Since many pages are being
>served out of Varnish cache,  they don't get logged by our
>application.

run varnishncsa or varnishlog to collect your loginfo.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish logging and data merging

2010-01-10 Thread Michael S. Fischer
Varnish does keep a log if you ask it to.

On Jan 10, 2010, at 10:37 PM, pub crawler   
wrote:

> Alright, up and running with Varnish successfully. Quite happy with
> Varnish.  Our app servers no longer are failing / overwhelmed.
>
> Here's our new problem...
>
> We have a lot of logging going on in our applications. Logs pages, IP
> info, time date, URL parameters, etc.  Since many pages are being
> served out of Varnish cache,  they don't get logged by our
> application.
>
> How is anyone else out there working around this sort of problem with
> an existing application?  Considering a 1x1 graphic file inclusion
> into our pages to facilitate logging and ensuring Varnish doesn't
> cache it.
>
> Share your ideas.
>
> -Paul
> ___
> varnish-misc mailing list
> varnish-misc@projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Varnish logging and data merging

2010-01-10 Thread pub crawler
Alright, up and running with Varnish successfully. Quite happy with
Varnish.  Our app servers no longer are failing / overwhelmed.

Here's our new problem...

We have a lot of logging going on in our applications. Logs pages, IP
info, time date, URL parameters, etc.  Since many pages are being
served out of Varnish cache,  they don't get logged by our
application.

How is anyone else out there working around this sort of problem with
an existing application?  Considering a 1x1 graphic file inclusion
into our pages to facilitate logging and ensuring Varnish doesn't
cache it.

Share your ideas.

-Paul
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish Logging

2007-12-07 Thread Poul-Henning Kamp
In message <[EMAIL PROTECTED]>, Saint-Genest Gwenael writes:
>Rich Christiansen wrote:

>> transfered.  We deal a lot with large files (30MB-100MB), so many of our
>> file transfers are cut short.  We need to be able to log exactly how
>> many bytes were actually transfered.  Is this possible?
>
>if (WRK_Flush(w) == 0 &&
>sendfile(*w->wfd, fd, &off, len) != len)
>w->werr++;
>
>The number of bytes copied (returned by sendfile) is not saved :(
>
>With a little patch you can save this value and use it for statistics.

Please open a ticket for this, so we get the number properly integrated
in the statistics.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
[EMAIL PROTECTED] | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish Logging

2007-12-07 Thread Saint-Genest Gwenael
Rich Christiansen wrote:
> Hello, all!

Hi !

> 1) varnishncsa seems to only log the size (in bytes) of the file being
> transfered.  We deal a lot with large files (30MB-100MB), so many of our
> file transfers are cut short.  We need to be able to log exactly how
> many bytes were actually transfered.  Is this possible?

I think you can't get this information for the moment.

In your case, file transfer is made by RES_WriteObj() (cache_response.c)
with a call to WRK_Sendfile() (cache_pool.c). If you look into thins
function :

if (WRK_Flush(w) == 0 &&
sendfile(*w->wfd, fd, &off, len) != len)
w->werr++;

The number of bytes copied (returned by sendfile) is not saved :(

With a little patch you can save this value and use it for statistics.

Gwen

-- 
Saint-Genest Gwenael <[EMAIL PROTECTED]>
Makina Corpus - http://www.makina-corpus.com/
44 boulevard des Pas enchantés FR-44230 Saint-Sébastien-sur-Loire
T : +33 (0) 2 51 79 80 80  P: +33 (0)6 81 05 03 00
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Varnish Logging

2007-12-06 Thread Rich Christiansen

Hello, all!

First of all, thanks to everyone for making such a great product.  
Varnish has performed wonderfully.  It really does 'just work'.


I am having some problems regarding logging, however.  Our specific app 
requires detailed logging data for every transaction.  Namely, we have 
to track the number of bytes leaving our app so that we can charge 
clients based on data transfer usage.  So, for using varnish, we've got 
two problems:


1) varnishncsa seems to only log the size (in bytes) of the file being 
transfered.  We deal a lot with large files (30MB-100MB), so many of our 
file transfers are cut short.  We need to be able to log exactly how 
many bytes were actually transfered.  Is this possible?


2) We've also noticed that if the file transfer is aborted before about 
2MB is transfered, the file transfer isn't logged at all to access.log.  
Is this by design?  For the same reasons stated in #1, we need to find a 
reason to log these transfers as well.


On a side note, I tried using Squid (yuck) and found that it satisfied 
point 1, but /very/ strangely also had the same 'problem' described in 
point 2.  (Although the 'logging threshold' was lower, around 1MB or 
so)  So weird.



I'd love to hear your thoughts!  If we can't get this working, I'm going 
to have to resort to a suboptimal solution using lighttpd and rsync...(yuck)


Thanks in advance,
-Rich
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish logging

2007-04-30 Thread Dirk Gomez

No output is written with this command:

varnishncsa -adw foo

(Just tried: it now segfaults)


I'm unable to reproduce this.  Are you sure varnishd and varnishncsa
are using the same library versions?  And do you have a backtrace for
the segfault?


Hmm, not able to reproduce twenty minutes later.

Here are the library versions:

anadyr:~# ldd /usr/bin/varnishncsa
linux-gate.so.1 =>  (0xe000)
libvarnish.so.0 => /usr/lib/libvarnish.so.0 (0x4001e000)
libvarnishapi.so.0 => /usr/lib/libvarnishapi.so.0 (0x40023000)
libdl.so.2 => /lib/tls/libdl.so.2 (0x40027000)
librt.so.1 => /lib/tls/librt.so.1 (0x4002b000)
libc.so.6 => /lib/tls/libc.so.6 (0x40033000)
/lib/ld-linux.so.2 (0x4000)
libpthread.so.0 => /lib/tls/libpthread.so.0 (0x40166000)
anadyr:~# ldd /usr/sbin/varnishd
linux-gate.so.1 =>  (0xe000)
libvarnish.so.0 => /usr/lib/libvarnish.so.0 (0x4001e000)
libvcl.so.0 => /usr/lib/libvcl.so.0 (0x40023000)
libdl.so.2 => /lib/tls/libdl.so.2 (0x4003)
librt.so.1 => /lib/tls/librt.so.1 (0x40034000)
libc.so.6 => /lib/tls/libc.so.6 (0x4003c000)
libpthread.so.0 => /lib/tls/libpthread.so.0 (0x4016f000)
/lib/ld-linux.so.2 (0x4000)

(Debian Etch, self-built varnish-103 package with your "HOST"  patch  
from trunk for varnishncsa)


varnishncsa returns to its nanosleep routine.

Should I recompile with   --enable-developer-warnings and  --enable- 
debugging-symbols set to yes and retry strace?





___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish logging

2007-04-30 Thread Dirk Gomez

(Debian Etch, self-built varnish-103 package with your "HOST"  patch
from trunk for varnishncsa)


I assume you rebuilt and reinstalled the entire package?


Yes (so that everything fits together).




varnishncsa returns to its nanosleep routine.


Does varnishlog behave normally?


Yes.


Should I recompile with   --enable-developer-warnings and  --enable-
debugging-symbols set to yes and retry strace?


Those flags won't make any difference for strace, but they will make
it easier to get a backtrace if it segfaults again.  Please remember
to 'make clean' before rebuilding with these flags, otherwise they
won't be applied.


No more segfaults.

varnishncsa -d outputs something now - just for one of our "virtual  
hosts", none other makes it to varnishncsa's output. varnishlog works  
fine tho: it outputs the other vhosts as well.


___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish logging

2007-04-30 Thread Dag-Erling Smørgrav
Dirk Gomez <[EMAIL PROTECTED]> writes:
> (Debian Etch, self-built varnish-103 package with your "HOST"  patch
> from trunk for varnishncsa)

I assume you rebuilt and reinstalled the entire package?

> varnishncsa returns to its nanosleep routine.

Does varnishlog behave normally?

> Should I recompile with   --enable-developer-warnings and  --enable-
> debugging-symbols set to yes and retry strace?

Those flags won't make any difference for strace, but they will make
it easier to get a backtrace if it segfaults again.  Please remember
to 'make clean' before rebuilding with these flags, otherwise they
won't be applied.

DES
-- 
Dag-Erling Smørgrav
Senior Software Developer
Linpro AS - www.linpro.no
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish logging

2007-04-30 Thread Dag-Erling Smørgrav
Dirk Gomez <[EMAIL PROTECTED]> writes:
> varnishncsa -d outputs something now - just for one of our "virtual
> hosts", none other makes it to varnishncsa's output. varnishlog works
> fine tho: it outputs the other vhosts as well.

OK, I would like you to do the following:

 - capture a few minutes of traffic to a file using 'varnishlog -w file'

 - verify that 'varnishncsa -r file' exhibits the bug you experience
   when running it "live"

 - send me that file (directly, not to the list)

Hopefully, I will be able to find and correct the error quickly.  I'm
very busy this week, though, so please be patient.

DES
-- 
Dag-Erling Smørgrav
Senior Software Developer
Linpro AS - www.linpro.no
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish logging

2007-04-30 Thread Dag-Erling Smørgrav
Dirk Gomez <[EMAIL PROTECTED]> writes:
> DES, here's what I did:
>
> * Checked out the 1.0.3 tag.
> * Checkout above file from trunk.
> * Stuck it into the obvious place
> * Compiled, created and installed a Debian varnish-103 package.
>
> varnishncsa doesn't write to a log file anymore though.

You mean -w is broken, or that it stops?  Does it work if you do not
specify -w?  What exact command line are you using?

DES
-- 
Dag-Erling Smørgrav
Senior Software Developer
Linpro AS - www.linpro.no
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish logging

2007-04-30 Thread Dag-Erling Smørgrav
Dirk Gomez <[EMAIL PROTECTED]> writes:
> No output is written with this command:
>
> varnishncsa -adw foo
>
> (Just tried: it now segfaults)

I'm unable to reproduce this.  Are you sure varnishd and varnishncsa
are using the same library versions?  And do you have a backtrace for
the segfault?

DES
-- 
Dag-Erling Smørgrav
Senior Software Developer
Linpro AS - www.linpro.no
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish logging

2007-04-30 Thread Dirk Gomez

You mean -w is broken, or that it stops?  Does it work if you do not
specify -w?  What exact command line are you using?


No output is written with this command:

varnishncsa -adw foo

(Just tried: it now segfaults)

___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish logging

2007-04-30 Thread Dirk Gomez

Dirk, if you are comfortable building Varnish from sources, try
grabbing the following file:

http://varnish.projects.linpro.no/svn/trunk/varnish-cache/bin/ 
varnishncsa/varnishncsa.c


Stick it in the obvious place in your source tree, rebuild and
reinstall.  It will work fine in a 1.0 tree.


DES, here's what I did:

* Checked out the 1.0.3 tag.
* Checkout above file from trunk.
* Stuck it into the obvious place
* Compiled, created and installed a Debian varnish-103 package.

varnishncsa doesn't write to a log file anymore though.

execve("/usr/bin/varnishncsa", ["/usr/bin/varnishncsa", "-w", "foo"],  
[/* 19 vars */]) = 0

uname({sys="Linux", node="test01.dirkgomez.de", ...}) = 0
brk(0)  = 0x804b000
access("/etc/ld.so.nohwcap", F_OK)  = -1 ENOENT (No such file or  
directory)
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,  
-1, 0) = 0x40017000
access("/etc/ld.so.preload", R_OK)  = -1 ENOENT (No such file or  
directory)

open("/etc/ld.so.cache", O_RDONLY)  = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=19929, ...}) = 0
mmap2(NULL, 19929, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40019000
close(3)= 0
access("/etc/ld.so.nohwcap", F_OK)  = -1 ENOENT (No such file or  
directory)

open("/usr/lib/libvarnish.so.0", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20\21\0"...,  
512) = 512

fstat64(3, {st_mode=S_IFREG|0644, st_size=16468, ...}) = 0
mmap2(NULL, 19480, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,  
0) = 0x4001e000
mmap2(0x40022000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED| 
MAP_DENYWRITE, 3, 0x3) = 0x40022000

close(3)= 0
access("/etc/ld.so.nohwcap", F_OK)  = -1 ENOENT (No such file or  
directory)

open("/usr/lib/libvarnishapi.so.0", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\360\f\0"...,  
512) = 512

fstat64(3, {st_mode=S_IFREG|0644, st_size=11352, ...}) = 0
mmap2(NULL, 14600, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,  
0) = 0x40023000
mmap2(0x40026000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED| 
MAP_DENYWRITE, 3, 0x2) = 0x40026000

close(3)= 0
access("/etc/ld.so.nohwcap", F_OK)  = -1 ENOENT (No such file or  
directory)

open("/lib/tls/libpthread.so.0", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\360G\0"...,  
512) = 512

fstat64(3, {st_mode=S_IFREG|0755, st_size=85010, ...}) = 0
mmap2(NULL, 70104, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,  
0) = 0x40166000
mmap2(0x40174000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED| 
MAP_DENYWRITE, 3, 0xd) = 0x40174000
mmap2(0x40176000, 4568, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED| 
MAP_ANONYMOUS, -1, 0) = 0x40176000

close(3)= 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,  
-1, 0) = 0x40178000

mprotect(0x4015b000, 20480, PROT_READ)  = 0
set_thread_area({entry_number:-1 -> 6, base_addr:0x401786c0, limit: 
1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1,  
seg_not_present:0, useable:1}) = 0

munmap(0x40019000, 19929)   = 0
set_tid_address(0x40178708) = 8910
rt_sigaction(SIGRTMIN, {0x4016a450, [], SA_SIGINFO}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x4016a3c0, [], SA_RESTART|SA_SIGINFO}, NULL,  
8) = 0

rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY})  
= 0

uname({sys="Linux", node="test01.dirkgomez.de", ...}) = 0
brk(0)  = 0x804b000
brk(0x807c000)  = 0x807c000
open("/tmp/_.vsl", O_RDONLY)= 3
read(3, "2\332y\371\214\1\0\0=\3501F\344\1\0\0\0\0\200\0+\362\1"...,  
396) = 396

mmap2(NULL, 8389004, PROT_READ, MAP_SHARED, 3, 0) = 0x40179000
open("foo", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4
rt_sigaction(SIGHUP, {0x8048b90, [HUP], SA_RESTART}, {SIG_DFL}, 8) = 0
nanosleep({0, 5000}, NULL)  = 0
nanosleep({0, 5000}, NULL)  = 0
nanosleep({0, 5000}, NULL)  = 0
...

and then it continues to nanosleep.

varnishlog though does show activity on the server.

-- Dirk

___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish logging

2007-04-23 Thread Dirk Gomez

Done in r1362.

Dirk, if you are comfortable building Varnish from sources, try
grabbing the following file:

http://varnish.projects.linpro.no/svn/trunk/varnish-cache/bin/ 
varnishncsa/varnishncsa.c


Stick it in the obvious place in your source tree, rebuild and
reinstall.  It will work fine in a 1.0 tree.


DES, I checked out varnishncsa from trunk head, compiled it and  
replaced the binary that came with Debian (new)stable with the newly  
built one - works fine.


Thanks a lot!

-- Dirk
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish logging

2007-04-22 Thread Dirk Gomez

The setting is: a lot of virtual hosts running on top of Zope. We
deliver statistics generated with server logfiles.

Basically, you want NCSA format logfiles split by the value of
the "Host:" header ?


No, he wants the Host: header prepended to the request URL in NCSA
logs.  It's easy to do, I'll take care of it.


Exactly. We have statistics processing on one logfile that has worked  
for years and we would like to continue using it.


___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish logging

2007-04-21 Thread Dag-Erling Smørgrav
[EMAIL PROTECTED] (Dag-Erling Smørgrav) writes:
> "Poul-Henning Kamp" <[EMAIL PROTECTED]> writes:
> > Dirk Gomez <[EMAIL PROTECTED]> writes:
> > > The setting is: a lot of virtual hosts running on top of Zope. We
> > > deliver statistics generated with server logfiles.
> > Basically, you want NCSA format logfiles split by the value of
> > the "Host:" header ?
> No, he wants the Host: header prepended to the request URL in NCSA
> logs.  It's easy to do, I'll take care of it.

Done in r1362.

Dirk, if you are comfortable building Varnish from sources, try
grabbing the following file:

http://varnish.projects.linpro.no/svn/trunk/varnish-cache/bin/varnishncsa/varnishncsa.c

Stick it in the obvious place in your source tree, rebuild and
reinstall.  It will work fine in a 1.0 tree.

DES
-- 
Dag-Erling Smørgrav
Senior Software Developer
Linpro AS - www.linpro.no
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish logging

2007-04-21 Thread Dag-Erling Smørgrav
"Poul-Henning Kamp" <[EMAIL PROTECTED]> writes:
> Dirk Gomez <[EMAIL PROTECTED]> writes:
> > The setting is: a lot of virtual hosts running on top of Zope. We
> > deliver statistics generated with server logfiles.
> Basically, you want NCSA format logfiles split by the value of
> the "Host:" header ?

No, he wants the Host: header prepended to the request URL in NCSA
logs.  It's easy to do, I'll take care of it.

DES
-- 
Dag-Erling Smørgrav
Senior Software Developer
Linpro AS - www.linpro.no
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish logging

2007-04-19 Thread Poul-Henning Kamp
In message <[EMAIL PROTECTED]>, Dirk Gomez writ
es:
>Hi list,
>
>The setting is: a lot of virtual hosts running on top of Zope. We  
>deliver statistics generated with server logfiles.

Basically, you want NCSA format logfiles split by the value of
the "Host:" header ?


-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
[EMAIL PROTECTED] | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Varnish logging

2007-04-19 Thread Dirk Gomez

Hi list,

The setting is: a lot of virtual hosts running on top of Zope. We  
deliver statistics generated with server logfiles.


We are perfectly content with post-processing a big logfile with  
entries for several domains - we had a script like that when we were  
still using Squid.


Comparing the logfiles generated by Squid and Varnish we find that  
those generated by Varnish lack any Host prefix in the relevant 'GET'  
field so we are unable to process them as we did with Squid's.


As the manpage does not say anything about a configuration option to  
alter this behaviour, we'd be thankful for hints in this case.  
Filtering before logging was proposed in an earlier post- in our case  
this does not seem to be a good solution as there are lots of  
changing domains involved and so the configuration would be quite  
tedious- it would be entirely sufficient to have the host logged by  
default.


Appendix: Some lines of the logfiles, for comparison

Squid:

89.59.80.86 - - [06/Apr/2007:13:28:05 +0200] "GET http:// 
op.elseware.de:8080/content/logobao.gif HTTP/1.1" 200 17554 "http:// 
op.elseware.de/content/e3224/e10/e589/e594/e702/index_ger.html"  
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR  
2.0.50727)" TCP_REFRESH_MISS:DIRECT
89.59.80.86 - - [06/Apr/2007:13:28:05 +0200] "GET http:// 
op.elseware.de:8080/common/masses.jpg HTTP/1.1" 200 2681 "http:// 
op.elseware.de/content/e3224/e10/e589/e594/e702/index_ger.html"  
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR  
2.0.50727)" TCP_MEM_HIT:NONE
89.59.80.86 - - [06/Apr/2007:13:28:05 +0200] "GET http:// 
op.elseware.de:8080/favicon.ico HTTP/1.1" 200 1407 "-" "Mozilla/4.0  
(compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727)"  
TCP_MEM_HIT:NONE
87.160.144.136 - - [06/Apr/2007:13:28:08 +0200] "GET http:// 
www.csg.elseware.com:8080/content/e1756/index_html? HTTP/1.1" 200  
14826 "http://www.csg.elseware.com/content/e1756"; "Mozilla/5.0  
(Windows; U; Windows NT 5.1; de; rv:1.8.0.11) Gecko/20070312 Firefox/ 
1.5.0.11" TCP_MISS:DIRECT



Varnish:

87.160.200.220 - - [10/Apr/2007:00:36:05 +0200] "GET /sites/nele- 
elseware.de/myzms/content/imgb01_ger.jpg HTTP/1.1" 304 - "http:// 
www.nele-elseware.de/content/e531/index_ger.html" "Mozilla/4.0  
(compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)"
87.160.200.220 - - [10/Apr/2007:00:36:26 +0200] "GET /sites/nele- 
elseware.de/myzms/content/e531/e706/milonga_del_serafin_ger.mp3 HTTP/ 
1.1" 304 - "http://www.nele-elseware.de/content/e531/index_ger.html";  
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)"
87.160.200.220 - - [10/Apr/2007:00:36:31 +0200] "GET /sites/nele- 
elseware.de/myzms/content/e531/e706/milonga_del_serafin_ger.mp3 HTTP/ 
1.1" 304 - "-" "NSPlayer/9.0.0.3250 WMFSDK/9.0"
87.160.200.220 - - [10/Apr/2007:00:41:23 +0200] "GET /sites/nele- 
elseware.de/myzms/content/e531/e686/el_choclo_ger.mp3 HTTP/1.1" 304 -  
"http://www.nele-elseware.de/content/e531/index_ger.html"; "Mozilla/ 
4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)"
87.160.200.220 - - [10/Apr/2007:00:45:40 +0200] "GET /sites/nele- 
elseware.de/myzms/content/e531/e543/a_los_paisanos_ger.pdf HTTP/1.1"  
200 479110 "http://www.nele-elseware.de/content/e531/index_ger.html";  
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)"
87.160.200.220 - - [10/Apr/2007:00:46:29 +0200] "GET /common/logo.gif  
HTTP/1.1" 304 - "http://www.nele-elseware.de/content/e531/ 
index_ger.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;  
SV1; HbTools 4.8.4)"
87.160.200.220 - - [10/Apr/2007:00:46:29 +0200] "GET /sites/nele- 
elseware.de/myzms/content/imga01_ger.jpg HTTP/1.1" 304 - "http:// 
www.nele-elseware.de/content/e531/index_ger.html" "Mozilla/4.0  
(compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)"
87.160.200.220 - - [10/Apr/2007:00:46:30 +0200] "GET /common/deu.jpg  
HTTP/1.1" 304 - "http://www.nele-elseware.de/content/e531/ 
index_ger.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;  
SV1; HbTools 4.8.4)"
87.160.200.220 - - [10/Apr/2007:00:46:30 +0200] "GET /common/eng.jpg  
HTTP/1.1" 304 - "http://www.nele-elseware.de/content/e531/ 
index_ger.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;  
SV1; HbTools 4.8.4)"
87.160.200.220 - - [10/Apr/2007:00:46:30 +0200] "GET /common/esp.jpg  
HTTP/1.1" 304 - "http://www.nele-elseware.de/content/e531/ 
index_ger.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;  
SV1; HbTools 4.8.4)"
87.160.200.220 - - [10/Apr/2007:00:46:30 +0200] "GET /sites/nele- 
elseware.de/myzms/content/imgb01_ger.jpg HTTP/1.1" 304 - "http:// 
www.nele-elseware.de/content/e531/index_ger.html" "Mozilla/4.0  
(compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)"
87.160.200.220 - - [10/Apr/2007:00:46:30 +0200] "GET /misc_/zms/ 
mime_type.application_pdf.gif HTTP/1.1" 304 - "http://www.nele- 
elseware.de/content/e531/index_ger.html" "Mozilla/4.0 (compatible;  
MSIE 6.0; Windows NT 5.1; SV1; HbToo