Re: Virtual Host Logging Perl Script
Hi there, On Tue, 15 Jul 2003, Jez Hancock wrote: Does anyone how one could log errorlog entries in a similar manner to the script above - ie pipe the errorlog to a script which appends one copy of the error entry to a main error logfile and another copy to the virtual host's error logfile? It's possible, but I don't think you really want to do it. You would be asking the server to do more than is necessary while handling each request (you probably already are:). Would it not be better for example to rotate your logs frequently, and to process them afterwards, off-line? You would then have a choice of doing things with the various log file analysis tools too. If you are a little bit creative in what you log, it can be simple to extract the information you need for each vhost from one file. Have you considered using something other than flat files for logging? 73, Ged.
Re: Virtual Host Logging Perl Script
On Wed, Jul 16, 2003 at 12:40:06PM +0200, Guillaume Fougnies wrote: Wed, Jul 16, 2003 at 11:07:47AM +0100: Jez Hancock wrote: I would do this but we wanted to give our users 'live' logfiles, rather than making them wait until log rotation before being able to view them (or did I misunderstand you?). During my work on the ENodes project, i've developed a tiny threaded logger in c to manage internal logfiles (by webmaster/website/version). (you can rewrite it in perl with a 5.8.x built with thread...) Perl handlers send log lines through a UNIX socket to the logger which is opening logfiles when needed and keeping them opened. It avoids the overhead of opening and locking each time the file and provides as many online debug logfile as you ask. /me nods. This would save a lot of overhead. The only benefit of opening a new pipe each time I suppose is that you don't need to worry about rotation. The logger is launched in a 'PerlRequire' file. The client have a persistant connection to the logger. If it fits your needs, you can get sources here: http://www.enodes.org Server : utils/enodes_logger/enodes_logger.c Client module: enodes_core/modules/ENodes/Core/ApLog.pm Many thanks, I'll investigate now. Well ENodes sounds very interesting, particularly the part about webmasters being able to test new versioning without applying changes and the reduced httpd restart feature. Very interesting. Will certainly have a look at this, we're currently considering something called webcp, a PHP project which has much potential but is sitll unfortunately very much in beta. Thanks! -- Jez http://www.munk.nu/
Re: Virtual Host Logging Perl Script
Hi Jez, On Wed, 16 Jul 2003, Jez Hancock wrote: [snip] We started looking at mod_log_sql: [snip] but had trouble getting it to work on FreeBSD unfortunately. I'd have thought something a bit lighter might do for this. Right now it seems a bit silly having a separate ErrorLog line in each of the apache virtual host stubs, but as far as I am aware there isn't an easier way is there? You could look at mod_macro. 73, Ged.
RE: Virtual Host Logging Perl Script (mod_macro note)
Right now it seems a bit silly having a separate ErrorLog line in each
of the apache virtual host stubs, but as far as I am aware there isn't
an easier way is there?
You could look at mod_macro.
mod_macro (http://www.coelho.net/mod_macro) works great for me. I found
that I had to make a few changes to get it to build for Apache 2.0.46. To
wit:
110c110
module macro_module;
---
module AP_MODULE_DECLARE_DATA macro_module;
810c810,811
char * name, * where, * recursion;
---
char * name, * recursion;
char * where = ???;
951c952
AP_DECLARE_DATA module macro_module = {
---
module AP_MODULE_DECLARE_DATA macro_module = {
The preload of 'where' may be redundant, I don't remember. I believe that
the use of AP_MODULE_DECLARE_DATA was change that I had to make or nothing
would build. YMMV
mma
Re: Virtual Host Logging Perl Script (mod_macro note)
On Wed, Jul 16, 2003 at 12:40:10PM -0700, Marc M. Adkins wrote: Right now it seems a bit silly having a separate ErrorLog line in each of the apache virtual host stubs, but as far as I am aware there isn't an easier way is there? You could look at mod_macro. mod_macro (http://www.coelho.net/mod_macro) works great for me. I found that I had to make a few changes to get it to build for Apache 2.0.46. To My main motivation for running apache2 would be to make use of the PerChild MPM which from what I can glean is still not working :( This directive would save me a lot of hassles wrt running cgi scripts (php in particular) with the euid of the script owner. All this as I'm tussling with mod_suphp... -- Jez http://www.munk.nu/
Re: Virtual Host Logging Perl Script
Jez Hancock wrote: Hi, I've just written a short perl script to perform logging for our virtual hosts. The code has plenty of comments so I'll paste it below. My question is: would it be possible to use mod_perl in some way to perform the function of the script? In testing the speed of the script seems reasonable enough, is there a better way to do what I'm doing below? Take a look at: http://perl.apache.org/docs/2.0/user/handlers/http.html#PerlLogHandler a similar code will work for mp1 if you don't use 2.0. __ Stas BekmanJAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide --- http://perl.apache.org mailto:[EMAIL PROTECTED] http://use.perl.org http://apacheweek.com http://modperlbook.org http://apache.org http://ticketmaster.com
Re: Virtual Host Logging Perl Script
On Tue, Jul 15, 2003 at 01:32:11PM +0300, Stas Bekman wrote: Jez Hancock wrote: My question is: would it be possible to use mod_perl in some way to perform the function of the script? In testing the speed of the script seems reasonable enough, is there a better way to do what I'm doing below? Take a look at: http://perl.apache.org/docs/2.0/user/handlers/http.html#PerlLogHandler a similar code will work for mp1 if you don't use 2.0. Much obliged, that does look to be something I could use. Reminds me I need to be locking the logfile as well ;) Thanks Stas, -- Jez http://www.munk.nu/
Re: Virtual Host Logging Perl Script
On Tue, 2003-07-15 at 14:27, Jez Hancock wrote: On Tue, Jul 15, 2003 at 01:32:11PM +0300, Stas Bekman wrote: Take a look at: http://perl.apache.org/docs/2.0/user/handlers/http.html#PerlLogHandler a similar code will work for mp1 if you don't use 2.0. Much obliged, that does look to be something I could use. Reminds me I need to be locking the logfile as well ;) Actually, if you are just printing one short line I think you'll be fine without locking on unix systems. See the FAQ: http://www.perldoc.com/perl5.8.0/pod/perlfaq5.html#All-I-want-to-do-is-append-a-small-amount-of-text-to-the-end-of-a-file.--Do-I-still-have-to-use-locking- - Perrin
Re: Virtual Host Logging Perl Script
Perrin Harkins wrote: On Tue, 2003-07-15 at 14:27, Jez Hancock wrote: On Tue, Jul 15, 2003 at 01:32:11PM +0300, Stas Bekman wrote: Take a look at: http://perl.apache.org/docs/2.0/user/handlers/http.html#PerlLogHandler a similar code will work for mp1 if you don't use 2.0. Much obliged, that does look to be something I could use. Reminds me I need to be locking the logfile as well ;) Actually, if you are just printing one short line I think you'll be fine without locking on unix systems. See the FAQ: http://www.perldoc.com/perl5.8.0/pod/perlfaq5.html#All-I-want-to-do-is-append-a-small-amount-of-text-to-the-end-of-a-file.--Do-I-still-have-to-use-locking- Is this still correct in the threaded environment where the filehandle is shared across several threads? The writes are atomic 4k or so, indeed, but what happens if two threads write to the same filehandle at the same time on the SMP machine? Won't things get messed up? I haven't checked, perhaps perlio already provides an internal locking. Arthur would be a good person to ask. __ Stas BekmanJAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide --- http://perl.apache.org mailto:[EMAIL PROTECTED] http://use.perl.org http://apacheweek.com http://modperlbook.org http://apache.org http://ticketmaster.com
Re: Virtual Host Logging Perl Script
On Tue, 2003-07-15 at 16:11, Stas Bekman wrote: Is this still correct in the threaded environment where the filehandle is shared across several threads? Why would the filehandle be shared? Wouldn't you open a new handle in each thread? I expect this would be fine, since the behavior is enforced by the system library rather than by Perl. You can always use flock if you're feeling paranoid about it though. - Perrin
Re: Virtual Host Logging Perl Script
On Tue, Jul 15, 2003 at 02:39:18PM -0400, Perrin Harkins wrote: On Tue, 2003-07-15 at 14:27, Jez Hancock wrote: Much obliged, that does look to be something I could use. Reminds me I need to be locking the logfile as well ;) Actually, if you are just printing one short line I think you'll be fine without locking on unix systems. See the FAQ: http://www.perldoc.com/perl5.8.0/pod/perlfaq5.html#All-I-want-to-do-is-append-a-small-amount-of-text-to-the-end-of-a-file.--Do-I-still-have-to-use-locking- Ok thanks Perrin. The script is actually running nicely as it is right now on a server hosting 50 or so domains, although admittedly it's not taking a massive number of hits (10,000 or so per day). Does anyone how one could log errorlog entries in a similar manner to the script above - ie pipe the errorlog to a script which appends one copy of the error entry to a main error logfile and another copy to the virtual host's error logfile? -- Jez http://www.munk.nu/
Re: Virtual Host Logging Perl Script
Perrin Harkins wrote: On Tue, 2003-07-15 at 16:11, Stas Bekman wrote: Is this still correct in the threaded environment where the filehandle is shared across several threads? Why would the filehandle be shared? Wouldn't you open a new handle in each thread? Because it's shared on the OS level. You open a new perl IO handle in each thread, but I think that they all use the same filedescriptor underneath. I think Arthur was mentioning that at his ithreads talk. Need to run some tests to confirm that. I expect this would be fine, since the behavior is enforced by the system library rather than by Perl. You can always use flock if you're feeling paranoid about it though. Are you sure? Won't flock maintain a process-wide flag? Unless there is a special threads-aware implementation that does it right. This is of course just rambling based on known issuws with all the other process-wide variables like cwd, chroot, environ, etc. __ Stas BekmanJAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide --- http://perl.apache.org mailto:[EMAIL PROTECTED] http://use.perl.org http://apacheweek.com http://modperlbook.org http://apache.org http://ticketmaster.com
Virtual Host Logging Perl Script
Hi,
I've just written a short perl script to perform logging for our virtual
hosts. The code has plenty of comments so I'll paste it below.
My question is: would it be possible to use mod_perl in some way to
perform the function of the script? In testing the speed of the script
seems reasonable enough, is there a better way to do what I'm doing
below?
On a related note, I'd ideally like to be able to do similar for Apache
Error logging - ie log apache error log entries once into a main
errorlog file and once into a vhost errorlog file. Is this possible
with ErrorLog lines?
My initial understanding is this isn't possible using the framework
outlined in the code below.
Code follows:
#!/usr/bin/perl
# Script to pipe apache log entries to virtually hosted log files
# Assumes httpd.conf has the following:
# LogFormat %v %h %l %u %t \%r\ %s %b commonvhost
# and that ONLY the following logging line is used in the httpd.conf:
# CustomLog | /path/to/logger.pl commonvhost
# DO NOT configure the CustomLog directive in the vhost stubs for vhosts
# or this will not work.
# Script logs commonvhost entries to a logfile with a template of:
# /var/log/httpd/virtual.domain/$year/$month/$day
use strict;
my $logEntry = ;
# get the vhost from this log entry:
$logEntry=~/(.*?) /;
my $vhost = $1;
my ($year, $month, $day) = (
(localtime)[5]+1900,
sprintf(%02d, (localtime)[4]+1),
sprintf(%02d, (localtime)[3])
);
# Name of access logfiles:
my $accessLogName = httpd-access.log;
=comment
$logdir:
Location to put all logfiles
This will log everything into:
$logdir/all/$year/$month/$day/httpd-access.log
and put individual vhost logfiles into:
$logdir/$vhost/$year/$month/$day/httpd-access.log
=cut
my $logDir = /var/log/httpd;
my $allLogDir = $logDir/all/$year/$month/$day;
my $vhostLogDir = $logDir/$vhost/$year/$month/$day;
writeLog($allLogDir, all);
writeLog($vhostLogDir, vhost);
# write a log entry to a file
sub writeLog(){
my $logDir = shift @_;
my $type = shift @_;
if( ! -d $logDir ) {
`mkdir -p $logDir`;
}
open(FD, $logDir/$accessLogName);
# if type is vhost, strip off the vhost data:
if($type eq vhost){
$logEntry =~s/.*? //;
}
print FD $logEntry;
close FD;
}
--
Jez
http://www.munk.nu/
Virtual Host Logging Perl Script
Hi,
I've just written a short perl script to perform logging for our virtual
hosts. The code has plenty of comments so I'll paste it below.
My question is: would it be possible to use mod_perl in some way to
perform the function of the script? In testing the speed of the script
seems reasonable enough, is there a better way to do what I'm doing
below?
On a related note, I'd ideally like to be able to do similar for Apache
Error logging - ie log apache error log entries once into a main
errorlog file and once into a vhost errorlog file. Is this possible
with ErrorLog lines?
My initial understanding is this isn't possible using the framework
outlined in the code below.
Code follows:
#!/usr/bin/perl
# Script to pipe apache log entries to virtually hosted log files
# Assumes httpd.conf has the following:
# LogFormat %v %h %l %u %t \%r\ %s %b commonvhost
# and that ONLY the following logging line is used in the httpd.conf:
# CustomLog | /path/to/logger.pl commonvhost
# DO NOT configure the CustomLog directive in the vhost stubs for vhosts
# or this will not work.
# Script logs commonvhost entries to a logfile with a template of:
# /var/log/httpd/virtual.domain/$year/$month/$day
use strict;
my $logEntry = ;
# get the vhost from this log entry:
$logEntry=~/(.*?) /;
my $vhost = $1;
my ($year, $month, $day) = (
(localtime)[5]+1900,
sprintf(%02d, (localtime)[4]+1),
sprintf(%02d, (localtime)[3])
);
# Name of access logfiles:
my $accessLogName = httpd-access.log;
=comment
$logdir:
Location to put all logfiles
This will log everything into:
$logdir/all/$year/$month/$day/httpd-access.log
and put individual vhost logfiles into:
$logdir/$vhost/$year/$month/$day/httpd-access.log
=cut
my $logDir = /var/log/httpd;
my $allLogDir = $logDir/all/$year/$month/$day;
my $vhostLogDir = $logDir/$vhost/$year/$month/$day;
writeLog($allLogDir, all);
writeLog($vhostLogDir, vhost);
# write a log entry to a file
sub writeLog(){
my $logDir = shift @_;
my $type = shift @_;
if( ! -d $logDir ) {
`mkdir -p $logDir`;
}
open(FD, $logDir/$accessLogName);
# if type is vhost, strip off the vhost data:
if($type eq vhost){
$logEntry =~s/.*? //;
}
print FD $logEntry;
close FD;
}
--
Jez
http://www.munk.nu/
