Re: [us...@httpd] Reliable combined log sorting
Hi Jason,
On Sun, May 9, 2010 at 5:41 AM, Jason Nunnelley ja...@jasonn.com wrote:
In our configuration of Apache we push all logs to a single file via
syslog-ng. I need to create some basic log analysis (like AWstats style) for
a single domain out of several domains that post to that file. Any
suggestions on a simple solution?
What LogFormat are you using, precisely ? And what, precisely, is the
problem you are having (you mention sorting in the subject, but nothing
related to it in the body).
If all you require is the ability to filter for a single vhost, have a look
at http://httpd.apache.org/docs/2.2/mod/mod_log_config.html, make sure your
LogFormat-string contains %v somewhere, and filter by the respective field
in the syslog-ng file -- which can trivially be done with awk or similar. If
your line looks something like this :
yourvhost.example.com 66.249.xx.xx - - [09/May/2010:15:25:56 +] GET
/xxx HTTP/1.1 200 8284 - Mozilla/5.0 (compatible; Googlebot/2.1; +
http://www.google.com/bot.html) 261 8619
due to a LogFormat of %v %h %l %u %t \%r\ %s %b \%{Referer}i\
\%{User-Agent}i\ %I %O combined_2 (or something similar), then
awk '$1 ~ /yourvhost\.example\.com/ { print }' logfilename
will filter out just those lines where the first field ($1) matches your
VirtualHost ServerName of yourvhost.example.com -- which you are then free
to pipe to whatever log analysis method you want. If you want this to also
filter out the syslog-related timestamps, awk can do that as well;
http://www.gnu.org/manual/gawk/gawk.html should provide sufficient detail on
how manipulate the print-statement.
If your problem is that the log entries are not in strictly chronological
order (which most decent log analysis software can deal with as long as the
delta is not too large), the problem becomes different beast altogether --
one which you'd likely want to employ a more sophisticated perl or python
script since you require more state -- code for which I don't have handy at
the moment ;-)
If this does not answer your question, please provide some more example log
and code.
--Eike
Re: [us...@httpd] Reliable combined log sorting
On 5/9/10 10:36 AM, Eike Frost wrote: Hi Jason, On Sun, May 9, 2010 at 5:41 AM, Jason Nunnelley ja...@jasonn.com mailto:ja...@jasonn.com wrote: In our configuration of Apache we push all logs to a single file via syslog-ng. I need to create some basic log analysis (like AWstats style) for a single domain out of several domains that post to that file. Any suggestions on a simple solution? What LogFormat are you using, precisely ? And what, precisely, is the problem you are having (you mention sorting in the subject, but nothing related to it in the body). I decided to punt and just grep out the lines I needed. It does have the %v config in the Apache conf. Most of what I'm dealing with are old logs, so I'm not so much looking (yet) to create better logs in the future (which seems to me would be best to split out by virtual host). This is the config I have for cranking out some AWstats. My lazy punt solution was to just grep out the lines that relate to my vhost of choice, perhaps I could set a simple perl script to split those into appropriate segregated logs. My log config: LogFormat = %other %other %other %other %virtualname %host %other %logname %time1 %methodurl %code %bytesd This is an actual log example for one vhost in the combined file. 2010-05-09T16:18:44-0400 local3.info domID-XX-XX-XX-XX-XX-XX http: www.domain.com 10.10.10.XX - - [09/May/2010:16:18:43 -0400] GET /galleries/image.jpg HTTP/1.1 200 28553 http://www.domain.com/swf/include/nav.swf; Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 2.0.50727) I have two sets of log types to deal with: historical logs; future logs. -- Jason A. Nunnelley +1 2562971652 http://www.google.com/profiles/imjasonn [Member Tekany, LLC]
[us...@httpd] Reliable combined log sorting
In our configuration of Apache we push all logs to a single file via syslog-ng. I need to create some basic log analysis (like AWstats style) for a single domain out of several domains that post to that file. Any suggestions on a simple solution? -- Jason A. Nunnelley +1 2562971652 http://www.google.com/profiles/imjasonn [Member Tekany, LLC] - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
