On Fri, 7 Dec 2012, Ben Bradley wrote:
Hi everyone
I've just started investigating centralised logging and I'm gradually building
up a plan of action.
I'd like to store the logs on a central server running logstash/ElasticSearch so they can be
searched and monitored using Kibana. With rsyslog sending the logs over the network to a logstash
server. I don't want to run logstash as the log "sender" on each server, I'd prefer to
keep the servers (log "clients") as lean and simple possible. So that means either using
syslog, syslog-ng or the one I'm testing now, rsyslog.
1) Should I have rsyslog sending to logstash over the network? Or should I be
running another rsyslog on the collector server, which then sends to logstash
for processing?
This is up to you, there are advantages in each direction.
Using rsyslog for all network transport and having it deliver locally to
logstash/elasticsearch/other for processing means that you you can take
advantage of all rsyslog features for your transport.
In a centralized environment your traffic volumes can be high, rsyslog can
handle very high traffic levels, can your other software?
If you are really comforatable with logstash, you may want to eliminate the need
to run one more daemon, but when you hire new people and hand the system over,
should they need to be as comfortable with logstash? They will have to be
comforatable with rsyslog in any case. At that point which is the 'extra' thing
to deal with, rsyslog or logstash?
For Apache, I would like to have separate vhost log files on the web server, in
addition to these logs being sent to a remote log collector.
I've tested rsyslog using the imfile module to watch each Apache log files, but
this means I have to hard-code each vhost log file into my rsyslog.conf. This
is not ideal as people will invariably forget when they add/remove sites on the
server.
2) What's the best way to log to both vhost-specific log files on the web
server and to send these logs over the network, without using imfile and
manually watching tens of individual log files?
Get Apache to log to rsyslog, then have rsyslog split the log to both a file
and over the network to logstash?
Are there big performance implications for logging both locally and over the
network?
I could change my Apache config to log to a single access/error log for all
vhosts, then watch these main log files with imfile. So long as rsyslog is
then able to produce vhost-specific log files somewhere on the web server
machine.
it depends on how you format the log file. If you have the logfile start with
the vhost name, then rsyslog can easily produce per-host files (look in the
rsyslog documentation for the dynafile templates.
another approach you can do is have apache log to a local named pipe and have a
process listen on that named pipe and tagging/reformatting the log file and pass
it to your syslog server.
David Lang
Any comments/suggestions?
I am sure others have had a similar need. I just don't want to ditch local log
files until we fully know how well the centralised log server performs.
Thanks in advance!
Cheers, Ben
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
