On Fri, 7 Dec 2012, Ben Bradley wrote:

Hi everyone

I've just started investigating centralised logging and I'm gradually building 
up a plan of action.

I'd like to store the logs on a central server running logstash/ElasticSearch so they can be 
searched and monitored using Kibana. With rsyslog sending the logs over the network to a logstash 
server. I don't want to run logstash as the log "sender" on each server, I'd prefer to 
keep the servers (log "clients") as lean and simple possible. So that means either using 
syslog, syslog-ng or the one I'm testing now, rsyslog.

1) Should I have rsyslog sending to logstash over the network? Or should I be running another rsyslog on the collector server, which then sends to logstash for processing?

This is up to you, there are advantages in each direction.

Using rsyslog for all network transport and having it deliver locally to logstash/elasticsearch/other for processing means that you you can take advantage of all rsyslog features for your transport.

In a centralized environment your traffic volumes can be high, rsyslog can handle very high traffic levels, can your other software?

If you are really comforatable with logstash, you may want to eliminate the need to run one more daemon, but when you hire new people and hand the system over, should they need to be as comfortable with logstash? They will have to be comforatable with rsyslog in any case. At that point which is the 'extra' thing to deal with, rsyslog or logstash?

For Apache, I would like to have separate vhost log files on the web server, in 
addition to these logs being sent to a remote log collector.

I've tested rsyslog using the imfile module to watch each Apache log files, but 
this means I have to hard-code each vhost log file into my rsyslog.conf. This 
is not ideal as people will invariably forget when they add/remove sites on the 
server.

2) What's the best way to log to both vhost-specific log files on the web 
server and to send these logs over the network, without using imfile and 
manually watching tens of individual log files?
Get Apache to log to rsyslog, then have rsyslog split the log to both a file 
and over the network to logstash?
Are there big performance implications for logging both locally and over the 
network?

I could change my Apache config to log to a single access/error log for all vhosts, then watch these main log files with imfile. So long as rsyslog is then able to produce vhost-specific log files somewhere on the web server machine.

it depends on how you format the log file. If you have the logfile start with the vhost name, then rsyslog can easily produce per-host files (look in the rsyslog documentation for the dynafile templates.

another approach you can do is have apache log to a local named pipe and have a process listen on that named pipe and tagging/reformatting the log file and pass it to your syslog server.

David Lang


Any comments/suggestions?
I am sure others have had a similar need. I just don't want to ditch local log 
files until we fully know how well the centralised log server performs.

Thanks in advance!
Cheers, Ben

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Reply via email to