On Tue, 7 Sep 2010, Joe Williams wrote:
On Sep 7, 2010, at 11:11 AM, Aaron Wiebe wrote:
On Tue, Sep 7, 2010 at 2:07 PM, <[email protected]> wrote:
1. stats that are cheap to gather, generate
examples current length of queues, age of oldest message in each queue,
number of messages processed (possibly even per output)
2. stats that are expensive to gather or generate
examples: average length of queues, average time to process messages
Expensive stats shouldn't be the responsibility of rsyslog though -
trending over time and the like should be the responsibility of a
third party product (ie, cacti/mrtg). As long as the stats are easily
made available through snmp or other methods, the trending should be
done outside rsyslog.
Agreed, rsyslog should just report the current stats and any
trending/analysis should be done externally. No different than running
iostat or other tools.
iostat and vmstat actually provide some of these expensive stats.
an example of an expensive stat would be the average age of messages in
the queue.
finding the age of the oldest message is relativly cheap (one lookup to
the oldest message in the queue, that lookup may delay something
processing the message so it's only 'relativly' cheap)
but finding the average age of all messages in the queue will require
looking up the age of every message in the queue, and probably freezing
the queue while you do so to make sure that it's not changing under you.
One other timing stat I would love to see is the amount of time in each
step of the pipeline (time to read the message from the socket, time
processing it through any rules, time spent in any queues, time spent
writing to disk, etc). Haproxy has a good example of this in it's
logging (8.2.3. HTTP log format -
http://haproxy.1wt.eu/download/1.4/doc/configuration.txt).
message processing in rsyslog is done at such a high rate that you can get
a noticable performance improvement by doing one gettimeofday call for
every 10 messages instead of one call per message. looking up the time at
different stages within rsyslog definantly qualifies as an expensive stat
to gather.
cheap things to gather are things that can be gathered by modifying a
per-thread variable.
relativly cheap things to gather are things that require locking to gather
(either to update a central variable, or to prevent other threads from
modifying what you are reading)
expensive things to gather include things that require syscalls
(for example, gettimeofday()), or that require looking at a lot of data
(average age of all messages in a queue)
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
