In message <[email protected]>,
[email protected] writes:
>On Tue, 2 Oct 2012, Joseph Guanzon wrote:
>> Thanks very much for your response, do you know a good way to formulate
>> the required resources for a monitoring server that can handle a large
>> amount of server, polling server (network, CPU, memory, disk space,
>> logs, processes, etc).
>
>sorry, I don't have a good rule of thumb to use for sizing.
>
>network: enough to handle logs without dropping them
The chance of dropping logs can be reduced by using rsyslog or some
other syslog agent that uses tcp (rather than udp) for transport.
Since it will retry to send the logs. Also I would implement a
heartbeat on all my servers to watch for dropped log messages. (Search
the mailing list for an example. HEARTBEAT should be a good search
term.) This can sometimes spot a flakey switch or other network issue.
>disk space: SEC doesn't need any
While that is true, SEC can be fed from a pipe in syslog-ng or
rsyslog, you should probably plan on syslogging to a file to feed
SEC. This allows you to grep for additional info through the file (s)
in addition to info you glean from SEC. Plus you can run analysis
tools over the collected logs to figure out what your most frequent
alert are and to build rules to filter/suppress them.
I would plan on enough disk space to keep at least one weeks worth of
logs so add up all the syslog files written in a week on all your
servers. This can also give you an idea of an upper bound to the
amount of traffic you will ship across the network.
>> Also if I understand you correctly the number of
>> rules made would affect the memory use while the difficulty of the rules
>> may affect the cpu?
>
>not quite
>
>the more state you track, the more memory you need. That said, you can
>probably store a million pieces of state in a gig of memory, so it's not
>likely to be your limiting factor.
Right now I have 66000 contexts with various amounts od content
associated with each and I am using a bit over 261MB of memory.
Some performance statistics:
Run time: 978037 seconds
User time: 439450.53 seconds
System time: 275.49 seconds
Child user time: 50.73 seconds
Child system time: 39.36 seconds
Processed input lines: 49026021
for a set of 161 rules. My input is somewhat spiky but I average 50
events/second with bursts to 900. It usually takes about 5 seconds to
digest the 500 second burst and get back to realtime, so my current
ruleset maxes out at about 200 events/sec for procesing. I have a
rewritten ruleset that I expect will be able to max at 500-600
events/second. So care in writing the rules and reduing the number of
rules that each event has to be considered by is a win.
>the more complex your ruleset, the more CPU you need. This includes more
>rules, and more complex rules. Maintaining state costs CPU (especially
>state that will time out)
Also SEC is single threaded. If you want to use multiple SEC
processes, you can connect them together, but as a rule of thumb SEC
operates better with faster cpu's and doesn't scale as well with more
cpu's.
--
-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.
------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
_______________________________________________
Simple-evcorr-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
