On Tue, 2 Oct 2012, John P. Rouillard wrote: > In message <[email protected]>, > [email protected] writes: >> On Tue, 2 Oct 2012, Joseph Guanzon wrote: >>> Thanks very much for your response, do you know a good way to formulate >>> the required resources for a monitoring server that can handle a large >>> amount of server, polling server (network, CPU, memory, disk space, >>> logs, processes, etc). >> >> sorry, I don't have a good rule of thumb to use for sizing. >> >> network: enough to handle logs without dropping them > > The chance of dropping logs can be reduced by using rsyslog or some > other syslog agent that uses tcp (rather than udp) for transport. > Since it will retry to send the logs. Also I would implement a > heartbeat on all my servers to watch for dropped log messages. (Search > the mailing list for an example. HEARTBEAT should be a good search > term.) This can sometimes spot a flakey switch or other network issue.
If the network is fast enough, you can get pretty good speeds with UDP syslog without dropping packets. I've tested up to 380K logs/sec (250 byte logs, which works out to approximately gig-E saturated network speeds) without loosing any logs. the point is that the network is not likly to be significant, but if your network is busy, TCP syslog (or RELP with rsyslog) can help you. Keep in mind that this pushes the backlog out to the clients, and if they get too backlogged they can stop processing your production workload. >> disk space: SEC doesn't need any > > While that is true, SEC can be fed from a pipe in syslog-ng or > rsyslog, you should probably plan on syslogging to a file to feed > SEC. This allows you to grep for additional info through the file (s) > in addition to info you glean from SEC. Plus you can run analysis > tools over the collected logs to figure out what your most frequent > alert are and to build rules to filter/suppress them. > > I would plan on enough disk space to keep at least one weeks worth of > logs so add up all the syslog files written in a week on all your > servers. This can also give you an idea of an upper bound to the > amount of traffic you will ship across the network. good point, I have a large enough scale of logs that I tend to think in terms of SEC on one set of boxes, and my log archiving/reporting on a different set of boxes. SEC is great for alerts, but it's not good for reports. I have a separate server that archives all the logs, but also uses rsyslog configs to split different types of logs into different files. I then have a batch of scripts that runs every hour on the last hour's logs (and every night on the last day's logs). I don't want that log processing to interfere with the SEC alerting. This is a real risk since the nightly log processing run can peg the box for a couple of hours. the way I argue for minimum log retention is that you need to have enough logs so that after a long (say 4day thanksgiving) weekend, you can come back and a day or two after you get back you can go back to "something strange happened on wednesday when everyone left early, we need to find out what" so a week is a good, practical minimum. David Lang ------------------------------------------------------------------------------ Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev _______________________________________________ Simple-evcorr-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
