Thanks Alex! Couple of questions:
>Nobody is specifically assigned to the task, the severity of the
>problem determines the response - generally speaking it just creates a
>ticket which someone on support rotation can take action upon, really
>urgent stuff triggers SMS messages.
I assume by severity you mean log mnemonic severity, correct? (like
sys-5-config,etc.)
What sev levels do you automatically create tickets on?
How are messages determined to be "really urgent"?
On 10/29/07, Alex Howells <[EMAIL PROTECTED]> wrote:
>
> Hi Clayton
>
> > I need input on:
> > 1. Number of devices you are managing logs for (large scale being over
> > 10,000 devices)
>
> We're currently managing logs for ~200 network devices and another
> 500-600 servers using syslog-ng; perhaps not big enough, but who knows
> ;)
>
> > 2. What log levels you are sending from the devices ( i.e. 0-6 for
> normal
> > operation, 0-7 when troubleshooting?)
>
> We have a central loghost in each physical site which accepts and
> queues logs, forwarding them onto our master box as long as it's
> available.
>
> On the master box, I have a piece of Ruby which filters out common
> garbage we're unable to remove; mostly this is legacy software which
> logs far too much to inappropriate levels and we don't care about much
> any more.
>
> In normal operating mode, we actively track 0-5; when we're tracing a
> problem you can flip the mangler into a more verbose mode where it
> inserts 0-6 into MySQL -- typically we don't look at the DEBUG level
> at all, since if you're working with an application in that much depth
> you might as well configure it to log locally for a while :)
>
> > 3. What log levels you are reacting on (if not all).
>
> Stated above.
>
> > 4. How many people are assigned to look at log messages
>
> Nobody is specifically assigned to the task, the severity of the
> problem determines the response - generally speaking it just creates a
> ticket which someone on support rotation can take action upon, really
> urgent stuff triggers SMS messages.
>
> > 5. What program(s) are used to do log analysis
>
> Bespoke bits and pieces written in Ruby. We have a very interesting
> 'attack analysis' module in development which scans for common
> dictionary-based attacks and, subject to certain conditions being met,
> should be able to null route persistent buggers.
>
> > 6. How are you analyzing the logs? Are you doing a baseline analysis
> (based
> > on number of events per device) or are you reacting on every incoming
> > message...or do you just ignore them because there are too many to look
> at?
>
> Baseline analysis to detect host downtime, ie: we specify a minimum
> number of INSERT queries per second we expect to see generated by each
> host.
>
> Apart from that, messages are filtered and classified after insertion.
>
> > 7. Anything I missed?
>
> Don't think so :) I suppose my only other comment is that we mostly
> use syslog-ng and PHP-Syslog-NG in a reactive fashion, to track down
> the cause of problems and assess the extent of any potential issue -
> active monitoring is handled by Nagios.
>
> Best Regards,
> Alex
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems? Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> Php-syslog-ng-support mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/php-syslog-ng-support
>
--
______________________________________________________________
Clayton Dukes
______________________________________________________________
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Php-syslog-ng-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/php-syslog-ng-support
