On Mon, 24 Jan 2011, Leon Towns-von Stauber wrote: > On Jan 24, 2011, at 6:51 AM, Jeremy Charles wrote: > >> Most of the log analysis/correlation threads in my E-mail archive seem to >> get a lot of "yeah we use Splunk for that." Allow me to ask the question >> this way and see what happens... >> >> I pressed my coworkers for some specific examples of what they'd like to >> have a log correlator do for them. Below is the non-trivial example that I >> received. Do any log correlation products have this sort of sophistication? >> >> >> The following sequence of events happen. Program sees the log entry for each >> event, makes the correlation, and sends us an email that [insert user here] >> is running bittorrent. >> Link comes up on switch port >> 802.1x authentication succeeds >> DHCP address issued >> User logs into domain >> Firewall starts logging the BitTorrent URL signature > > I would imagine most any log correlation tool could do that. I know > that SEC can, although you might have to use a series of contexts if > the sequence is important. (That is, see the first message, set the > first context. If the second message is seen while the first context > is active, set the second context. Etc.)
I'll second the SEC suggestion. I have both SEC and Splunk in my system, and I prefer to use each where they do a good job. Splunk is great for doing ad-hoc searches through your logs (investigations, troubleshooting, etc), but having it do event correlation is really inefficient. What you do for 'event correlation' with splunk is to define a search and run it repeatedly. if you have enough logs that your search time is larger than what will fit in ram, this will really hammer your I/O system (on multiple systems if you have scaled your splunk install). I have a 20 machine cluster running splunk (each with 8 cores, 64 G ram and 16 drives), doing a search on splunk ties up this entire cluster for the time of the search I have a single box running SEC watching the same flow of logs. Searching a 5 min window on splunk only takes a few seconds, but a few seconds per conditition that you are searching for starts adding up fast, and once you have a few dozen conditions that you are looking for on a frequent (say 1 min) basis, you can find that the splunk cluster is always busy (and therefor will not respond as fast when you need to do a search manually), beyond that everything starts slowing down more. I currently have SEC looking for a few hundred events and combinations of conditions (with the expectation that I will have many more in the future) and it's loafing along Splunk is expensive, and the hardware to run it well with a large amount of data (both high volume and long retention period) is also very expensive. I wouldn't willingly eliminate the capabilities that it gives me, but I also do not want to eat up that cluster's capacity with things that are better done elsewhere. David Lang _______________________________________________ Tech mailing list Tech@lists.lopsa.org https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech This list provided by the League of Professional System Administrators http://lopsa.org/
