Risto, The numerous types of logs currently needing a single probe are created and written to by log4j, so as you noted to consolidate the many logs into a single file, I could configure log4j to write to a single log file. Or we could also publish disparate events to a single JMS queue/topic. I can then monitor a single data stream rather than deal with monitoring so many log files. The problem is that some folks prefer to have a separate log file for the many different parts of the application. We'll figure it out. Logpp still may be a pre-processing monitoring option, if we leave it as is. We'll see how it ends up.
As for using logpp with multiple ssh tail files, we're not using syslog or any other logger like log4j for those logs. And the files are located across several servers too. I think this is a common problem where people often pull up multiple xterms and tail -f separate files "grepping" for specific sets of messages. I was curious if logpp could support input from something other than files for a simple central "grep" of sorts. If it turned out to be robust, it could then feed SEC to generate notifications as problems are detected. Thanks for the feedback. Regards, Rock -----Original Message----- From: Risto Vaarandi [mailto:rvaara...@yahoo.com] Sent: Thursday, April 23, 2009 3:23 PM To: simple-evcorr-users@lists.sourceforge.net; Mills, Rocky Subject: Re: [Simple-evcorr-users] Logpp and SEC input sources > From: Mills, Rocky <rx4...@att.com> > Subject: [Simple-evcorr-users] Logpp and SEC input sources > To: simple-evcorr-users@lists.sourceforge.net > Date: Wednesday, April 22, 2009, 2:23 AM > Risto, Anyone, > > I was considering counting various string matches using SEC > across > numerous (over potentially 20) logs simultaneously. > There should be few > string matches, but when combined at peak times there could > be perhaps > several thousand extraneous lines to parse per > second. SEC may be able > to handle it but I recall you noting another utility > "logpp". I > reviewed its man page and it seems to be a good fit to more > efficiently > trim the logs before evaluating them with SEC. > > >From the logpp output I need to know its input > source. I haven't tried > it but I'm thinking there is no glob-like loading of a > dynamic list of > log files via logpp config like this: > > input app-log-input { > file /app/log/*.log > } > > So I'm thinking I could build the input part of my logpp > config file > dynamically (before logpp startup) and then load it with > each file > explicitly specified like this: > > input app-logs-input { > file /app/log/A.log > file /app/log/B.log > and so on... > } > > My filter's template could then prefix each log line with > the filename > like this: > > filter app-logs-filter { > regexp something X > regexp another thing Y > template filename $~: $0 > } > > Where $~ is the filename and $0 is the log's line of text. > > I could then use SEC to extract the filename from each line > and setup my > counts and such using it. > > Any other solution you'd recommend? how are these log files created? If they are created by syslogd/syslog-ng, then maybe you could set up a single file instead of many, and process this file with logpp. > > > Another logpp question (with a similar need for SEC to > determine the > input sources): > > Any preferred/simple way to setup logpp to read multiple > ssh inputs? > For example, could I configure input from "ssh m...@serverx > tail -f > /app/log/X.log" and "ssh m...@servery tail -f /app/log/X.log" > to be read > by logpp? Actually I have potentially 12 separate hosts > with ssh inputs > I'd like to gather into a single event stream to feed SEC > such that SEC > can also extract the input source (hostname in this > case). This is not > a high volume scenario. Just curious if logpp could > easily consolidate > distributed logs that could be monitored from a central > location. Have you thought about another scenario -- logpp can also convert non-syslog logs into syslog format, and you could have logpp running on 12 hosts for sending input events to central host with syslog protocol. If you need to encrypt the data exchange, then you could use ssh/stunnel for that. It might be somewhat more complex to implement, but the events will be converted to syslog format early on and you have the flexibility that comes with syslog-style logging. br, risto > > > Regards, > Rock > > > > > ***** > > The information transmitted is intended only for the person > or entity to which it is addressed and may contain > confidential, proprietary, and/or privileged material. Any > review, retransmission, dissemination or other use of, or > taking of any action in reliance upon this information by > persons or entities other than the intended recipient is > prohibited. If you received this in error, please contact > the sender and delete the material from all computers. > GA625 > > > > ------------------------------------------------------------------------------ > Stay on top of everything new and different, both inside > and > around Java (TM) technology - register by April 22, and > save > $200 on the JavaOne (SM) conference, June 2-5, 2009, San > Francisco. > 300 plus technical and hands-on sessions. Register today. > Use priority code J9JMT32. http://p.sf.net/sfu/p > _______________________________________________ > Simple-evcorr-users mailing list > Simple-evcorr-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users > ***** The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential, proprietary, and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from all computers. GA622 ------------------------------------------------------------------------------ Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensign option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects _______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
