On Jun 18, 2007, at 11:26 PM, Paul Smith wrote:
Hi,
I have been in the background working on a companion to log4j that
I'm initially calling 'pinpoint'. It is designed as a log context
tool to facilitate production queries.
Logging events to a DB for querying is fine, except SQL queries,
IHMO, on that are time consuming and unsatisfying, and miss some
important features such as full text indexing. I find for my own
use I tend to rely on grep/fgrep to analyse logs files because the
regexp is handy, but that often trims off important stack traces.
Usually you have to dice the file several times to work with it.
It seemed to make sense to index these log files and be able to
analyse them using Lucene.
pinpoint is designed to create a Context which is a container of
indexed LoggingEvents. The Context contains an Lucene index, and
a serialized version of the events. searches against the index
then return hits that contain offset+length pointers to the
serialization file. That way the full LoggingEvent can be brought
back for display in the results, formatted anyway you'd like.
Using a CLI client, one can open that index and do arbitrary queries.
An example is worth a thousand words:
$ sh pinpoint-cli/target/appassembler/bin/pinpoint -h /temp/
pinpoint-home/ -c testlog -s
pinpoint>search level:INFO factory -english
+message:factory -message:english
[2007-06-08 12:13:00,122 INFO ][eNotificationsAction][Maintenance
Tasks Scheduler_Worker-9] Executing
OverdueCorrespondenceNotificationsAction: exclusionDate=2006-06-08,
maxEmailSendRate=240, maxNumEmails=2147483647, sendEmailEnabled=TRUE
[2007-06-08 12:13:00,553 ERROR][urityInfoRetrieveCmd][Maintenance
Tasks Scheduler_Worker-9][][] Could not find class for command
factory. Class com.aconex.security.command.SecuredAssetListCmd
.......
pinpoint>
Only the basics are there right now, with a simple importer utility
that uses LogFilePatternReceiver to soak events from text-based log
files, but obviously eventually any Receiver might be usable to
soak events into a context. The CLI module has handy command-
line completion support, but it could definitely do with some
extensions.
Pinpoint is made up of:
* pinpoint-core
* pinpoint-importer
* pinpoint-search
* pinpoint-cli
and lives in a maven multi-project module setup. It relies on the
log4j companion 1.0-SNAPSHOTs and log4j 1.2.15-SNAPSHOT.
I'd like to eventually add a pinpoint-service module allowing
standard socket/JMS-based receivers to be configured out of the box
to allow easy connection to it as a remote service. Right now
indexing the logs is likely to be slower than the rate of incoming
events, so we'd need to build in some asynch behaviour to allow
pinpoint to buffer some logs (say via an internal JMS
implementation using ActiveMQ) as it processes them. This allows
the producing application to not be slowed down by the indexing.
I'd also like to create pinpoint-webapp to create a nice and easy
deployable webapp than can browse the pinpoint contexts and easily
search logs based on MDC variables etc. (Imagine browsing all logs
generated by a User when using, say, MDC), and once found an
interesting logging event quickly see more context lines from
around that time.
I'd be happy to have this added as a log4j module if other people
are interested. Otherwise I'm just as happy to tinker with it and
use it for my own use. Perhaps I'm the only one that wants to use
logging this way?
cheers,
Paul
If this was a project proposal, I'd have no issue with you starting a
sandbox project and allow it to develop out in the open. However,
based on your discussion it appears to be pretty far along and it
looks like it should be at least reviewed by the Incubator PMC before
going into Logging Services. I'd suggest putting it in Apache Labs.
If it gains traction in the Labs, then we can migrate it over to LS
with appropriate Incubator PMC oversight.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
