On 15/10/2008, at 7:37 AM, Chris Gilliam wrote:
Hello Everyone,
New to Lucene..
We currently roughly 100Gig of log files. We are needing to build
a search
application that can return rows of data from the files and combine
the
results?
Does Lucene index the content in the files?
Will it be able to find matching criteria say a date and then
return the
next X number of lines?
The indexing piece, how does it work if I am looking for something
that has
been added to the file since the indexing has taken place?
I'm working on just such a thing over on my Apache Lab project called
"Pinpoint":
http://svn.apache.org/repos/asf/labs/pinpoint/trunk/
It is designed to process log4j environments, soaking from network-
based appenders, of by importing log files based on a pattern. It's
relatively early days but the pinpoint-service layer recently ran
through a production load test environment soaking 500-1000 events
per/second no problems at all.
The pinpoint-search layer provides an API to query a
'context' (usually a days worth of logs) to find matching events, and
then to be able to view the temporal context of specific matching
events so you can see them inline with their related logs (either by
the Thread that created the original matching event, +/-10 minutes,
or by showing all logging events around that time +/- 10 minutes).
Really good for production triage. I'm trying to work on a Web-based
app that allows a dynamic sort-of-OLAP analysis of log files (charts,
trends, etc).
cheers,
Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
