Hello Robert,
At my company, we are working on a generic log collector that uses Solr
to provide search capabilities.
What the collector does basically is this (this is greatly dumbed down !) :
* collect a log line (read it from a file, receive it from the network,
... )
* parse it through a set of regular expressions, searching for known log
formats ( apache CLF, ... )
* if there is a match, store the results as a set of keys/values ( url :
http://www.apache.org , source : XXX , raw_log : xxxxxx , ... )
* insert the set as a document in the Solr backend, using the REST
interface.
Therefore I would advise you to adapt this workflow to suit your own
needs : have a script looking for new lines in your log file, parse them
in order to extract the relevant information you need, store the results
as keys/values sets, then insert them into Solr via a http call.
My company's product is probably overkill for what you need to do, and
we'd probably need to develop a specific log parser for your log format,
but if you are willing to give it a try feel free to contact me !
Greetings,
Matthieu HUIN
On 06/05/2011 21:40, Robert Naczinski wrote:
Hi,
thanks for the reply. I did not know that.
Is there still a way to use Solr or Lucene? Or Apache Nutch would be not be bad.
Could I maybe write a customized DIH?
Greetings,
Robert
2011/5/6 Otis Gospodnetic<otis_gospodne...@yahoo.com>:
Loggly.com
