I would also be interested to know what other existing solutions exist.
Splunk's advantage is that it does extraction of the fields with
advanced searching functionality (it has lexers/parsers for multiple
content types). I believe that's the Solr's function desired in
original posting. At the time they came out (2004), I was not aware of
any good open source solutions to do what they did. And I would have
loved one, as I was analyzing multi-gigabite logs.
Hadoop might be a way to process the files, but what would do the
indexing and searching?
Regards,
Alex.
On Thu, Jun 4, 2009 at 11:56 AM, Walter Underwood<wunderw...@netflix.com> wrote:
> Why build one? Don't those already exist?
>
> Personally, I'd start with Hadoop instead of Solr. Putting logs in a
> search index is guaranteed to not scale. People were already trying
> different approaches ten years ago.
>
> wunder
>
> On 6/4/09 8:41 AM, "Silent Surfer" <silentsurfe...@yahoo.com> wrote:
>
>> Hi,
>> Any help/pointers on the following message would really help me..
>> Thanks,Surfer
>>
>> --- On Tue, 6/2/09, Silent Surfer <silentsurfe...@yahoo.com> wrote:
>>
>> From: Silent Surfer <silentsurfe...@yahoo.com>
>> Subject: Questions regarding IT search solution
>> To: solr-user@lucene.apache.org
>> Date: Tuesday, June 2, 2009, 5:45 PM
>>
>> Hi,
>> I am new to Lucene forum and it is my first question.I need a clarification
>> from you.
>> Requirement:------------------1. Build a IT search tool for logs similar to
>> that of Splunk(Only wrt searching logs but not in terms of reporting, graphs
>> etc) using solr/lucene. The log files are mainly the server logs like JBoss,
>> Custom application server logs (May or may not be log4j logs) and the files
>> size can go potentially upto 100 MB2. The logs are spread across multiple
>> servers (25 to 30 servers)2. Capability to be do search almost realtime3.
>> Support distributed search
>>
>> Our search criterion can be based on a keyword or timestamp or IP address
>> etc.
>> Can anyone throw some light if solr/lucene is right solution for this ?
>> Appreciate any quick help in this regard.
>> Thanks,Surfer
