Hi,
As Alex correctly pointed out my main intention is to figure out whether
Solr/lucene offer functionalities to replicate what Splunk is doing in terms of
building indexes etc for enabling search capabilities.
We evaluated Splunk, but it is not very cost effective solution for us as we
may have logs running into few GBs per day as there can be around 25-20 servers
running, and Splunk licensing model is based of size of logs per day that too,
the license valid for only 1 year.
With this back ground, any further inputs on this are greatly appreciated.
Thanks,Surfer
--- On Thu, 6/4/09, Alexandre Rafalovitch <arafa...@gmail.com> wrote:
From: Alexandre Rafalovitch <arafa...@gmail.com>
Subject: Re: Questions regarding IT search solution
To: solr-user@lucene.apache.org
Date: Thursday, June 4, 2009, 9:27 PM
I would also be interested to know what other existing solutions exist.
Splunk's advantage is that it does extraction of the fields with
advanced searching functionality (it has lexers/parsers for multiple
content types). I believe that's the Solr's function desired in
original posting. At the time they came out (2004), I was not aware of
any good open source solutions to do what they did. And I would have
loved one, as I was analyzing multi-gigabite logs.
Hadoop might be a way to process the files, but what would do the
indexing and searching?
Regards,
Alex.
On Thu, Jun 4, 2009 at 11:56 AM, Walter Underwood<wunderw...@netflix.com> wrote:
> Why build one? Don't those already exist?
>
> Personally, I'd start with Hadoop instead of Solr. Putting logs in a
> search index is guaranteed to not scale. People were already trying
> different approaches ten years ago.
>
> wunder
>
> On 6/4/09 8:41 AM, "Silent Surfer" <silentsurfe...@yahoo.com> wrote:
>
>> Hi,
>> Any help/pointers on the following message would really help me..
>> Thanks,Surfer
>>
>> --- On Tue, 6/2/09, Silent Surfer <silentsurfe...@yahoo.com> wrote:
>>
>> From: Silent Surfer <silentsurfe...@yahoo.com>
>> Subject: Questions regarding IT search solution
>> To: solr-user@lucene.apache.org
>> Date: Tuesday, June 2, 2009, 5:45 PM
>>
>> Hi,
>> I am new to Lucene forum and it is my first question.I need a clarification
>> from you.
>> Requirement:------------------1. Build a IT search tool for logs similar to
>> that of Splunk(Only wrt searching logs but not in terms of reporting, graphs
>> etc) using solr/lucene. The log files are mainly the server logs like JBoss,
>> Custom application server logs (May or may not be log4j logs) and the files
>> size can go potentially upto 100 MB2. The logs are spread across multiple
>> servers (25 to 30 servers)2. Capability to be do search almost realtime3.
>> Support distributed search
>>
>> Our search criterion can be based on a keyword or timestamp or IP address
>> etc.
>> Can anyone throw some light if solr/lucene is right solution for this ?
>> Appreciate any quick help in this regard.
>> Thanks,Surfer
