Hi, This is encouraging to know that solr/lucene solution may work. Can anyone using solr/lucene for such scenario can confirm that the solution is used and working fine? That would be really helpful, as I just started looking into the solr/lucene solution only couple of days back and might be difficult to be 100% confident before proposing the solution approach in next couple of days. Thanks,Surfer
--- On Thu, 6/4/09, Otis Gospodnetic <otis_gospodne...@yahoo.com> wrote: From: Otis Gospodnetic <otis_gospodne...@yahoo.com> Subject: Re: Questions regarding IT search solution To: solr-user@lucene.apache.org Date: Thursday, June 4, 2009, 10:26 PM My guess is Solr/Lucene would work. Not sure how well/fast, but it would, esp. if you avoid range queries (or use tdate), and esp. if you shard/segment indices smartly, so that at query time you send (or distribute if you have to) the query to only those shards that have the data (if your query is for a limited time period). Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch ----- Original Message ---- > From: Silent Surfer <silentsurfe...@yahoo.com> > To: solr-user@lucene.apache.org > Sent: Thursday, June 4, 2009 5:52:21 PM > Subject: Re: Questions regarding IT search solution > > Hi, > As Alex correctly pointed out my main intention is to figure out whether > Solr/lucene offer functionalities to replicate what Splunk is doing in terms > of > building indexes etc for enabling search capabilities. > We evaluated Splunk, but it is not very cost effective solution for us as we > may > have logs running into few GBs per day as there can be around 25-20 servers > running, and Splunk licensing model is based of size of logs per day that > too, > the license valid for only 1 year. > With this back ground, any further inputs on this are greatly appreciated. > Thanks,Surfer > > --- On Thu, 6/4/09, Alexandre Rafalovitch wrote: > > From: Alexandre Rafalovitch > Subject: Re: Questions regarding IT search solution > To: solr-user@lucene.apache.org > Date: Thursday, June 4, 2009, 9:27 PM > > I would also be interested to know what other existing solutions exist. > > Splunk's advantage is that it does extraction of the fields with > advanced searching functionality (it has lexers/parsers for multiple > content types). I believe that's the Solr's function desired in > original posting. At the time they came out (2004), I was not aware of > any good open source solutions to do what they did. And I would have > loved one, as I was analyzing multi-gigabite logs. > > Hadoop might be a way to process the files, but what would do the > indexing and searching? > > Regards, > Alex. > > On Thu, Jun 4, 2009 at 11:56 AM, Walter Underwoodwrote: > > Why build one? Don't those already exist? > > > > Personally, I'd start with Hadoop instead of Solr. Putting logs in a > > search index is guaranteed to not scale. People were already trying > > different approaches ten years ago. > > > > wunder > > > > On 6/4/09 8:41 AM, "Silent Surfer" wrote: > > > >> Hi, > >> Any help/pointers on the following message would really help me.. > >> Thanks,Surfer > >> > >> --- On Tue, 6/2/09, Silent Surfer wrote: > >> > >> From: Silent Surfer > >> Subject: Questions regarding IT search solution > >> To: solr-user@lucene.apache.org > >> Date: Tuesday, June 2, 2009, 5:45 PM > >> > >> Hi, > >> I am new to Lucene forum and it is my first question.I need a clarification > >> from you. > >> Requirement:------------------1. Build a IT search tool for logs similar to > >> that of Splunk(Only wrt searching logs but not in terms of reporting, > >> graphs > >> etc) using solr/lucene. The log files are mainly the server logs like JBoss, > >> Custom application server logs (May or may not be log4j logs) and the files > >> size can go potentially upto 100 MB2. The logs are spread across multiple > >> servers (25 to 30 servers)2. Capability to be do search almost realtime3. > >> Support distributed search > >> > >> Our search criterion can be based on a keyword or timestamp or IP address > etc. > >> Can anyone throw some light if solr/lucene is right solution for this ? > >> Appreciate any quick help in this regard. > >> Thanks,Surfer
