Hey,
Your system sounds similar to the work don by Stu Hood at Rackspace in their
Mailtrust unit. See
http://highscalability.com/how-rackspace-now-uses-mapreduce-and-hadoop-query-terabytes-datafor
more details and inspiration.
Regards,
Jeff
On Thu, Jun 4, 2009 at 4:58 PM, silentsurfe...@yahoo.com wrote:
Hi,
This is encouraging to know that solr/lucene solution may work.
Can anyone using solr/lucene for such scenario can confirm that the
solution is used and working fine? That would be really helpful, as I just
started looking into the solr/lucene solution only couple of days back and
might be difficult to be 100% confident before proposing the solution
approach in next couple of days.
Thanks,Surfer
--- On Thu, 6/4/09, Otis Gospodnetic otis_gospodne...@yahoo.com wrote:
From: Otis Gospodnetic otis_gospodne...@yahoo.com
Subject: Re: Questions regarding IT search solution
To:
solr-user@lucene.apache.org
Date: Thursday, June 4, 2009, 10:26 PM
My guess is Solr/Lucene would work. Not sure how well/fast, but it would,
esp. if you avoid range queries (or use tdate), and esp. if you
shard/segment indices smartly, so that at query time you send (or distribute
if you have to) the query to only those shards that have the data (if your
query is for a limited time period).
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
- Original Message
From: Silent Surfer silentsurfe...@yahoo.com
To: solr-user@lucene.apache.org
Sent: Thursday, June 4, 2009 5:52:21 PM
Subject: Re:
Questions regarding IT search solution
Hi,
As Alex correctly pointed out my main intention is to figure out whether
Solr/lucene offer functionalities to replicate what Splunk is doing in
terms of
building indexes etc for enabling search capabilities.
We evaluated Splunk, but it is not very cost effective solution for us as
we may
have logs running into few GBs per day as there can be around 25-20
servers
running, and Splunk licensing model is based of size of logs per day that
too,
the license valid for only 1 year.
With this back ground, any further inputs on this are greatly
appreciated.
Thanks,Surfer
--- On Thu, 6/4/09, Alexandre Rafalovitch wrote:
From: Alexandre Rafalovitch
Subject: Re: Questions regarding IT search solution
To: solr-user@lucene.apache.org
Date: Thursday, June 4, 2009, 9:27 PM
I would also be interested to know what other existing solutions exist.
Splunk's advantage is that it does extraction of the fields with
advanced searching functionality (it has lexers/parsers for multiple
content types). I believe that's the Solr's function desired in
original posting. At the time they came out (2004), I was not aware of
any good open source solutions to do what they did. And I would have
loved one, as I was analyzing multi-gigabite logs.
Hadoop might be a way to process the files, but what would do the
indexing and searching?
Regards,
Alex.
On Thu, Jun 4, 2009 at 11:56 AM, Walter Underwoodwrote:
Why build one? Don't those already exist?
Personally, I'd start with Hadoop instead of Solr. Putting
logs in a
search index is guaranteed to not scale. People were already trying
different approaches ten years ago.
wunder
On 6/4/09 8:41 AM, Silent Surfer wrote:
Hi,
Any help/pointers on the following message would really help me..
Thanks,Surfer
--- On Tue, 6/2/09, Silent Surfer wrote:
From: Silent Surfer
Subject: Questions regarding IT search solution
To: solr-user@lucene.apache.org
Date: Tuesday, June 2, 2009, 5:45 PM
Hi,
I am new to Lucene forum and it is my first question.I need a
clarification
from you.
Requirement:--1. Build a IT search tool for logs
similar to
that of Splunk(Only wrt searching logs but not in terms of reporting,
graphs
etc) using
solr/lucene. The log files are mainly the server logs like JBoss,
Custom application server logs (May or may not be log4j logs) and the
files
size can go potentially upto 100 MB2. The logs are spread across
multiple
servers (25 to 30 servers)2. Capability to be do search almost
realtime3.
Support distributed search
Our search criterion can be based on a keyword or timestamp or IP
address
etc.
Can anyone throw some light if solr/lucene is right solution for this
?
Appreciate any quick help in this regard.
Thanks,Surfer
