HI,
What you say is done by hadoop that support Hardware FailurećData
Replication and some else .
If we want to implement such a good system by ourselves without HDFS
but Solr , it's a very very complex work I think. :)
I just want to know whether there is a component existed can do the
distributed search based on Solr.
Thanks
Jarvis.
-----Original Message-----
From: Norberto Meijome [mailto:[EMAIL PROTECTED]
Sent: Thursday, September 20, 2007 10:06 AM
To: solr-user@lucene.apache.org
Cc: [EMAIL PROTECTED]
Subject: Re: How can i make a distribute search on Solr?
On Thu, 20 Sep 2007 09:37:51 +0800
"Jarvis" <[EMAIL PROTECTED]> wrote:
> If we use the RPC call in nutch .
Hi,
I wasn't suggesting to use nutch in solr...I'm only a young grasshopper in
this
league to be suggesting architecture stuff :) but i imagine there's nothing
wrong with using what they've built if it addresses solr's needs.
> Manually separate the index is required .
hmm i imagine this really depends on the application. In my case, this
separation of which docs go where happens @ a completely different layer.
> We will receive reduplicate result if there is reduplicate index document
on
> different servers.
Maybe I got this wrong...but isn't this what mapreduce is meant to deal
with?
eg,
1) get the job (a query)
2) map it to workers ( servers that provide search results from their own
indexing)
3) wait for the results from all workers that reply within acceptable
timeframe.
4) comb through the lot of results from all workers, reduce them according
to
your own biz rules (eg, remove dupes, sort them by quality / priority...
here possibly relying on the original parameters of the query in 1)
5) return the reduced results to the frontend.
> And also the data updating and single server's error is
> hard to deal with.
this really depends on your infrastructure + design.
Having the indexing , searching and providing of results in different layers
should make for some interesting design options...
If each searcher (or wherever the index resides) is really a small cluster
of
servers , the issue of data safety / server error is addressed @ that point.
You can also have repeated data across indexes (again, independent indexes)
and
that's a more ... randomised :) way of keeping the docs safe... For example,
IIRC, googleFS keeps copies of each file in 3 servers or more...
cheers,
B
_________________________
{Beto|Norberto|Numard} Meijome
"He uses statistics as a drunken man uses lamp-posts ... for support rather
than illumination." Andrew Lang (1844-1912)
I speak for myself, not my employer. Contents may be hot. Slippery when wet.
Reading disclaimers makes you go blind. Writing them is worse. You have been
Warned.
