You're right. Currently clients need to take care of this, in this case, Nutch would be the client but it cannot be configured as such. It would, indeed, be more appropriate for Solr to take care of this. We can already query any server with a set of shard hosts specified, so it would make sense if Solr also supported some kind of consistent hashing and shard management configuration.
With CouchDB-Lounge we can easily create a shard map that supports redundant shards on different servers for fail-over. It would be marvelous if Solr would support it as well. -----Original message----- From: Otis Gospodnetic <otis_gospodne...@yahoo.com> Sent: Wed 16-06-2010 21:41 To: solr-user@lucene.apache.org; Subject: Re: Re: Solr and Nutch/Droids - to use or not to use? Well, it's not that Nutch doesn't support it. Solr itself doesn't support it. Indexing applications need to know which shard they want to send documents to. This may be a good case for a new wish issue in Solr JIRA? Otis ---- Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ ----- Original Message ---- > From: Markus Jelsma <markus.jel...@buyways.nl> > To: solr-user@lucene.apache.org > Sent: Wed, June 16, 2010 3:31:49 PM > Subject: RE: Re: Solr and Nutch/Droids - to use or not to use? > > Nutch does not, at this moment, support some form of consistent hashing to > select an appropriate shard. It would be nice if someone could file an issue > in > Nutch' Jira to add sharding support to it, perhaps someone with a better > understanding and more experience with Solr's distributed search than i have > at > the moment. I can't point Nutch' developers to the right piece of > documentation > on this one ;) -----Original message----- From: Otis Gospodnetic > < > href="mailto:otis_gospodne...@yahoo.com";>otis_gospodne...@yahoo.com> Sent: > Wed 16-06-2010 21:03 To: > href="mailto:solr-user@lucene.apache.org";>solr-user@lucene.apache.org; > Subject: Re: Solr and Nutch/Droids - to use or not to use? Hi > Mitch, Solr can do distributed search, so it can definitely handle > indices that can't fit on a single server without sharding. What I think > *might* be the case that the Nutch indexer that sends docs to Solr might not > be > capable of sending documents to multiple Solr cores/shards. If that is the > case, I think you need to move this to the Nutch user/dev list and see how to > feed multiple Solr indices/cores/shards with Nutch > data. Otis ---- Sematext :: > target=_blank >http://sematext.com/ :: Solr - Lucene - Nutch Lucene > ecosystem search :: http://search-lucene.com/ > ----- Original Message ---- > From: MitchK < > ymailto="mailto:mitc...@web.de"; > href="mailto:mitc...@web.de";>mitc...@web.de> > To: > ymailto="mailto:solr-user@lucene.apache.org"; > href="mailto:solr-user@lucene.apache.org";>solr-user@lucene.apache.org > > Sent: Wed, June 16, 2010 2:27:16 PM > Subject: Re: Solr and Nutch/Droids - > to use or not to use? > > Thanks, that really helps to find the > right beginning for such a journey. > :-) > * Use Solr, > not Nutch's search webapp > As > far as I have read, Solr > can't scale, if the index gets too large for > > one Server > The setup explained here has one significant > > caveat you also need to keep > in mind: scale. You cannot use > this kind of > setup with vertical scale > (collection size) that > goes beyond one Solr > box. The horizontal scaling > (query > throughput) is still possible with > the standard Solr > replication > tools. > ...from > > Lucidimagination.com Is this still the case? Furthermore, as far as I > > have understood this blogpost: > href=" > href="http://www.lucidimagination.com/blog/2009/03/09/nutch-solr/"; > target=_blank > >http://www.lucidimagination.com/blog/2009/03/09/nutch-solr/"; target=_blank > > > > href="http://www.lucidimagination.com/blog/2009/03/09/nutch-solr/ > " target=_blank >http://www.lucidimagination.com/blog/2009/03/09/nutch-solr/ > Lucidimagination.com > : Nutch and Solr , they index the whole > stuff with nutch and reindex it to > Solr - sounds like a lot of > redundant work. Lucid, Sematext and the > Nutch-wiki are the only > information-sources where I can find talks about > Nutch and Solr, but > no one seems to talk about these facts - except this one > > blogpost. If you say this is wrong or contingent on the shown setup, can > > you tell me how to avoid these problems? A lot of questions, > but it's > such an exciting topic... Hopefully you can answer some > of > them. Again, thank you for the feedback, Otis. - > Mitch -- > View this message in context: > href=" > href="http://lucene.472066.n3.nabble.com/Solr-and-Nutch-Droids-to-use-or-not-to-use-tp900069p900604.html"; > > target=_blank > >http://lucene.472066.n3.nabble.com/Solr-and-Nutch-Droids-to-use-or-not-to-use-tp900069p900604.html"; > > > > target=_blank > > > href="http://lucene.472066.n3.nabble.com/Solr-and-Nutch-Droids-to-use-or-not-to-use-tp900069p900604.html > " target=_blank > >http://lucene.472066.n3.nabble.com/Solr-and-Nutch-Droids-to-use-or-not-to-use-tp900069p900604.html > Sent > from the Solr - User mailing list archive at > Nabble.com.
