Hi Otis, I followed your recommendation and decided to implement the SearchComponent::modifyRequest(ResponseBuilder rb, SearchComponent who, ShardRequest sreq) method, where the query routing happens. So far it is working OK for the non-facet search, this is good news. The bad news is that it fails on the facet search.
This is how request modification happens: [code_snippet, SearchComponent::modifyRequest] SolrQueryRequest req_routed = rb.req; req_routed = routeRequest(req_routed); rb.req = req_routed; sreq.shards = shards.toString().split(","); [/code_snippet] where shards is StringBuilder, that accumulates the shards the request should go to. req_routed also contains the target shards. Those are set like this: [code_snippet, my function routeRequest(SolrQueryRequest req)] // could not find clone(), used ref reassignment SolrQueryRequest req_local = req; ModifiableSolrParams params = new ModifiableSolrParams(req_local.getParams()); ... params.remove(ShardParams.SHARDS); params.set(ShardParams.SHARDS, getShardsParams(yearToQuarterMap)); params.remove(ShardParams.IS_SHARD); params.set(ShardParams.IS_SHARD, true); req_local.setParams(params); ... return req_local; [/code_snippet] The NPE happens down the road during the facet search, in the FacetComponent::countFacets(), the cause of which is that OpenBitSet obs is null for shardNum=0. Do you have any idea why this happens, should some other field of ResponseBuilder, SearchComponent or ShardRequest be changed? BTW, I have tried to call FacetInfo::parse method inside FacetComponent::modifyRequest() and countFacets(). Where do the fi.facets.values() get initiated, is there some method to call? Thanks, Dmitry On Fri, Jun 3, 2011 at 8:00 PM, Otis Gospodnetic <otis_gospodne...@yahoo.com > wrote: > Nah, if you can quickly figure out which shard a given query maps to, then > all > this component needs to do is stick the appropriate shards param value in > the > request and let the request pass through to the other SearchComponents in > the > chain, including QueryComponent, which will know what to do with the > shards > param. > > Otis > ---- > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch > Lucene ecosystem search :: http://search-lucene.com/ > > > > ----- Original Message ---- > > From: Dmitry Kan <dmitry....@gmail.com> > > To: solr-user@lucene.apache.org > > Sent: Fri, June 3, 2011 12:56:15 PM > > Subject: Re: query routing with shards > > > > Hi Otis, > > > > Thanks! This sounds promising. This custom implementation, will it hurt > in > > any way the stability of the front end SOLR? After implementing it, can > I > > run some tests to verify the stability / performance? > > > > Dmitry > > On Fri, Jun 3, 2011 at 4:49 PM, Otis Gospodnetic < > otis_gospodne...@yahoo.com > > > wrote: > > > > > Hi Dmitry, > > > > > > Yes, you could also implement your own custom SearchComponent. In > this > > > component you could grab the query param, examine the query value, and > > > based on > > > that add the shards URL param with appropriate value, so that when the > > > regular > > > QueryComponent grabs stuff from the request, it has the correct shard > in > > > there > > > already. > > > > > > Otis > > > ---- > > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch > > > Lucene ecosystem search :: http://search-lucene.com/ > > > > > > > > > > > > ----- Original Message ---- > > > > From: Dmitry Kan <dmitry....@gmail.com> > > > > To: solr-user@lucene.apache.org > > > > Sent: Fri, June 3, 2011 2:47:00 AM > > > > Subject: Re: query routing with shards > > > > > > > > Hi Otis, > > > > > > > > I merely followed on the gmail's suggestion to include other people > into > > > the > > > > recipients list, Yonik was the first one :) I won't do it next > time. > > > > > > > > Thanks for a rapid reply. The reason for doing this query routing > is > > > that we > > > > abstract the distributed SOLR from the client code for security > reasons > > > > (that is, we don't want to expose the entire shard farm to the > world, > > > but > > > > only the frontend SOLR) and for better decoupling. > > > > > > > > Is it possible to implement a plugin to SOLR that would map queries > to > > > > shards? > > > > > > > > We have other choices too, they'll take quite some time, that's why > I > > > > decided to quickly ask, if I was missing something from the SOLR > main > > > > components design and configuration. > > > > > > > > Dmitry > > > > > > > > On Fri, Jun 3, 2011 at 8:25 AM, Otis Gospodnetic < > > > otis_gospodne...@yahoo.com > > > > > wrote: > > > > > > > > > Hi Dmitry (you may not want to additionally copy Yonik, he's > > > subscribed to > > > > > this > > > > > list, too) > > > > > > > > > > > > > > > It sounds like you have the knowledge of which query maps to > which > > > shard. > > > > > If > > > > > so, why not control/change the value of "shards" param in the > request > > > to > > > > > your > > > > > front-end Solr (aka distributed request dispatcher) within your > app, > > > which > > > > > is > > > > > the one calling Solr? > > > > > > > > > > Otis > > > > > ---- > > > > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch > > > > > Lucene ecosystem search :: http://search-lucene.com/ > > > > > > > > > > > > > > > > > > > > ----- Original Message ---- > > > > > > From: Dmitry Kan <dmitry....@gmail.com> > > > > > > To: solr-user@lucene.apache.org; yo...@lucidimagination.com > > > > > > Sent: Thu, June 2, 2011 7:00:53 AM > > > > > > Subject: query routing with shards > > > > > > > > > > > > Hello all, > > > > > > > > > > > > We have currently several pretty fat logically isolated shards > with > > > the > > > > > same > > > > > > schema / solrconfig (indices are separate). We currently have > one > > > single > > > > > > front end SOLR (1.4) for the client code calls. Since a client > code > > > > > query > > > > > > usually hits only one shard, we are considering making a smart > > > routing > > > > > of > > > > > > queries to the shards they map to. Can you please give some > > > pointers as > > > > > to > > > > > > what would be an optimal way to achieve such a routing inside > the > > > front > > > > > end > > > > > > solr? Is there a way to configure mapping inside the > solrconfig? > > > > > > > > > > > > Thanks. > > > > > > > > > > > > -- > > > > > > Regards, > > > > > > > > > > > > Dmitry Kan > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Regards, > > > > > > > > Dmitry Kan > > > > > > > > > > > > > > > -- > > Regards, > > > > Dmitry Kan > > >