Hi Otis,
I followed your recommendation and decided to implement the
SearchComponent::modifyRequest(ResponseBuilder rb, SearchComponent who,
ShardRequest sreq) method, where the query routing happens. So far it is
working OK for the non-facet search, this is good news. The bad news is that
it fails on the facet search.
This is how request modification happens:
[code_snippet, SearchComponent::modifyRequest]
SolrQueryRequest req_routed = rb.req;
req_routed = routeRequest(req_routed);
rb.req = req_routed;
sreq.shards = shards.toString().split(",");
[/code_snippet]
where shards is StringBuilder, that accumulates the shards the request
should go to. req_routed also contains the target shards. Those are set like
this:
[code_snippet, my function routeRequest(SolrQueryRequest req)]
// could not find clone(), used ref reassignment
SolrQueryRequest req_local = req;
ModifiableSolrParams params = new
ModifiableSolrParams(req_local.getParams());
...
params.remove(ShardParams.SHARDS);
params.set(ShardParams.SHARDS, getShardsParams(yearToQuarterMap));
params.remove(ShardParams.IS_SHARD);
params.set(ShardParams.IS_SHARD, true);
req_local.setParams(params);
...
return req_local;
[/code_snippet]
The NPE happens down the road during the facet search, in the
FacetComponent::countFacets(), the cause of which is that OpenBitSet obs is
null for shardNum=0.
Do you have any idea why this happens, should some other field
of ResponseBuilder, SearchComponent or ShardRequest be changed?
BTW, I have tried to call FacetInfo::parse method inside
FacetComponent::modifyRequest() and countFacets(). Where do
the fi.facets.values() get initiated, is there some method to call?
Thanks,
Dmitry
On Fri, Jun 3, 2011 at 8:00 PM, Otis Gospodnetic <otis_gospodne...@yahoo.com
> wrote:
> Nah, if you can quickly figure out which shard a given query maps to, then
> all
> this component needs to do is stick the appropriate shards param value in
> the
> request and let the request pass through to the other SearchComponents in
> the
> chain, including QueryComponent, which will know what to do with the
> shards
> param.
>
> Otis
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Lucene ecosystem search :: http://search-lucene.com/
>
>
>
> ----- Original Message ----
> > From: Dmitry Kan <dmitry....@gmail.com>
> > To: solr-user@lucene.apache.org
> > Sent: Fri, June 3, 2011 12:56:15 PM
> > Subject: Re: query routing with shards
> >
> > Hi Otis,
> >
> > Thanks! This sounds promising. This custom implementation, will it hurt
> in
> > any way the stability of the front end SOLR? After implementing it, can
> I
> > run some tests to verify the stability / performance?
> >
> > Dmitry
> > On Fri, Jun 3, 2011 at 4:49 PM, Otis Gospodnetic <
> otis_gospodne...@yahoo.com
> > > wrote:
> >
> > > Hi Dmitry,
> > >
> > > Yes, you could also implement your own custom SearchComponent. In
> this
> > > component you could grab the query param, examine the query value, and
> > > based on
> > > that add the shards URL param with appropriate value, so that when the
> > > regular
> > > QueryComponent grabs stuff from the request, it has the correct shard
> in
> > > there
> > > already.
> > >
> > > Otis
> > > ----
> > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> > > Lucene ecosystem search :: http://search-lucene.com/
> > >
> > >
> > >
> > > ----- Original Message ----
> > > > From: Dmitry Kan <dmitry....@gmail.com>
> > > > To: solr-user@lucene.apache.org
> > > > Sent: Fri, June 3, 2011 2:47:00 AM
> > > > Subject: Re: query routing with shards
> > > >
> > > > Hi Otis,
> > > >
> > > > I merely followed on the gmail's suggestion to include other people
> into
> > > the
> > > > recipients list, Yonik was the first one :) I won't do it next
> time.
> > > >
> > > > Thanks for a rapid reply. The reason for doing this query routing
> is
> > > that we
> > > > abstract the distributed SOLR from the client code for security
> reasons
> > > > (that is, we don't want to expose the entire shard farm to the
> world,
> > > but
> > > > only the frontend SOLR) and for better decoupling.
> > > >
> > > > Is it possible to implement a plugin to SOLR that would map queries
> to
> > > > shards?
> > > >
> > > > We have other choices too, they'll take quite some time, that's why
> I
> > > > decided to quickly ask, if I was missing something from the SOLR
> main
> > > > components design and configuration.
> > > >
> > > > Dmitry
> > > >
> > > > On Fri, Jun 3, 2011 at 8:25 AM, Otis Gospodnetic <
> > > otis_gospodne...@yahoo.com
> > > > > wrote:
> > > >
> > > > > Hi Dmitry (you may not want to additionally copy Yonik, he's
> > > subscribed to
> > > > > this
> > > > > list, too)
> > > > >
> > > > >
> > > > > It sounds like you have the knowledge of which query maps to
> which
> > > shard.
> > > > > If
> > > > > so, why not control/change the value of "shards" param in the
> request
> > > to
> > > > > your
> > > > > front-end Solr (aka distributed request dispatcher) within your
> app,
> > > which
> > > > > is
> > > > > the one calling Solr?
> > > > >
> > > > > Otis
> > > > > ----
> > > > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> > > > > Lucene ecosystem search :: http://search-lucene.com/
> > > > >
> > > > >
> > > > >
> > > > > ----- Original Message ----
> > > > > > From: Dmitry Kan <dmitry....@gmail.com>
> > > > > > To: solr-user@lucene.apache.org; yo...@lucidimagination.com
> > > > > > Sent: Thu, June 2, 2011 7:00:53 AM
> > > > > > Subject: query routing with shards
> > > > > >
> > > > > > Hello all,
> > > > > >
> > > > > > We have currently several pretty fat logically isolated shards
> with
> > > the
> > > > > same
> > > > > > schema / solrconfig (indices are separate). We currently have
> one
> > > single
> > > > > > front end SOLR (1.4) for the client code calls. Since a client
> code
> > > > > query
> > > > > > usually hits only one shard, we are considering making a smart
> > > routing
> > > > > of
> > > > > > queries to the shards they map to. Can you please give some
> > > pointers as
> > > > > to
> > > > > > what would be an optimal way to achieve such a routing inside
> the
> > > front
> > > > > end
> > > > > > solr? Is there a way to configure mapping inside the
> solrconfig?
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > >
> > > > > > Dmitry Kan
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Dmitry Kan
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Dmitry Kan
> >
>
