Mikhail,
Thanks for sharing your thoughts. Yes I have tried checking for NULL and
the entire chain of queries between tiers seems to work. But I suspect,
that some docs will be missing. In principle, unless there is an
OutOfMemory or a shard down, the doc ids should be retrieving valid
documents. So this is just a design, as Yonik pointed out.
I would be willing to contribute a patch, it is just an issue of
understanding what exactly should be fixed in the architecture, and I
suspect it isn't a small change..
Dmitry
On Thu, Apr 12, 2012 at 9:22 PM, Mikhail Khludnev <
mkhlud...@griddynamics.com> wrote:
> Dmitry,
>
> The last NPE in HighlightingComponent is just a sad coding issue.
> few rows later we can see that developer expected to have some docs not
> found
> // remove nulls in case not all docs were able to be retrieved
> rb.rsp.add("highlighting", SolrPluginUtils.removeNulls(new
> SimpleOrderedMap(arr)));
> But as you already know he forgot to check if(sdoc!=null){.
> Is there anything that stopping you from contributing the patch, beside of
> the lack of time, of course?
>
> about the core issue I can't get into it and, particularly, how the using
> disjunction query in place of IDS can help you. Could you please provide
> more detailed info like stacktraces, etc. Btw, have you checked trunk for
> your case?
>
> On Thu, Apr 12, 2012 at 7:08 PM, Dmitry Kan <dmitry....@gmail.com> wrote:
>
> > Can anyone help me out with this? Is this too complicated / unclear? I
> > could share more detail if needed.
> >
> > On Wed, Apr 11, 2012 at 3:16 PM, Dmitry Kan <dmitry....@gmail.com>
> wrote:
> >
> > > Hello,
> > >
> > > Hopefully this question is not too complex to handle, but I'm currently
> > > stuck with it.
> > >
> > > We have a system with nTiers, that is:
> > >
> > > Solr front base ---> Solr front --> shards
> > >
> > > Inside QueryComponent there is a method
> > createRetrieveDocs(ResponseBuilder
> > > rb) which collects doc ids of each shard and sends them in different
> > > queries using the ids parameter:
> > >
> > > [code]
> > > sreq.params.add(ShardParams.IDS, StrUtils.join(ids, ','));
> > > [/code]
> > >
> > > This actually produces NPE (same as in
> > > https://issues.apache.org/jira/browse/SOLR-1477) in the first tier,
> > > because Solr front (on the second tier) fails to process such a query.
> I
> > > have tried to fix this by using a unique field with a value of ids ORed
> > > (the following code substitutes the code above):
> > >
> > > [code]
> > > StringBuffer idsORed = new StringBuffer();
> > > for (Iterator<String> iterator = ids.iterator();
> > iterator.hasNext();
> > > ) {
> > > String next = iterator.next();
> > >
> > > if (iterator.hasNext()) {
> > > idsORed.append(next).append(" OR ");
> > > } else {
> > > idsORed.append(next);
> > > }
> > > }
> > >
> > > sreq.params.add(rb.req.getSchema().getUniqueKeyField().getName(),
> > > idsORed.toString());
> > > [/code]
> > >
> > > This works perfectly if for rows=n there is n or less hits from a
> > > distributed query. However, if there are more than 2*n hits, the
> querying
> > > fails with an NPE in a completely different component, which is
> > > HighlightComponent (highlights are requested in the same query with
> > >
> >
> hl=true&hl.fragsize=50000&hl.requireFieldMatch=true&hl.fl=targetTextField):
> > >
> > > SEVERE: java.lang.NullPointerException
> > > at
> > >
> >
> org.apache.solr.handler.component.HighlightComponent.finishStage(HighlightComponent.java:161)
> > > at
> > >
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
> > > at
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
> > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
> > > at
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
> > > at
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
> > > at
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> > > at
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > > at
> > >
> >
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> > > at
> > >
> >
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> > > at
> > >
> >
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> > > at
> > >
> >
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> > > at
> > >
> >
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> > > at
> > >
> >
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
> > > at
> > >
> >
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
> > > at
> > >
> >
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
> > > at
> > > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
> > > at java.lang.Thread.run(Thread.java:619)
> > >
> > > It sounds like the ids of documents somehow get shuffled and the
> > > instruction (only a hypothesis)
> > >
> > > [code]
> > > ShardDoc sdoc = rb.resultIds.get(id);
> > > [/code]
> > >
> > > returns sdoc=null, which causes the next line of code to fail with an
> > NPE:
> > >
> > > [code]
> > > int idx = sdoc.positionInResponse;
> > > [/code]
> > >
> > > Am I missing anything? Can something be done for solving this issue?
> > >
> > > Thanks.
> > >
> > > --
> > > Regards,
> > >
> > > Dmitry Kan
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Dmitry Kan
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> ge...@yandex.ru
>
> <http://www.griddynamics.com>
> <mkhlud...@griddynamics.com>
>
--
Regards,
Dmitry Kan
