Re: Unified highlighter with storeOffsetsWithPositions and termVectors giving an exception
On 22 Jul 2019, at 11:32 am, Richard Walker wrote: > I'm trying out the advice in the user guide > ( > https://lucene.apache.org/solr/guide/8_1/highlighting.html#schema-options-and-performance-considerations > ) > for using the unified highlighter. > > ... > * "set storeOffsetsWithPositions to true" > * "set termVectors to true but no other term vector > related options on the field being highlighted" ... I completely forgot to mention that I also tried _just_: > * "set storeOffsetsWithPositions to true" i.e., without _also_ setting termVectors, and this _doesn't_ give the exception. So it seems to be the _combination_ of: * unified highlighter * storeOffsetsWithPositions * termVectors that seems to be giving the exception.
Re: Returning multiple fields in graph streaming expression response documents
Good to hear.
Joel Bernstein
http://joelsolr.blogspot.com/
On Sun, Jul 21, 2019 at 5:21 PM Ahmed Adel wrote:
> Yeah, it turned out to be related to the data. The “fetch” method works
> fine as you described, it’s just the data distribution that caused name
> field not to be fetched in a number of responses. I tested it with two
> other collections and it worked as expected as well. Thank you for your
> help getting this running.
>
> Best,
> A. Adel
>
> On Sun, Jul 21, 2019 at 2:36 AM Joel Bernstein wrote:
>
> > Ok, then it sounds like a different issue. Let's look at the logs
> following
> > a request and see what the issue is. There will be a log record that
> shows
> > the query that is sent to Solr by the fetch expression. When we look at
> > that log we'll be able to see what the query is, and if results are
> > returned. It could be a bug in the code or it could be something related
> to
> > the data that's being fetched.
> >
> >
> > Joel Bernstein
> > http://joelsolr.blogspot.com/
> >
> >
> > On Sat, Jul 20, 2019 at 5:21 PM Ahmed Adel wrote:
> >
> > > To validate this, I indexed the datasets and ran the same query on Solr
> > > 6.5.0 environment (https://archive.apache.org/dist/lucene/solr/6.5.0/)
> > > before cb9f15 commit gets into release but got the same response, no
> > > additional fields, as Solr 8.1.1. I have used the default managed
> schema
> > > settings in both Solr versions, which I guess means qparser is not used
> > for
> > > /select in this case, is it?
> > >
> > > On Sat, Jul 20, 2019 at 2:02 AM Joel Bernstein
> > wrote:
> > >
> > > > I suspect fetch is having problem due to this commit:
> > > >
> > > >
> > > >
> > >
> >
> https://github.com/apache/lucene-solr/commit/cb9f151db4b5ad5c5f581b6b8cf2e5916ddb0f35#diff-98abfc8855d347035205c6f3afc2cde3
> > > >
> > > > Later local params were turned off for anything but the lucene
> qparser.
> > > > Which means this query doesn't work if /select is using edismax
> etc...
> > > >
> > > > This needs to be fixed.
> > > > Can you check to see if the qparser is for the /select handler on
> your
> > > > install?
> > > >
> > > > Anyway fetch needs to be reverted back to it's previous
> implementation
> > > > before the above commit basically broke it.
> > > >
> > > >
> > > >
> > > >
> > > > Joel Bernstein
> > > > http://joelsolr.blogspot.com/
> > > >
> > > >
> > > > On Fri, Jul 19, 2019 at 2:20 PM Ahmed Adel
> wrote:
> > > >
> > > > > Hi - Tried swapping the equality sides but (surprisingly?) got the
> > same
> > > > > exact response. Any additional thoughts are appreciated.
> > > > >
> > > > > Best,
> > > > > A.
> > > > > http://aadel.io
> > > > >
> > > > > On Fri, Jul 19, 2019 at 5:27 PM Joel Bernstein >
> > > > wrote:
> > > > >
> > > > > > Try:
> > > > > >
> > > > > > fetch(names,
> > > > > > select(
> > > > > > nodes(emails,
> > > > > > walk="john...@apache.org->from",
> > > > > > gather="to"),
> > > > > > node as to_s),
> > > > > > fl="name",
> > > > > > on="to_s=email")
> > > > > >
> > > > > >
> > > > > > According to the docs it looks like you have the fields reversed
> on
> > > the
> > > > > > fetch. If that doesn't work, I'll investigate further.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Joel Bernstein
> > > > > > http://joelsolr.blogspot.com/
> > > > > >
> > > > > >
> > > > > > On Fri, Jul 19, 2019 at 5:51 AM Ahmed Adel
> > > wrote:
> > > > > >
> > > > > > > Hi Joel,
> > > > > > >
> > > > > > > Thank you for your thoughts. I tried the fetch function,
> however,
> > > the
> > > > > > > response does not contain "fl" fields of the "fetch"
> expression.
> > > For
> > > > > the
> > > > > > > above example, the modified query is as follows:
> > > > > > >
> > > > > > > fetch(names, select(nodes(emails,
> > > > > > > walk="john...@apache.org->from",
> > > > > > > gather="to"), node as to_s), fl="name", on="email=to_s")
> > > > > > >
> > > > > > >
> > > > > > > where "names" is a collection that contains two fields
> > representing
> > > > > pairs
> > > > > > > of name and email: ("name", "email")
> > > > > > >
> > > > > > > The response returned is:
> > > > > > >
> > > > > > > { "result-set": { "docs": [ { "to_s": "john...@apache.org"
> > > > > > > }, { "to_s": "johnsm...@apache.org"
> > > > > > > },
> > > > > > > ... { "EOF": true, "RESPONSE_TIME": 33 } ] } }
> > > > > > >
> > > > > > > The response should have an additional "name" field in each
> > > document
> > > > > > > returned. Any additional thoughts are appreciated.
> > > > > > >
> > > > > > > Best,
> > > > > > > A.
> > > > > > >
> > > > > > > On Thu, Jul 18, 2019 at 6:12 PM Joel Bernstein <
> > joels...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Ahmed,
> > > > > > > >
> > > > > > > > Take a look at the fetch
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Unified highlighter with storeOffsetsWithPositions and termVectors giving an exception
I'm trying out the advice in the user guide ( https://lucene.apache.org/solr/guide/8_1/highlighting.html#schema-options-and-performance-considerations ) for using the unified highlighter. I saw the note: "This is definitely the fastest option for highlighting wildcard queries on large text fields." and decided to try this, namely: * "set storeOffsetsWithPositions to true" * "set termVectors to true but no other term vector related options on the field being highlighted" I've set these options on two fields, but I now get an exception during highlighting of the results of a phrase query. (I'm not even testing with wildcards yet.) Here's an extract of the schema before making the change: And here are the only two lines I changed: Here's a sample minimal query that worked perfectly before making the change: defType=edismax q="space administration" fl=id,title qf=fulltext concept_search hl=true hl.method=unified hl.fl=* After making the change to the schema, I now get this exception in the Solr log: o.a.s.s.HttpSolrCall null:java.lang.IllegalStateException: field "fulltext" was indexed without position data; cannot run PhraseQuery (phrase=fulltext:"space administr") at org.apache.lucene.search.PhraseQuery$1.getPhraseMatcher(PhraseQuery.java:446) at org.apache.lucene.search.PhraseWeight.lambda$matches$0(PhraseWeight.java:89) at org.apache.lucene.search.MatchesUtils.forField(MatchesUtils.java:101) at org.apache.lucene.search.PhraseWeight.matches(PhraseWeight.java:88) at org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.matches(DisjunctionMaxQuery.java:125) at org.apache.lucene.search.uhighlight.FieldOffsetStrategy.createOffsetsEnumsWeightMatcher(FieldOffsetStrategy.java:138) at org.apache.lucene.search.uhighlight.FieldOffsetStrategy.createOffsetsEnumFromReader(FieldOffsetStrategy.java:74) at org.apache.lucene.search.uhighlight.TermVectorOffsetStrategy.getOffsetsEnum(TermVectorOffsetStrategy.java:49) at org.apache.lucene.search.uhighlight.FieldHighlighter.highlightFieldForDoc(FieldHighlighter.java:76) at org.apache.lucene.search.uhighlight.UnifiedHighlighter.highlightFieldsAsObjects(UnifiedHighlighter.java:639) at org.apache.lucene.search.uhighlight.UnifiedHighlighter.highlightFields(UnifiedHighlighter.java:508) at org.apache.solr.highlight.UnifiedSolrHighlighter.doHighlighting(UnifiedSolrHighlighter.java:149) at org.apache.solr.handler.component.HighlightComponent.process(HighlightComponent.java:171) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:298) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199) at org.apache.solr.core.SolrCore.execute(SolrCore.java:2566) etc. The response includes search results, but no highlighting information. Of interest is that the exception is against the field "fulltext", whose definition I _didn't_ change. If I remove the "fulltext" field from qf, so that the query is now this: defType=edismax q="space administration" fl=id,title qf=concept_search hl=true hl.method=unified hl.fl=* the log now has this exception: o.a.s.s.HttpSolrCall null:java.lang.IllegalStateException: field "concept_search" was indexed without position data; cannot run PhraseQuery (phrase=concept_search:"space administr") at org.apache.lucene.search.PhraseQuery$1.getPhraseMatcher(PhraseQuery.java:446) at org.apache.lucene.search.PhraseWeight.lambda$matches$0(PhraseWeight.java:89) at org.apache.lucene.search.MatchesUtils.forField(MatchesUtils.java:101) at org.apache.lucene.search.PhraseWeight.matches(PhraseWeight.java:88) at org.apache.lucene.search.uhighlight.FieldOffsetStrategy.createOffsetsEnumsWeightMatcher(FieldOffsetStrategy.java:138) at org.apache.lucene.search.uhighlight.FieldOffsetStrategy.createOffsetsEnumFromReader(FieldOffsetStrategy.java:74) at org.apache.lucene.search.uhighlight.TermVectorOffsetStrategy.getOffsetsEnum(TermVectorOffsetStrategy.java:49) at org.apache.lucene.search.uhighlight.FieldHighlighter.highlightFieldForDoc(FieldHighlighter.java:76) at org.apache.lucene.search.uhighlight.UnifiedHighlighter.highlightFieldsAsObjects(UnifiedHighlighter.java:639) at org.apache.lucene.search.uhighlight.UnifiedHighlighter.highlightFields(UnifiedHighlighter.java:508) at org.apache.solr.highlight.UnifiedSolrHighlighter.doHighlighting(UnifiedSolrHighlighter.java:149) at org.apache.solr.handler.component.HighlightComponent.process(HighlightComponent.java:171) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:298) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199) at
Re: Returning multiple fields in graph streaming expression response documents
Yeah, it turned out to be related to the data. The “fetch” method works
fine as you described, it’s just the data distribution that caused name
field not to be fetched in a number of responses. I tested it with two
other collections and it worked as expected as well. Thank you for your
help getting this running.
Best,
A. Adel
On Sun, Jul 21, 2019 at 2:36 AM Joel Bernstein wrote:
> Ok, then it sounds like a different issue. Let's look at the logs following
> a request and see what the issue is. There will be a log record that shows
> the query that is sent to Solr by the fetch expression. When we look at
> that log we'll be able to see what the query is, and if results are
> returned. It could be a bug in the code or it could be something related to
> the data that's being fetched.
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
>
> On Sat, Jul 20, 2019 at 5:21 PM Ahmed Adel wrote:
>
> > To validate this, I indexed the datasets and ran the same query on Solr
> > 6.5.0 environment (https://archive.apache.org/dist/lucene/solr/6.5.0/)
> > before cb9f15 commit gets into release but got the same response, no
> > additional fields, as Solr 8.1.1. I have used the default managed schema
> > settings in both Solr versions, which I guess means qparser is not used
> for
> > /select in this case, is it?
> >
> > On Sat, Jul 20, 2019 at 2:02 AM Joel Bernstein
> wrote:
> >
> > > I suspect fetch is having problem due to this commit:
> > >
> > >
> > >
> >
> https://github.com/apache/lucene-solr/commit/cb9f151db4b5ad5c5f581b6b8cf2e5916ddb0f35#diff-98abfc8855d347035205c6f3afc2cde3
> > >
> > > Later local params were turned off for anything but the lucene qparser.
> > > Which means this query doesn't work if /select is using edismax etc...
> > >
> > > This needs to be fixed.
> > > Can you check to see if the qparser is for the /select handler on your
> > > install?
> > >
> > > Anyway fetch needs to be reverted back to it's previous implementation
> > > before the above commit basically broke it.
> > >
> > >
> > >
> > >
> > > Joel Bernstein
> > > http://joelsolr.blogspot.com/
> > >
> > >
> > > On Fri, Jul 19, 2019 at 2:20 PM Ahmed Adel wrote:
> > >
> > > > Hi - Tried swapping the equality sides but (surprisingly?) got the
> same
> > > > exact response. Any additional thoughts are appreciated.
> > > >
> > > > Best,
> > > > A.
> > > > http://aadel.io
> > > >
> > > > On Fri, Jul 19, 2019 at 5:27 PM Joel Bernstein
> > > wrote:
> > > >
> > > > > Try:
> > > > >
> > > > > fetch(names,
> > > > > select(
> > > > > nodes(emails,
> > > > > walk="john...@apache.org->from",
> > > > > gather="to"),
> > > > > node as to_s),
> > > > > fl="name",
> > > > > on="to_s=email")
> > > > >
> > > > >
> > > > > According to the docs it looks like you have the fields reversed on
> > the
> > > > > fetch. If that doesn't work, I'll investigate further.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Joel Bernstein
> > > > > http://joelsolr.blogspot.com/
> > > > >
> > > > >
> > > > > On Fri, Jul 19, 2019 at 5:51 AM Ahmed Adel
> > wrote:
> > > > >
> > > > > > Hi Joel,
> > > > > >
> > > > > > Thank you for your thoughts. I tried the fetch function, however,
> > the
> > > > > > response does not contain "fl" fields of the "fetch" expression.
> > For
> > > > the
> > > > > > above example, the modified query is as follows:
> > > > > >
> > > > > > fetch(names, select(nodes(emails,
> > > > > > walk="john...@apache.org->from",
> > > > > > gather="to"), node as to_s), fl="name", on="email=to_s")
> > > > > >
> > > > > >
> > > > > > where "names" is a collection that contains two fields
> representing
> > > > pairs
> > > > > > of name and email: ("name", "email")
> > > > > >
> > > > > > The response returned is:
> > > > > >
> > > > > > { "result-set": { "docs": [ { "to_s": "john...@apache.org"
> > > > > > }, { "to_s": "johnsm...@apache.org"
> > > > > > },
> > > > > > ... { "EOF": true, "RESPONSE_TIME": 33 } ] } }
> > > > > >
> > > > > > The response should have an additional "name" field in each
> > document
> > > > > > returned. Any additional thoughts are appreciated.
> > > > > >
> > > > > > Best,
> > > > > > A.
> > > > > >
> > > > > > On Thu, Jul 18, 2019 at 6:12 PM Joel Bernstein <
> joels...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi Ahmed,
> > > > > > >
> > > > > > > Take a look at the fetch
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://lucene.apache.org/solr/guide/8_0/stream-decorator-reference.html#fetch
> > > > > > >
> > > > > > > It probably makes sense to allow more field to be returned
> from a
> > > > nodes
> > > > > > > expression as well.
> > > > > > >
> > > > > > > Joel Bernstein
> > > > > > > http://joelsolr.blogspot.com/
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Jul 17, 2019 at 3:12 AM Ahmed Adel
> > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > Thank
Re: Add certain documents right after particular document
Thanks for your reply Alex and Yogendra. @Yogendra: I have tried with grouping but it did not work. I will recheck again if I am missing somethin. @Alex : Recommendations are some of the products that comes from Solr. Here is the scenario: 1) We get 100 products from Solr on product listing page. 2) Customer viewed 21st product and comes back to product listing page. 3) While load products listing page again , we invoke a third party service and get some recommendations ( which are some products out of those hundred products). Now we want to put those recommendations right after last viewed product. Is there anyway we can ask Solr to put those products rt after a product in the response. Thanks, Venkat. On Sun, Jul 21, 2019 at 7:07 AM Alexandre Rafalovitch wrote: > So, if the recommendations are dynamic and come from outside Solr, why > do you need Solr to do anything at this stage? Sounds like the > original result list is where Solr responsibility ends. > > You are not exposing Solr directly to the UI (you should not), so > whatever your middleware is, can be coded any way you want. > > Regards, >Alex. > > On Sun, 21 Jul 2019 at 00:17, Venkateswarlu Bommineni > wrote: > > > > Thanks Saurabh for quick response. > > > > Did you mean to index recommended products as child documents ? and get > > them back using block join to get the results ? > > > > If yes , difficulty is recommendations are very dynamic and they totally > > depends on the user's session and how many products and type of the > product > > he has viewed. so Its really impossible to index them. > > > > Thanks, > > Venkat. > > > > On Sat, Jul 20, 2019 at 9:07 PM Saurabh Sharma < > saurabh.infoe...@gmail.com> > > wrote: > > > > > Hi, > > > > > > You can add recommendations as child documents and use block join for > > > retrieval. By adding as child it will easy for you to keep track of the > > > recommendations. > > > > > > Thanks > > > Saurabh > > > > > > On Sun, Jul 21, 2019, 8:16 AM Venkateswarlu Bommineni < > bvr...@gmail.com> > > > wrote: > > > > > > > Hello Team, > > > > > > > > Apologies , if you feel this question is silly and weird. > > > > > > > > Here is the scenario I am trying to achieve. > > > > > > > > 1) Customers come to PCAT (Product category/listing pages) [we hit > Solr > > > to > > > > get the results] > > > > 2) Customer selects a product and goes to product details page. > > > > 3) When customer comes back to PCAT , we will hit third party > service and > > > > get some recommendations. > > > > *problem : W*e want to add those recommendations right after the > product > > > > that customer viewed previously. > > > > > > > > Is there any way we can tell to Solr to add those products right > after a > > > > particular product while giving us the response ? > > > > > > > > > > > > Thanks, > > > > Venkat. > > > > > > > >
Re: Add certain documents right after particular document
So, if the recommendations are dynamic and come from outside Solr, why do you need Solr to do anything at this stage? Sounds like the original result list is where Solr responsibility ends. You are not exposing Solr directly to the UI (you should not), so whatever your middleware is, can be coded any way you want. Regards, Alex. On Sun, 21 Jul 2019 at 00:17, Venkateswarlu Bommineni wrote: > > Thanks Saurabh for quick response. > > Did you mean to index recommended products as child documents ? and get > them back using block join to get the results ? > > If yes , difficulty is recommendations are very dynamic and they totally > depends on the user's session and how many products and type of the product > he has viewed. so Its really impossible to index them. > > Thanks, > Venkat. > > On Sat, Jul 20, 2019 at 9:07 PM Saurabh Sharma > wrote: > > > Hi, > > > > You can add recommendations as child documents and use block join for > > retrieval. By adding as child it will easy for you to keep track of the > > recommendations. > > > > Thanks > > Saurabh > > > > On Sun, Jul 21, 2019, 8:16 AM Venkateswarlu Bommineni > > wrote: > > > > > Hello Team, > > > > > > Apologies , if you feel this question is silly and weird. > > > > > > Here is the scenario I am trying to achieve. > > > > > > 1) Customers come to PCAT (Product category/listing pages) [we hit Solr > > to > > > get the results] > > > 2) Customer selects a product and goes to product details page. > > > 3) When customer comes back to PCAT , we will hit third party service and > > > get some recommendations. > > > *problem : W*e want to add those recommendations right after the product > > > that customer viewed previously. > > > > > > Is there any way we can tell to Solr to add those products right after a > > > particular product while giving us the response ? > > > > > > > > > Thanks, > > > Venkat. > > > > >
