RE: Highlighter not working on some documents
I managed to miss that. Thanks very much. I have some very large documents. I will look at index size and look at posting instead. -Original Message- From: David Smiley [mailto:david.w.smi...@gmail.com] Sent: Monday, 12 June 2017 2:40 p.m. To: solr-user@lucene.apache.org Subject: Re: Highlighter not working on some documents Probably the most common reason is the default hl.maxAnalyzedChars -- thus your highlightable text might not be in the first 51200 chars of text. The first Solr release with the unified highlighter had an even lower default of 10k chars. On Fri, Jun 9, 2017 at 9:58 PM Phil Scadden wrote: > Tried hard to find difference between pdfs returning no highlighter > and ones that do for same search term. Includes pdfs that have been > OCRed and ones that were text to begin with. Head scratching to me. > > -Original Message- > From: Erick Erickson [mailto:erickerick...@gmail.com] > Sent: Saturday, 10 June 2017 6:22 a.m. > To: solr-user > Subject: Re: Highlighter not working on some documents > > Need lots more information. I.e. schema definitions, query you use, > handler configuration and the like. Note that highlighted fields must > have stored="true" set and likely the _text_ field doesn't. At least > in the default schemas stored is set to false for the catch-all field. > And you don't want to store that information anyway since it's usually > the destination of copyField directives and you'd highlight _those_ fields. > > Best, > Erick > > On Thu, Jun 8, 2017 at 8:37 PM, Phil Scadden wrote: > > Do a search with: > > fl=id,title,datasource&hl=true&hl.method=unified&limit=50&page=1&q=p > > re > > ssure+AND+testing&rows=50&start=0&wt=json > > > > and I get back a good list of documents. However, some documents are > returning empty fields in the highlighter. Eg, in the highlight array have: > > "W:\\Reports\\OCR\\4272.pdf":{"_text_":[]} > > > > Getting this well up the list of results with good highlighted > > matchers > above and below this entry. Why would the highlighter be failing? > > > > Notice: This email and any attachments are confidential and may not > > be > used, published or redistributed without the prior written consent of > the Institute of Geological and Nuclear Sciences Limited (GNS > Science). If received in error please destroy and immediately notify > GNS Science. Do not copy or disclose the contents. > Notice: This email and any attachments are confidential and may not be > used, published or redistributed without the prior written consent of > the Institute of Geological and Nuclear Sciences Limited (GNS > Science). If received in error please destroy and immediately notify > GNS Science. Do not copy or disclose the contents. > -- Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker LinkedIn: http://linkedin.com/in/davidwsmiley | Book: http://www.solrenterprisesearchserver.com Notice: This email and any attachments are confidential and may not be used, published or redistributed without the prior written consent of the Institute of Geological and Nuclear Sciences Limited (GNS Science). If received in error please destroy and immediately notify GNS Science. Do not copy or disclose the contents.
Re: Highlighter not working on some documents
Probably the most common reason is the default hl.maxAnalyzedChars -- thus your highlightable text might not be in the first 51200 chars of text. The first Solr release with the unified highlighter had an even lower default of 10k chars. On Fri, Jun 9, 2017 at 9:58 PM Phil Scadden wrote: > Tried hard to find difference between pdfs returning no highlighter and > ones that do for same search term. Includes pdfs that have been OCRed and > ones that were text to begin with. Head scratching to me. > > -Original Message- > From: Erick Erickson [mailto:erickerick...@gmail.com] > Sent: Saturday, 10 June 2017 6:22 a.m. > To: solr-user > Subject: Re: Highlighter not working on some documents > > Need lots more information. I.e. schema definitions, query you use, > handler configuration and the like. Note that highlighted fields must have > stored="true" set and likely the _text_ field doesn't. At least in the > default schemas stored is set to false for the catch-all field. > And you don't want to store that information anyway since it's usually the > destination of copyField directives and you'd highlight _those_ fields. > > Best, > Erick > > On Thu, Jun 8, 2017 at 8:37 PM, Phil Scadden wrote: > > Do a search with: > > fl=id,title,datasource&hl=true&hl.method=unified&limit=50&page=1&q=pre > > ssure+AND+testing&rows=50&start=0&wt=json > > > > and I get back a good list of documents. However, some documents are > returning empty fields in the highlighter. Eg, in the highlight array have: > > "W:\\Reports\\OCR\\4272.pdf":{"_text_":[]} > > > > Getting this well up the list of results with good highlighted matchers > above and below this entry. Why would the highlighter be failing? > > > > Notice: This email and any attachments are confidential and may not be > used, published or redistributed without the prior written consent of the > Institute of Geological and Nuclear Sciences Limited (GNS Science). If > received in error please destroy and immediately notify GNS Science. Do not > copy or disclose the contents. > Notice: This email and any attachments are confidential and may not be > used, published or redistributed without the prior written consent of the > Institute of Geological and Nuclear Sciences Limited (GNS Science). If > received in error please destroy and immediately notify GNS Science. Do not > copy or disclose the contents. > -- Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker LinkedIn: http://linkedin.com/in/davidwsmiley | Book: http://www.solrenterprisesearchserver.com
RE: Highlighter not working on some documents
Tried hard to find difference between pdfs returning no highlighter and ones that do for same search term. Includes pdfs that have been OCRed and ones that were text to begin with. Head scratching to me. -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Saturday, 10 June 2017 6:22 a.m. To: solr-user Subject: Re: Highlighter not working on some documents Need lots more information. I.e. schema definitions, query you use, handler configuration and the like. Note that highlighted fields must have stored="true" set and likely the _text_ field doesn't. At least in the default schemas stored is set to false for the catch-all field. And you don't want to store that information anyway since it's usually the destination of copyField directives and you'd highlight _those_ fields. Best, Erick On Thu, Jun 8, 2017 at 8:37 PM, Phil Scadden wrote: > Do a search with: > fl=id,title,datasource&hl=true&hl.method=unified&limit=50&page=1&q=pre > ssure+AND+testing&rows=50&start=0&wt=json > > and I get back a good list of documents. However, some documents are > returning empty fields in the highlighter. Eg, in the highlight array have: > "W:\\Reports\\OCR\\4272.pdf":{"_text_":[]} > > Getting this well up the list of results with good highlighted matchers above > and below this entry. Why would the highlighter be failing? > > Notice: This email and any attachments are confidential and may not be used, > published or redistributed without the prior written consent of the Institute > of Geological and Nuclear Sciences Limited (GNS Science). If received in > error please destroy and immediately notify GNS Science. Do not copy or > disclose the contents. Notice: This email and any attachments are confidential and may not be used, published or redistributed without the prior written consent of the Institute of Geological and Nuclear Sciences Limited (GNS Science). If received in error please destroy and immediately notify GNS Science. Do not copy or disclose the contents.
RE: Highlighter not working on some documents
Managed-schema attached (not a default) and the solrconfig.xml. _text_ is stored. (not sure how else highlighting could work??). The indexer puts the body text of the pdf into _text_ field. What the value be in putting it into a different field and then using copyField?? Ie SolrInputDocument up = new SolrInputDocument(); String content = textHandler.toString(); up.addField("_text_",content); solr.add(up); The puzzling thing for me is why are some documents producing highlights and others not. The highlighters in the documents that work are pulling body text fragments, not things stored in some other field. -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Saturday, 10 June 2017 6:22 a.m. To: solr-user Subject: Re: Highlighter not working on some documents Need lots more information. I.e. schema definitions, query you use, handler configuration and the like. Note that highlighted fields must have stored="true" set and likely the _text_ field doesn't. At least in the default schemas stored is set to false for the catch-all field. And you don't want to store that information anyway since it's usually the destination of copyField directives and you'd highlight _those_ fields. Best, Erick On Thu, Jun 8, 2017 at 8:37 PM, Phil Scadden wrote: > Do a search with: > fl=id,title,datasource&hl=true&hl.method=unified&limit=50&page=1&q=pre > ssure+AND+testing&rows=50&start=0&wt=json > > and I get back a good list of documents. However, some documents are > returning empty fields in the highlighter. Eg, in the highlight array have: > "W:\\Reports\\OCR\\4272.pdf":{"_text_":[]} > > Getting this well up the list of results with good highlighted matchers above > and below this entry. Why would the highlighter be failing? > > Notice: This email and any attachments are confidential and may not be used, > published or redistributed without the prior written consent of the Institute > of Geological and Nuclear Sciences Limited (GNS Science). If received in > error please destroy and immediately notify GNS Science. Do not copy or > disclose the contents. Notice: This email and any attachments are confidential and may not be used, published or redistributed without the prior written consent of the Institute of Geological and Nuclear Sciences Limited (GNS Science). If received in error please destroy and immediately notify GNS Science. Do not copy or disclose the contents. solrconfig.xml Description: solrconfig.xml
Re: Highlighter not working on some documents
Need lots more information. I.e. schema definitions, query you use, handler configuration and the like. Note that highlighted fields must have stored="true" set and likely the _text_ field doesn't. At least in the default schemas stored is set to false for the catch-all field. And you don't want to store that information anyway since it's usually the destination of copyField directives and you'd highlight _those_ fields. Best, Erick On Thu, Jun 8, 2017 at 8:37 PM, Phil Scadden wrote: > Do a search with: > fl=id,title,datasource&hl=true&hl.method=unified&limit=50&page=1&q=pressure+AND+testing&rows=50&start=0&wt=json > > and I get back a good list of documents. However, some documents are > returning empty fields in the highlighter. Eg, in the highlight array have: > "W:\\Reports\\OCR\\4272.pdf":{"_text_":[]} > > Getting this well up the list of results with good highlighted matchers above > and below this entry. Why would the highlighter be failing? > > Notice: This email and any attachments are confidential and may not be used, > published or redistributed without the prior written consent of the Institute > of Geological and Nuclear Sciences Limited (GNS Science). If received in > error please destroy and immediately notify GNS Science. Do not copy or > disclose the contents.