Hi, The Highlighter is way too slow for this customer's particular use case - which is veery large documents. We don't need highlighted snippets for now, but we need to accurately decide what words (offsets) in the real HTML display of the resulting page to highlight. For this we only need offset info, not the snippets/fragments from the stored field.
But I have not looked at the Highlighter code. Perhaps we could fork it into a new search component which pulls out only the necessary meta info and payloads for us and returns it to client? -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com Solr Training - www.solrtraining.com On 13. okt. 2011, at 16:23, Mike Sokolov wrote: > Is there some reason you don't want to leverage Highlighter to do this work? > It has all the necessary code for using the analyzed version of your query so > it will only match tokens that really contribute to the search match. > > You might also be interested in LUCENE-2878 (which is still under development > on a branch though). It aims to provide first-class access to payloads and > positions during scoring, and this will be very useful for complex > highlighting tasks. > > Another possible solution to the OCR problem could be: generate an XML file > with a tag for each word encoding its x,y coords, like : <word x="3" > y="10">This</word>; index that file using XmlCharFilter or > HTMLStripCharFilter. Then when you search, use the Solr highlighter to > highlight the entire document, and process it using XML tools to find the > locations of the matches. > > -Mike > > On 10/10/2011 10:19 AM, Jan Høydahl wrote: >> Hi, >> >> We index structured documents, with numbered chapters, paragraphs and >> sentences. After doing a (rather complex) search, we may get multiple >> matches in each result doc. We want to highlight those matches in our >> front-end and currently we do a simple string match of the query words >> against the raw text. >> >> However, this highlights some words that do not satisfy the original query, >> and also does not highlight other words where the match was in a stem, or >> synonym or wildcard. We thus need to improve this, and my plan was to >> utilize DocValues (Payloads). Would the following work? >> >> 1. For each term in the field "text", index DocValues with info about >> chapter#, paragraph#, sentence# and word#. >> This can be done in our application code, e.g. "foo|1,2,3,4" for chapter >> 1, paragraph 2, sentence 3 and word 4. >> >> 2. Then, for a specific document in the result list, retrieve a list of all >> matches in field "text", and for each match, >> retrieve the associated DocValues. >> >> 3. The client application can now use this information to highlight matches, >> as well as "jump to next match" etc, >> and would highlight the correct words only, e.g. it would be able to >> highlight "colour" even if the match was on the >> synonym "color". >> >> Another use case for this technique would be OCR applications where we store >> with each term its x,y offsets for where it occurs in >> the original TIFF image scan. >> >> What is in already in place and what code needs to be written? I don't >> currently see how to get a complete list of matches for a particular >> document. >> >> -- >> Jan Høydahl, search solution architect >> Cominvent AS - www.cominvent.com >> Solr Training - www.solrtraining.com >> >>
