I did that and it worked. Thanks very much for your expert assistance, Jan!
Paul From: Jan Høydahl / Cominvent <jan....@cominvent.com> To: solr-user@lucene.apache.org Date: 19/08/2010 16:15 Subject: Re: Missing tokens Hi, Your bug is right there in the WhitespaceTokenizer, where you see that it does NOT strip away the "." as whitespace. Try with StandardTokenizerFactory instead, as it removes punctuation. -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com Training in Europe - www.solrtraining.com On 19. aug. 2010, at 12.16, paul.mo...@dds.net wrote: > Great! Now I'm getting somewhere, this worked! The others didn't. > > http://localhost/solr/select?q=contents:"OB10."; > > Hope this makes sense to you. I'm still somewhat confused with the output > here. I had 'highlight matches' check, and from what I can tell, 'OB10' > wasn't found. When I enter 'OB10.' into the query, column 11 'ob10.' became > highlighted in the 'LowerCaseFilterFactory' table. > > Am I using the wrong analyser, or supplying the wrong parameters to an > analyser? > > Thanks for your help so far! > Paul > > Index Analyzer > org.apache.solr.analysis.WhitespaceTokenizerFactory {} > |--------------+----+-------+-----+------------+-----+-----+-----+------+--------+-----+------+-----+--------| > |term position |1 |2 |3 |4 |5 |6 |7 |8 | 9 |10 |11 |12 |13 | > |--------------+----+-------+-----+------------+-----+-----+-----+------+--------+-----+------+-----+--------| > | term text |To |produce|a |downloadable|file |using|a |format| suitable|for |OB10. |8-26 |Profiles| > |--------------+----+-------+-----+------------+-----+-----+-----+------+--------+-----+------+-----+--------| > | term type |word|word |word |word |word |word |word |word | word |word |word |word |word | > |--------------+----+-------+-----+------------+-----+-----+-----+------+--------+-----+------+-----+--------| > | source |0,2 |3,10 |11,12|13,25 |26,30|31,36|37,38|39,45 | 46,54 |55,58|59,64 |65,69|70,78 | > | start,end | | | | | | | | | | | | | | > |--------------+----+-------+-----+------------+-----+-----+-----+------+--------+-----+------+-----+--------| > | payload | | | | | | | | | | | | | | > |--------------+----+-------+-----+------------+-----+-----+-----+------+--------+-----+------+-----+--------| > > > org.apache.solr.analysis.StandardFilterFactory {} > |--------------+----+-------+-----+------------+-----+-----+-----+------+--------+-----+------+-----+--------| > |term position |1 |2 |3 |4 |5 |6 |7 |8 | 9 |10 |11 |12 |13 | > |--------------+----+-------+-----+------------+-----+-----+-----+------+--------+-----+------+-----+--------| > | term text |To |produce|a |downloadable|file |using|a |format| suitable|for |OB10. |8-26 |Profiles| > |--------------+----+-------+-----+------------+-----+-----+-----+------+--------+-----+------+-----+--------| > | term type |word|word |word |word |word |word |word |word | word |word |word |word |word | > |--------------+----+-------+-----+------------+-----+-----+-----+------+--------+-----+------+-----+--------| > | source |0,2 |3,10 |11,12|13,25 |26,30|31,36|37,38|39,45 | 46,54 |55,58|59,64 |65,69|70,78 | > | start,end | | | | | | | | | | | | | | > |--------------+----+-------+-----+------------+-----+-----+-----+------+--------+-----+------+-----+--------| > | payload | | | | | | | | | | | | | | > |--------------+----+-------+-----+------------+-----+-----+-----+------+--------+-----+------+-----+--------| > > > org.apache.solr.analysis.LowerCaseFilterFactory {} > |--------------+----+-------+-----+------------+-----+-----+-----+------+--------+-----+-----+-----+--------| > |term position |1 |2 |3 |4 |5 |6 |7 |8 | 9 |10 |11 |12 |13 | > |--------------+----+-------+-----+------------+-----+-----+-----+------+--------+-----+-----+-----+--------| > | term text |to |produce|a |downloadable|file |using|a |format| suitable|for |ob10.|8-26 |profiles| > |--------------+----+-------+-----+------------+-----+-----+-----+------+--------+-----+-----+-----+--------| > | term type |word|word |word |word |word |word |word |word | word |word |word |word |word | > |--------------+----+-------+-----+------------+-----+-----+-----+------+--------+-----+-----+-----+--------| > | source |0,2 |3,10 |11,12|13,25 |26,30|31,36|37,38|39,45 | 46,54 |55,58|59,64|65,69|70,78 | > | start,end | | | | | | | | | | | | | | > |--------------+----+-------+-----+------------+-----+-----+-----+------+--------+-----+-----+-----+--------| > | payload | | | | | | | | | | | | | | > |--------------+----+-------+-----+------------+-----+-----+-----+------+--------+-----+-----+-----+--------| > > > > Query Analyzer > org.apache.solr.analysis.WhitespaceTokenizerFactory {} > |--------------+-----------| > |term position |1 | > |--------------+-----------| > | term text |OB10 | > |--------------+-----------| > | term type |word | > |--------------+-----------| > | source |0,4 | > | start,end | | > |--------------+-----------| > | payload | | > |--------------+-----------| > > > org.apache.solr.analysis.StandardFilterFactory {} > |--------------+----------| > |term position |1 | > |--------------+----------| > | term text |OB10 | > |--------------+----------| > | term type |word | > |--------------+----------| > | source |0,4 | > | start,end | | > |--------------+----------| > | payload | | > |--------------+----------| > > > org.apache.solr.analysis.LowerCaseFilterFactory {} > |--------------+-------------| > |term position |1 | > |--------------+-------------| > | term text |ob10 | > |--------------+-------------| > | term type |word | > |--------------+-------------| > | source |0,4 | > | start,end | | > |--------------+-------------| > | payload | | > |--------------+-------------| > > > I did look at ExtractingRequestHandler a while ago, but I don't think it > supported password protected files. Just looked at it again, and it looks > like it does now. > > > > > > From: Jan Høydahl / Cominvent <jan....@cominvent.com> > To: solr-user@lucene.apache.org > Date: 18/08/2010 23:16 > Subject: Re: Missing tokens > > > > Cannot see anything obvious... > > Try > http://localhost/solr/select?q=contents:OB10* > http://localhost/solr/select?q=contents:"OB 10" > http://localhost/solr/select?q=contents:"OB10."; > http://localhost/solr/select?q=contents:ob10 > > Also, go to the Analysis page in admin, typie in your field name, enable > verbose output and copy paste the problematic sentence in the "Index" part > and then enter a OB10 in the "Query" part, and see how your doc and query > gets processed. > > PS: Why don't you try this instead of doing the PDF extraction yourselv: > http://wiki.apache.org/solr/ExtractingRequestHandler ?? > > -- > Jan Høydahl, search solution architect > Cominvent AS - www.cominvent.com > Training in Europe - www.solrtraining.com > > On 18. aug. 2010, at 16.25, paul.mo...@dds.net wrote: > >> Here's my field description. I mentioned 'contents' field in my original >> post. I've changed it to a different field, 'summary'. It's using the >> 'text' fieldType as you can see below. >> >> <field name="summary" type="text" indexed="true" stored="true"/> >> >> >> <fieldType name="text" class="solr.TextField" positionIncrementGap="100"> >> <analyzer type="index"> >> <tokenizer class="solr.WhitespaceTokenizerFactory"/> >> <!-- in this example, we will only use synonyms at query time >> > <filter class="solr.SynonymFilterFactory" >> synonyms="index_synonyms.txt" > ignoreCase="true" expand="false"/> >> --> >> <!-- Case insensitive stop word removal. >> > enablePositionIncrements=true ensures that a 'gap' is left to >> > allow for accurate phrase queries. >> --> >> <filter class="solr.StopFilterFactory" >> ignoreCase="true" >> words="stopwords.txt" >> enablePositionIncrements="true" >> /> >> <filter class="solr.WordDelimiterFilterFactory" generateWordParts= >> "1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" >> catenateAll="0" splitOnCaseChange="1"/> >> <filter class="solr.LowerCaseFilterFactory"/> >> <filter class="solr.EnglishPorterFilterFactory" protected= >> "protwords.txt"/> >> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> >> </analyzer> >> <analyzer type="query"> >> <tokenizer class="solr.WhitespaceTokenizerFactory"/> >> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" >> ignoreCase="true" expand="true"/> >> <filter class="solr.StopFilterFactory" ignoreCase="true" words= >> "stopwords.txt"/> >> <filter class="solr.WordDelimiterFilterFactory" generateWordParts= >> "1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" >> catenateAll="0" splitOnCaseChange="1"/> >> <filter class="solr.LowerCaseFilterFactory"/> >> <filter class="solr.EnglishPorterFilterFactory" protected= >> "protwords.txt"/> >> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> >> </analyzer> >> </fieldType> >> >> I parsed the pdf using pdfbox. I can see my alphanumeric search term > 'OB10' >> in the extracted text before I add it to the index. I can also go into > Luke >> and see the 'OB10' in the contents of the 'summary' field even though > Luke >> can't find it when I do a search. >> >> I can also use the browser to do a search in http://localhost/solr/admin >> and again that search term doesn't return any results. I thought it may > be >> an alphanumber word splitting issue, but that doesn't seem be be the case >> since I can search on ME26, and it returns a doc, and in fact, I can see >> the 'OB10' search term in the summary field of the doc returned. >> >> Here's a snippet of the summary field from that returned doc >> >> To produce a downloadable file using a format suitable >> for OB10. 8-26 Profiles >> >> I'm thinking that the extracted text from pdfbox may have hidden chars > that >> solr can't parse. However, before I go down that road, I just want to be >> sure I'm not making schoolboy errors with my solr setup. >> >> thanks >> Paul >> >> >> >> From: Jan Høydahl / Cominvent <jan....@cominvent.com> >> To: solr-user@lucene.apache.org >> Date: 18/08/2010 11:56 >> Subject: Re: Missing tokens >> >> >> >> Hi, >> >> Can you share with us how your schema looks for this field? What > FieldType? >> What tokenizer and analyser? >> How do you parse the PDF document? Before submitting to Solr? With what >> tool? >> How do you do the query? Do you get the same results when doing the query >> from a browser, not SolrJ? >> >> -- >> Jan Høydahl, search solution architect >> Cominvent AS - www.cominvent.com >> Training in Europe - www.solrtraining.com >> >> On 18. aug. 2010, at 11.34, paul.mo...@dds.net wrote: >> >>> >>> Hi, I'm having a problem with certain search terms not being found when > I >>> do a query. I'm using Solrj to index a pdf document, and add the > contents >>> to the 'contents' field. If I query the 'contents' field on the >>> SolrInputDocument doc object as below, I get 50k tokens. >>> >>> StringTokenizer to = new StringTokenizer((String)doc.getFieldValue( >>> "contents")); >>> System.out.println( "Tokens:" + to.countTokens() ); >>> >>> However, once the doc is indexed and I use Luke to analyse the index, it >>> has only 3300 tokens in that field. Where did the other 47k go? >>> >>> I read some other threads mentioning to increase the maxfieldLength in >>> solrconfig.xml, and my setting is below. >>> >>> <maxFieldLength>2147483647</maxFieldLength> >>> >>> Any advice is appreciated, >>> Paul >>> >> >> >> > > >
