Thanks Erick, I didn't get confused with multiple tokens vs multiValued :)
Before I go ahead and re-index 4m docs, and believe me I'm using the analysis page like a mad-man! What do I need to configure to have the following both indexed with and without the dots... .net sales manager. £12.50 Currently... <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="1" splitOnCaseChange="1" splitOnNumerics="1" types="wdftypes.txt" /> with nothing specific in wdftypes.txt for full-stops. Should there also be any difference when quoting my searches? The analysis page seems to just drop the quotes, but surely actual calls don't do this? --- IntelCompute Web Design & Local Online Marketing http://www.intelcompute.com On Wed, 8 Feb 2012 07:38:42 -0500, Erick Erickson <erickerick...@gmail.com> wrote: > Yes, WDDF creates multiple tokens. But that has > nothing to do with the multiValued suggestion. > > You can get exactly what you want by > 1> setting multiValued="true" in your schema file and re-indexing. Say > positionIncrementGap is set to 100 > 2> When you index, add the field for each sentence, so your doc > looks something like: > <doc> > <field name="sentences">i am a sales-manager in here</field> > <field name="sentences">using asp.net and .net daily</field> > ..... > </doc> > 3> search like "sales manager"~100 > > Best > Erick > > On Wed, Feb 8, 2012 at 3:05 AM, Rob Brown <r...@intelcompute.com> wrote: >> Apologies if things were a little vague. >> >> Given the example snippet to index (numbered to show searches needed to >> match)... >> >> 1: i am a sales-manager in here >> 2: using asp.net and .net daily >> 3: working in design. >> 4: using something called sage 200. and i'm fluent >> 5: german sausages. >> 6: busy A&E dept earning £10,000 annually >> >> >> ... all with newlines in place. >> >> able to match... >> >> 1. sales >> 1. "sales manager" >> 1. sales-manager >> 1. "sales-manager" >> 2. .net >> 2. asp.net >> 3. design >> 4. sage 200 >> 6. A&E >> 6. £10,000 >> >> But do NOT match "fluent german" from 4 + 5 since there's a newline >> between them when indexed, but not when searched. >> >> >> Do the filters (wdf in this case) not create multiple tokens, so if >> splitting on period in "asp.net" would create tokens for all of "asp", >> "asp.", "asp.net", ".net", "net". >> >> >> Cheers, >> Rob >> >> -- >> >> IntelCompute >> Web Design and Online Marketing >> >> http://www.intelcompute.com >> >> >> -----Original Message----- >> From: Chris Hostetter <hossman_luc...@fucit.org> >> Reply-to: solr-user@lucene.apache.org >> To: solr-user@lucene.apache.org >> Subject: Re: Which Tokeniser (and/or filter) >> Date: Tue, 7 Feb 2012 15:02:36 -0800 (PST) >> >> : This all seems a bit too much work for such a real-world scenario? >> >> You haven't really told us what your scenerio is. >> >> You said you want to split tokens on whitespace, full-stop (aka: >> period) and comma only, but then in response to some suggestions you added >> comments other things that you never mentioned previously... >> >> 1) evidently you don't want the "." in foo.net to cause a split in tokens? >> 2) evidently you not only want token splits on newlines, but also >> positition gaps to prevent phrases matching across newlines. >> >> ...these are kind of important details that affect suggestions people >> might give you. >> >> can you please provide some concrete examples of hte types of data you >> have, the types of queries you want them to match, and the types of >> queries you *don't* want to match? >> >> >> -Hoss >>