Erick, Thank you. This is awesome. I got it to work by just setting slop to 1 and returning 10 in my analyzer.getPositionIncrementGap. Here are my tests in case anyone else is interested:
public class TestPositionIncrementGap extends TestCase { Analyzer analyzer = new KeywordAnalyzer(); RAMDirectory dir = null; IndexWriter writer = null; IndexSearcher searcher = null; @Override protected void setUp() throws Exception { super.setUp(); dir = new RAMDirectory(); writer = new IndexWriter(dir, analyzer, MaxFieldLength.LIMITED); Document d = new Document(); d.add(new Field("keyword", "aaa bbb", Field.Store.YES, Field.Index.ANALYZED)); d.add(new Field("keyword", "ccc ddd eee", Field.Store.YES, Field.Index.ANALYZED)); d.add(new Field("keyword", "fff fff", Field.Store.YES, Field.Index.ANALYZED)); d.add(new Field("keyword", "banks sales", Field.Store.YES, Field.Index.ANALYZED)); d.add(new Field("keyword", "sales mans", Field.Store.YES, Field.Index.ANALYZED)); writer.addDocument(d); writer.close(); } @Override protected void tearDown() throws Exception { super.tearDown(); dir = null; writer = null; searcher = null; } public void testMatchOnSingleExactValue() throws Exception { searcher = new IndexSearcher(dir, true); PhraseQuery pq = new PhraseQuery(); pq.setSlop(1); pq.add(new Term("keyword", "aaa")); pq.add(new Term("keyword", "bbb")); TopDocs td = searcher.search(pq, 100); searcher.close(); assertEquals(1, td.totalHits); } public void testNoMatchOnAdjacentValue() throws Exception { searcher = new IndexSearcher(dir, true); PhraseQuery pq = new PhraseQuery(); pq.setSlop(1); pq.add(new Term("keyword", "bbb")); pq.add(new Term("keyword", "ccc")); TopDocs td = searcher.search(pq, 100); assertEquals(0, td.totalHits); searcher.close(); } public void testNoMatchOnSingleReveredValue() throws Exception { searcher = new IndexSearcher(dir, true); PhraseQuery pq = new PhraseQuery(); pq.setSlop(1); pq.add(new Term("keyword", "bbb")); pq.add(new Term("keyword", "aaa")); TopDocs td = searcher.search(pq, 100); assertEquals(0, td.totalHits); searcher.close(); } public void testNoMatchOnSpannedValue() throws Exception { searcher = new IndexSearcher(dir, true); PhraseQuery pq = new PhraseQuery(); pq.setSlop(1); pq.add(new Term("keyword", "aaa")); pq.add(new Term("keyword", "mans")); TopDocs td = searcher.search(pq, 100); assertEquals(0, td.totalHits); searcher.close(); } public void testMatchOnSingleExactValue2() throws Exception { searcher = new IndexSearcher(dir, true); PhraseQuery pq = new PhraseQuery(); pq.setSlop(1); pq.add(new Term("keyword", "fff")); pq.add(new Term("keyword", "fff")); TopDocs td = searcher.search(pq, 100); searcher.close(); assertEquals(1, td.totalHits); } } -----Original Message----- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Monday, October 12, 2009 1:48 PM To: java-user@lucene.apache.org Subject: Re: querying multi-value fields <<<I think Lucene sees all these values as one long value for the field "option">>> Not quite. Starting with the second add, a call will be made to getPositionIncrementGap in your analyzer. If you return a number larger than one, then the offsets between the last term of the preceeding add and the first term of this add will be that number. If you do nothing with getPositionIncrementGap, then Lucene does, indeed, see all the terms as one long value since it returns 1..... Here's an illusrtation in which PositionIncrementGap returns an offset of 10 using your example adds. Note, these numbers may be off by the infamous 1 or even 2. term offset value1 0 aaa 1 value2 11 bbb 12 value3 22 ccc 23 So, you really don't care about the slop, since you can set it to less than the magic number you return from PositionIncrementGap. BTW, slop indicates holes, not total terms. So with a slop of 0 all the words need to be next to each other, regardless of whether there are two words or 20. But you still have to do the trick with getPositionIncrementGap in order to fail to match on something like "bbb value3", where the last term is next to the frist term of the next token...... HTH Erick On Mon, Oct 12, 2009 at 4:31 PM, Angel, Eric <ean...@business.com> wrote: > I need to analyze these values since I also want the benefits > porterStemmer. The problem with using PhraseQuery is that I don't > always know the slop. I may have values like "value4 ddd aaa". It's a > tricky problem because I think Lucene sees all these values as one long > value for the field "option". > > -----Original Message----- > From: Jake Mannix [mailto:jake.man...@gmail.com] > Sent: Monday, October 12, 2009 1:25 PM > To: java-user@lucene.apache.org > Subject: Re: querying multi-value fields > > Or else just make sure that you use PhraseQuery to hit this field when > you > want "value1 aaa". If you don't tokenize these pairs, then you will > have to > > do prefix/wildcard matching to hit just "value1" by itself (if this is > allowed > by your business logic). > > -jake > > On Mon, Oct 12, 2009 at 1:21 PM, Adriano Crestani > <adrianocrest...@gmail.com > > wrote: > > > Hi Eric, > > > > To achieve what you want, do not tokenize the values you query/add to > this > > field. > > > > On Mon, Oct 12, 2009 at 4:05 PM, Angel, Eric <ean...@business.com> > wrote: > > > > > I have documents that store multiple values in some fields (using > the > > > document.add(new Field()) with the same field name). Here's what a > > > typical document looks like: > > > > > > > > > > > > doc.option="value1 aaa" > > > > > > doc.option="value2 bbb" > > > > > > doc.option="value3 ccc" > > > > > > > > > > > > I want my queries to only match individual values, for example, a > query > > > for "value2 bbb" would return the above document, but a query for > > > "value1 ccc" should not. Is this at all possible in lucene at query > > > time? Could payloads be used for this? > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org