Hi Mark, On Wed, 2008-06-18 at 21:09 +0100, markharw00d wrote: > This looks like it is related to an issue I first raised here: > http://markmail.org/message/37ywsemfudpos6uh > > At the time I identified 2 issues with FuzzyQuery - that the usual > "coord" and "idf" scoring factors shouldn't be applied to fuzzy queries. > The coord factor got fixed but idf remains an issue in FuzzyQuery I beleive. > There is a class I added to contrib/queries - "FuzzyLikeThisQuery", > which can be used as a replacement to FuzzyQuery and fixes the idf > issue. You can subclass the QueryParser to create an instance of that > rather than FuzzyQuery if required.
Frankly, I neighter understand the deeper anatomy of Lucene, nor do I want to. I'd expect Lucene to do the right thing with fuzzy queries and it obviously doesn't do the right thing in my case. I get clearly irrelevant results when I search for a word using fuzzy query, that world is stored in my Lucene index and Lucene returns me a word that is pretty different. It seems to me a bug. I've downloaded FuzzyQuery from SVN and replaced the references of the Fuzzy class with it. I got the same irrelevant results than before. Do you have any ideas what's going on here? > > Cheers, > Mark > > László Monda wrote: > > Hi List, > > > > I've been redirected from [EMAIL PROTECTED] to here to discuss > > my issue. > > > > > > ---------- My original email ---------- > > > > > > I try to provide relevant results for the users of a lyrics site, even > > in the case of misspellings by indexing artist and songs with Lucene. > > > > The problem is that Lucene provides irrelevant search results. For > > example searching for "Coldplay" returns "Longplay" as the most relevant > > result. > > > > This is how I create individual documents: > > > > Document document = new Document(); > > document.add(new Field("artist", artist, Field.Store.YES, > > Field.Index.UN_TOKENIZED)); > > document.add(new Field("song", song, Field.Store.YES, > > Field.Index.UN_TOKENIZED)); > > document.add(new Field("path", path, Field.Store.YES, Field.Index.NO)); > > indexWriter.addDocument(document); > > > > And this is how I compose the actual query: > > > > BooleanQuery query = new BooleanQuery(); > > if (artist.length() > 0) { > > FuzzyQuery artist_query = new FuzzyQuery(new Term("artist", > > artist)); > > query.add(artist_query, BooleanClause.Occur.MUST); > > } > > if (song.length() > 0) { > > FuzzyQuery song_query = new FuzzyQuery(new Term("song", song)); > > query.add(song_query, BooleanClause.Occur.MUST); > > } > > > > Please let me know what's wrong, I'd like to make this work right. > > > > Thanks in advance! > > > > > > ---------- My reply to an answer ---------- > > > > > > On Tue, 2008-06-17 at 20:38 +0200, Daniel Naber wrote: > > > >> On Dienstag, 17. Juni 2008, László Monda wrote: > >> > >> > >>> FuzzyQuery artist_query = new FuzzyQuery(new Term("artist", > >>> artist)); > >>> > >> You should try the FuzzyQuery constructor that takes a minimum > >> > > similarity > > > >> and a prefix length. The general problem is however, that the degree > >> > > of > > > >> similarity is only one factor. The other factors are the same as for > >> > > other > > > >> searches, e.g. the number of occurences of the term in the document > >> > > and in > > > >> the whole index. > >> > >> You could try to write your own similarity implementation that > >> > > disables all > > > >> these factors, see > >> > >> > > http://lucene.apache.org/java/2_3_1/api/org/apache/lucene/search/Similarity.html > > > > > > I understand some essential concepts related to Lucene such as the > > Levenshtein distance and tokenization, but I really don't want to go > > this deep if it's not necessary. > > > > Since fuzzy searching is based on the Levenshtein distance, the distance > > between "coldplay" and "coldplay" is 0 and the distance between > > "coldplay" and "downplay" is 3 so how on earth is possible that when > > searching for "coldplay", Lucene returns "longplay"? This shouldn't > > happen regardless of the minimum similarity and prefix length factors. > > > > Additional info: Lucene seems to do the right thing when only few > > documents are present, but goes crazy when there is about 1.5 million > > documents in the index. > > > > > > --------------------------------------------------------------------- > > > > > > I hope that some of you can help me because I don't have any ideas what > > can be wrong here. > > > > Thanks in advance! > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > -- Laci <http://monda.hu>
signature.asc
Description: This is a digitally signed message part