[jira] Updated: (LUCENE-1124) short circuit FuzzyQuery.rewrite when input token length is small compared to minSimilarity
[ https://issues.apache.org/jira/browse/LUCENE-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-1124: --- Attachment: LUCENE-1124.patch Attach patch (based on 2.9) showing the bug, along with the fix. Instead of rewriting to empty BooleanQuery when prefix term is not long enough, I rewrite to TermQuery with that prefix. This way the exact term matches. I'll commit shortly to trunk 2.9.x. short circuit FuzzyQuery.rewrite when input token length is small compared to minSimilarity --- Key: LUCENE-1124 URL: https://issues.apache.org/jira/browse/LUCENE-1124 Project: Lucene - Java Issue Type: Improvement Components: Query/Scoring Reporter: Hoss Man Assignee: Mark Miller Priority: Trivial Fix For: 2.9 Attachments: LUCENE-1124.patch, LUCENE-1124.patch, LUCENE-1124.patch, LUCENE-1124.patch I found this (unreplied to) email floating around in my Lucene folder from during the holidays... {noformat} From: Timo Nentwig To: java-dev Subject: Fuzzy makes no sense for short tokens Date: Mon, 31 Dec 2007 16:01:11 +0100 Message-Id: 200712311601.12255.luc...@nitwit.de Hi! it generally makes no sense to search fuzzy for short tokens because changing even only a single character of course already results in a high edit distance. So it actually only makes sense in this case: if( token.length() 1f / (1f - minSimilarity) ) E.g. changing one character in a 3-letter token (foo) results in an edit distance of 0.6. And if minSimilarity (which is by default: 0.5 :-) is higher we can save all the expensive rewrite() logic. {noformat} I don't know much about FuzzyQueries, but this reasoning seems sound ... FuzzyQuery.rewrite should be able to completely skip all TermEnumeration in the event that the input token is shorter then some simple math on the minSimilarity. (i'm not smart enough to be certain that the math above is right however ... it's been a while since i looked at Levenstein distances ... tests needed) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-1124) short circuit FuzzyQuery.rewrite when input token length is small compared to minSimilarity
[ https://issues.apache.org/jira/browse/LUCENE-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-1124: --- Fix Version/s: (was: 2.9) 3.0 2.9.1 short circuit FuzzyQuery.rewrite when input token length is small compared to minSimilarity --- Key: LUCENE-1124 URL: https://issues.apache.org/jira/browse/LUCENE-1124 Project: Lucene - Java Issue Type: Improvement Components: Query/Scoring Reporter: Hoss Man Assignee: Mark Miller Priority: Trivial Fix For: 2.9.1, 3.0 Attachments: LUCENE-1124.patch, LUCENE-1124.patch, LUCENE-1124.patch, LUCENE-1124.patch I found this (unreplied to) email floating around in my Lucene folder from during the holidays... {noformat} From: Timo Nentwig To: java-dev Subject: Fuzzy makes no sense for short tokens Date: Mon, 31 Dec 2007 16:01:11 +0100 Message-Id: 200712311601.12255.luc...@nitwit.de Hi! it generally makes no sense to search fuzzy for short tokens because changing even only a single character of course already results in a high edit distance. So it actually only makes sense in this case: if( token.length() 1f / (1f - minSimilarity) ) E.g. changing one character in a 3-letter token (foo) results in an edit distance of 0.6. And if minSimilarity (which is by default: 0.5 :-) is higher we can save all the expensive rewrite() logic. {noformat} I don't know much about FuzzyQueries, but this reasoning seems sound ... FuzzyQuery.rewrite should be able to completely skip all TermEnumeration in the event that the input token is shorter then some simple math on the minSimilarity. (i'm not smart enough to be certain that the math above is right however ... it's been a while since i looked at Levenstein distances ... tests needed) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-1124) short circuit FuzzyQuery.rewrite when input token length is small compared to minSimilarity
[ https://issues.apache.org/jira/browse/LUCENE-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller updated LUCENE-1124: Attachment: LUCENE-1124.patch Updated to trunk. Im going to commit in few days if no one objects. short circuit FuzzyQuery.rewrite when input token length is small compared to minSimilarity --- Key: LUCENE-1124 URL: https://issues.apache.org/jira/browse/LUCENE-1124 Project: Lucene - Java Issue Type: Improvement Components: Query/Scoring Reporter: Hoss Man Attachments: LUCENE-1124.patch, LUCENE-1124.patch, LUCENE-1124.patch I found this (unreplied to) email floating around in my Lucene folder from during the holidays... {noformat} From: Timo Nentwig To: java-dev Subject: Fuzzy makes no sense for short tokens Date: Mon, 31 Dec 2007 16:01:11 +0100 Message-Id: 200712311601.12255.luc...@nitwit.de Hi! it generally makes no sense to search fuzzy for short tokens because changing even only a single character of course already results in a high edit distance. So it actually only makes sense in this case: if( token.length() 1f / (1f - minSimilarity) ) E.g. changing one character in a 3-letter token (foo) results in an edit distance of 0.6. And if minSimilarity (which is by default: 0.5 :-) is higher we can save all the expensive rewrite() logic. {noformat} I don't know much about FuzzyQueries, but this reasoning seems sound ... FuzzyQuery.rewrite should be able to completely skip all TermEnumeration in the event that the input token is shorter then some simple math on the minSimilarity. (i'm not smart enough to be certain that the math above is right however ... it's been a while since i looked at Levenstein distances ... tests needed) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-1124) short circuit FuzzyQuery.rewrite when input token length is small compared to minSimilarity
[ https://issues.apache.org/jira/browse/LUCENE-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller updated LUCENE-1124: Fix Version/s: 2.9 Assignee: Mark Miller short circuit FuzzyQuery.rewrite when input token length is small compared to minSimilarity --- Key: LUCENE-1124 URL: https://issues.apache.org/jira/browse/LUCENE-1124 Project: Lucene - Java Issue Type: Improvement Components: Query/Scoring Reporter: Hoss Man Assignee: Mark Miller Fix For: 2.9 Attachments: LUCENE-1124.patch, LUCENE-1124.patch, LUCENE-1124.patch I found this (unreplied to) email floating around in my Lucene folder from during the holidays... {noformat} From: Timo Nentwig To: java-dev Subject: Fuzzy makes no sense for short tokens Date: Mon, 31 Dec 2007 16:01:11 +0100 Message-Id: 200712311601.12255.luc...@nitwit.de Hi! it generally makes no sense to search fuzzy for short tokens because changing even only a single character of course already results in a high edit distance. So it actually only makes sense in this case: if( token.length() 1f / (1f - minSimilarity) ) E.g. changing one character in a 3-letter token (foo) results in an edit distance of 0.6. And if minSimilarity (which is by default: 0.5 :-) is higher we can save all the expensive rewrite() logic. {noformat} I don't know much about FuzzyQueries, but this reasoning seems sound ... FuzzyQuery.rewrite should be able to completely skip all TermEnumeration in the event that the input token is shorter then some simple math on the minSimilarity. (i'm not smart enough to be certain that the math above is right however ... it's been a while since i looked at Levenstein distances ... tests needed) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-1124) short circuit FuzzyQuery.rewrite when input token length is small compared to minSimilarity
[ https://issues.apache.org/jira/browse/LUCENE-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller updated LUCENE-1124: Priority: Trivial (was: Major) short circuit FuzzyQuery.rewrite when input token length is small compared to minSimilarity --- Key: LUCENE-1124 URL: https://issues.apache.org/jira/browse/LUCENE-1124 Project: Lucene - Java Issue Type: Improvement Components: Query/Scoring Reporter: Hoss Man Assignee: Mark Miller Priority: Trivial Fix For: 2.9 Attachments: LUCENE-1124.patch, LUCENE-1124.patch, LUCENE-1124.patch I found this (unreplied to) email floating around in my Lucene folder from during the holidays... {noformat} From: Timo Nentwig To: java-dev Subject: Fuzzy makes no sense for short tokens Date: Mon, 31 Dec 2007 16:01:11 +0100 Message-Id: 200712311601.12255.luc...@nitwit.de Hi! it generally makes no sense to search fuzzy for short tokens because changing even only a single character of course already results in a high edit distance. So it actually only makes sense in this case: if( token.length() 1f / (1f - minSimilarity) ) E.g. changing one character in a 3-letter token (foo) results in an edit distance of 0.6. And if minSimilarity (which is by default: 0.5 :-) is higher we can save all the expensive rewrite() logic. {noformat} I don't know much about FuzzyQueries, but this reasoning seems sound ... FuzzyQuery.rewrite should be able to completely skip all TermEnumeration in the event that the input token is shorter then some simple math on the minSimilarity. (i'm not smart enough to be certain that the math above is right however ... it's been a while since i looked at Levenstein distances ... tests needed) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-1124) short circuit FuzzyQuery.rewrite when input token length is small compared to minSimilarity
[ https://issues.apache.org/jira/browse/LUCENE-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otis Gospodnetic updated LUCENE-1124: - Summary: short circuit FuzzyQuery.rewrite when input token length is small compared to minSimilarity (was: short circuit FuzzyQuery.rewrite when input okenlengh is small compared to minSimilarity) short circuit FuzzyQuery.rewrite when input token length is small compared to minSimilarity --- Key: LUCENE-1124 URL: https://issues.apache.org/jira/browse/LUCENE-1124 Project: Lucene - Java Issue Type: Improvement Components: Query/Scoring Reporter: Hoss Man Attachments: LUCENE-1124.patch, LUCENE-1124.patch I found this (unreplied to) email floating around in my Lucene folder from during the holidays... {noformat} From: Timo Nentwig To: java-dev Subject: Fuzzy makes no sense for short tokens Date: Mon, 31 Dec 2007 16:01:11 +0100 Message-Id: [EMAIL PROTECTED] Hi! it generally makes no sense to search fuzzy for short tokens because changing even only a single character of course already results in a high edit distance. So it actually only makes sense in this case: if( token.length() 1f / (1f - minSimilarity) ) E.g. changing one character in a 3-letter token (foo) results in an edit distance of 0.6. And if minSimilarity (which is by default: 0.5 :-) is higher we can save all the expensive rewrite() logic. {noformat} I don't know much about FuzzyQueries, but this reasoning seems sound ... FuzzyQuery.rewrite should be able to completely skip all TermEnumeration in the event that the input token is shorter then some simple math on the minSimilarity. (i'm not smart enough to be certain that the math above is right however ... it's been a while since i looked at Levenstein distances ... tests needed) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]