Thanks, it looks easy and probably good enough for the first cut. I am only hesitating because this approach allows only one type of the fuzzy query at a time (global) irrespective of the field/Token type. The direction I am thinking in is more like having one fuzzy query (no new tokens for parser...) that dynamically selects the distance function based on the Field name.
Argumentation for this approach is not difficult to explain, e.g. there are some nice heuristics for fuzzy date comparison, as well as for zip code, first names, emails (if the domain is the same and the rest is slightly different…), for gene matching (LCS)... The trick is that all of these have different comparison functions. I guess decision which distance measure should be used can be done based on the Field name (type?). Your comment about linear scan of all tokens, yes this hurts big time. Typical way to deal with this is to limit scan to the tokens that begin with prefix of the search term (how long prefix depends on how long is the search term). Also you can skip tokens whose length differs too much (way faster than calculating the distance). Something like that is a must if you have large token set. But this is compromise optimisation, will think about this later. I am not yet comfortable with the Lucene code and would highly appreciate your comments especially if I am missing the point. Cheers, Eks --- Robert Engels <[EMAIL PROTECTED]> wrote: > I think it is as simple as 'modifying' > FuzzyQuery.java, check a 'environment > variable', and then either instantiate a > FuzzTermEnum(), or a > SpecialFuzzyTermEnum(). > > All of the logic of 'FuzzyTerm' is contained in > FuzzyTermEnum.java. If it is > nothing more than a different difference > calculation, then just check the > environment variable in FuzzTermEnum.java, and call > the appropriate distance > calculation routine. > > If you create a new 'Query' class, then you have to > modify the expression > language to add a new 'term' character, which could > get messy eventually > (run out of characters). > > The only issue with the current FuzzyTerm > implementation is that is requires > a full linear search of all of the terms in index. > > Robert > > -----Original Message----- > From: Erik Hatcher > [mailto:[EMAIL PROTECTED] > Sent: Tuesday, April 20, 2004 4:44 AM > To: Lucene Developers List > Subject: Re: Fuzzy query with Jaro-Winkler distance > > > On Apr 20, 2004, at 5:11 AM, eks dev wrote: > > Hi All, > > I would like to use Fuzzy Query with another > > type(s) of string distance. > > You will have to write your own Query (probably > subclass > MultiTermQuery) to do this. The FuzzyQuery > calculations are buried > deep and not customizable (at least not currently). > > Erik ____________________________________________________________ Yahoo! Messenger - Communicate instantly..."Ping" your friends today! Download Messenger Now http://uk.messenger.yahoo.com/download/index.html --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
