[
https://issues.apache.org/jira/browse/LUCENE-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12781175#action_12781175
]
Mark Miller edited comment on LUCENE-2089 at 11/22/09 6:06 PM:
---------------------------------------------------------------
bq. we must "use" the prefix, so the results are the same as old FuzzyQuery for
back compat
I'm not so worried about that myself. The prefix on the old Fuzzy is just to
get around scalability - we could just deprecate and create a new Fuzzy without
it. Why carry along the hack when we fix the root problem?
*edit*
Basically, what I'm saying is the old FuzzyQuery is just garbage really - I've
even seen an email were Doug talked about just dropping it. He didn't like the
non scalable queries. Even if this new impl didn't exactly match the results of
the old, I'd have no problem just deprecating the old and saying we are
starting over with a "real" fuzzyquery - the old one goes away on the next
major - anyone dying to stick with it can just pull the query into their own
code.
I think if we keep the prefix, it should be for a good reason in itself, rather
than just back compat with the old crappy version.
was (Author: markrmil...@gmail.com):
bq. we must "use" the prefix, so the results are the same as old FuzzyQuery
for back compat
I'm not so worried about that myself. The prefix on the old Fuzzy is just to
get around scalability - we could just deprecate and create a new Fuzzy without
it. Why carry along the hack when we fix the root problem?
> explore using automaton for fuzzyquery
> --------------------------------------
>
> Key: LUCENE-2089
> URL: https://issues.apache.org/jira/browse/LUCENE-2089
> Project: Lucene - Java
> Issue Type: Wish
> Components: Search
> Reporter: Robert Muir
> Assignee: Mark Miller
> Priority: Minor
>
> Mark brought this up on LUCENE-1606 (i will assign this to him, I know he is
> itching to write that nasty algorithm)
> we can optimize fuzzyquery by using AutomatonTermEnum, here is my idea
> * up front, calculate the maximum required K edits needed to match the users
> supplied float threshold.
> * for at least common K (1,2,3, etc) we should use automatontermenum. if its
> outside of that, maybe use the existing slow logic. At high K, it will seek
> too much to be helpful anyway.
> i modified my wildcard benchmark to generate random fuzzy queries.
> * Pattern: 7N stands for NNNNNNN, etc.
> * AvgMS_DFA: this is the time spent creating the automaton (constructor)
> ||Pattern||Iter||AvgHits||AvgMS(old)||AvgMS (new,total)||AvgMS_DFA||
> |7N|10|64.0|4155.9|38.6|20.3|
> |14N|10|0.0|2511.6|46.0|37.9|
> |28N|10|0.0|2506.3|93.0|86.6|
> |56N|10|0.0|2524.5|304.4|298.5|
> as you can see, this prototype is no good yet, because it creates the DFA in
> a slow way. right now it creates an NFA, and all this wasted time is in
> NFA->DFA conversion.
> So, for a very long string, it just gets worse and worse. This has nothing to
> do with lucene, and here you can see, the TermEnum is fast (AvgMS -
> AvgMS_DFA), there is no problem there.
> instead we should just build a DFA to begin with, maybe with this paper:
> http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.16.652
> we can precompute the tables with that algorithm up to some reasonable K, and
> then I think we are ok.
> the paper references using http://portal.acm.org/citation.cfm?id=135907 for
> linear minimization, if someone wants to implement this they should not worry
> about minimization.
> in fact, we need to at some point determine if AutomatonQuery should even
> minimize FSM's at all, or if it is simply enough for them to be deterministic
> with no transitions to dead states. (The only code that actually assumes
> minimal DFA is the "Dumb" vs "Smart" heuristic and this can be rewritten as a
> summation easily). we need to benchmark really complex DFAs (i.e. write a
> regex benchmark) to figure out if minimization is even helping right now.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
