[ https://issues.apache.org/jira/browse/LUCENE-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1606: --------------------------------------- Attachment: LUCENE-1606-flex.patch First cut @ cutting over to flex API attached -- note that this applies to the flex branch, not trunk! I made some small changes to the benchmarker: use constant score filter mode, and print the min (not avg) time (less noise). Also, I ported the AutomatonTermEnum to the flex API, so this is now a better measure ("flex on flex") of what future perf will be. It's possible there's a bug here, though TestWildcard passes. I still need to investigate why "non-flex on non-flex" and "non-flex on flex" perform worse. I ran like this: java -server -Xmx1g -Xms1g BenchWildcard java is 1.6.0_14 64 bit, on OpenSolaris. Results (msec is min of 10 runs each); ||Pattern||ITrunk (min msec)||(Flex (min msec)|| |N?N?N?N0.0|13|18| |?NNNNNN|1|3| |??NNNNN|4|6| |???NNNN|23|28| |????NNN|210|170| |NN??NNN|3|3| |NN?N*|7|4| |?NN*|62|30| |*N|4332|2576| |NNNNN??|1|1| Looks like flex API is faster for the slow queries. Once I fix caching on trunk we should retest... > Automaton Query/Filter (scalable regex) > --------------------------------------- > > Key: LUCENE-1606 > URL: https://issues.apache.org/jira/browse/LUCENE-1606 > Project: Lucene - Java > Issue Type: New Feature > Components: Search > Reporter: Robert Muir > Assignee: Robert Muir > Priority: Minor > Fix For: 3.1 > > Attachments: automaton.patch, automatonMultiQuery.patch, > automatonmultiqueryfuzzy.patch, automatonMultiQuerySmart.patch, > automatonWithWildCard.patch, automatonWithWildCard2.patch, > BenchWildcard.java, LUCENE-1606-flex.patch, LUCENE-1606.patch, > LUCENE-1606.patch, LUCENE-1606.patch, LUCENE-1606.patch, LUCENE-1606.patch, > LUCENE-1606.patch, LUCENE-1606.patch, LUCENE-1606.patch, > LUCENE-1606_nodep.patch > > > Attached is a patch for an AutomatonQuery/Filter (name can change if its not > suitable). > Whereas the out-of-box contrib RegexQuery is nice, I have some very large > indexes (100M+ unique tokens) where queries are quite slow, 2 minutes, etc. > Additionally all of the existing RegexQuery implementations in Lucene are > really slow if there is no constant prefix. This implementation does not > depend upon constant prefix, and runs the same query in 640ms. > Some use cases I envision: > 1. lexicography/etc on large text corpora > 2. looking for things such as urls where the prefix is not constant (http:// > or ftp://) > The Filter uses the BRICS package (http://www.brics.dk/automaton/) to convert > regular expressions into a DFA. Then, the filter "enumerates" terms in a > special way, by using the underlying state machine. Here is my short > description from the comments: > The algorithm here is pretty basic. Enumerate terms but instead of a > binary accept/reject do: > > 1. Look at the portion that is OK (did not enter a reject state in the > DFA) > 2. Generate the next possible String and seek to that. > the Query simply wraps the filter with ConstantScoreQuery. > I did not include the automaton.jar inside the patch but it can be downloaded > from http://www.brics.dk/automaton/ and is BSD-licensed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org