[
https://issues.apache.org/jira/browse/LUCENE-7465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-7465:
---------------------------------------
Attachment: LUCENE-7465.patch
Patch.
I added the {{SimplePatternTokenizerFactory}} as well.
{{SimplePatternTokenizer}} takes either a String (parses as a regexp
and compiles it) or a DFA (expert user who pre-built their own
automaton).
I folded in a nice idea from [~rcmuir] to optimize the ascii code
points even when using {{CharacterRunAutomaton}}.
It's quite fast, ~46% faster than {{PatternTokenizer}} when tokenizing
1 MB chunks from the English Wikipedia export, using a simplistic
whitespace regexp {{\[^ \t\r\n]+}}.
And it's nice that it doesn't read the entire input into heap!
> Add a PatternTokenizer that uses Lucene's RegExp implementation
> ---------------------------------------------------------------
>
> Key: LUCENE-7465
> URL: https://issues.apache.org/jira/browse/LUCENE-7465
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: master (7.0), 6.3
>
> Attachments: LUCENE-7465.patch
>
>
> I think there are some nice benefits to a version of PatternTokenizer that
> uses Lucene's RegExp impl instead of the JDK's:
> * Lucene's RegExp is compiled to a DFA up front, so if a "too hard" RegExp
> is attempted the user discovers it up front instead of later on when a
> "lucky" document arrives
> * It processes the incoming characters as a stream, only pulling 128
> characters at a time, vs the existing {{PatternTokenizer}} which currently
> reads the entire string up front (this has caused heap problems in the past)
> * It should be fast.
> I named it {{SimplePatternTokenizer}}, and it still needs a factory and
> improved tests, but I think it's otherwise close.
> It currently does not take a {{group}} parameter because Lucene's RegExps
> don't yet implement sub group capture. I think we could add that at some
> point, but it's a bit tricky.
> This doesn't even have group=-1 support (like String.split) ... I think if we
> did that we should maybe name it differently
> ({{SimplePatternSplitTokenizer}}?).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
