This is an automated email from the ASF dual-hosted git repository.
krickert pushed a change to branch feat/GH-1479-tokenizer-character-policy
in repository https://gitbox.apache.org/repos/asf/opennlp.git
from 050922919 GH-1466: Describe current behaviour instead of 3.0 changes
in the user manual (#1467)
add ff7c07682 GH-1479: Configure the tokenizer alphanumeric optimization
with TokenizerCharacterPolicy instead of a regex
No new revisions were added by this update.
Summary of changes:
README.md | 1 +
dev/forbidden-regex.txt | 2 +-
.../opennlp/tools/tokenize/TokSpanEventStream.java | 25 +-
.../tools/tokenize/TokenizerCharacterPolicy.java | 160 +++++++++++
.../opennlp/tools/tokenize/TokenizerFactory.java | 131 ++++++---
.../java/opennlp/tools/tokenize/TokenizerME.java | 15 +-
.../java/opennlp/tools/tokenize/lang/Factory.java | 81 +++---
.../tools/tokenize/DummyTokenizerFactory.java | 9 +-
.../tools/tokenize/LegacyAlphaNumericPatterns.java | 55 ++++
.../tools/tokenize/TokSpanEventStreamTest.java | 83 ++++++
.../tokenize/TokenizerCharacterPolicyTest.java | 133 +++++++++
.../tools/tokenize/TokenizerFactoryTest.java | 297 ++++++++++++++++-----
.../opennlp/tools/tokenize/TokenizerMETest.java | 61 +++++
.../opennlp/tools/tokenize/TokenizerTestUtil.java | 12 +-
.../opennlp/tools/tokenize/lang/FactoryTest.java | 164 ++++++++++++
opennlp-docs/src/docbkx/tokenizer.xml | 42 ++-
16 files changed, 1100 insertions(+), 171 deletions(-)
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/LegacyAlphaNumericPatterns.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/lang/FactoryTest.java
