[ https://issues.apache.org/jira/browse/OPENNLP-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842864#comment-17842864 ]
ASF GitHub Bot commented on OPENNLP-1555: ----------------------------------------- mawiesne merged PR #599: URL: https://github.com/apache/opennlp/pull/599 > TokenizerME should detect multi-dot abbreviations > ------------------------------------------------- > > Key: OPENNLP-1555 > URL: https://issues.apache.org/jira/browse/OPENNLP-1555 > Project: OpenNLP > Issue Type: Improvement > Components: Tokenizer > Affects Versions: 2.1.0, 2.2.0, 2.3.0, 2.3.1, 2.3.2, 2.3.3 > Reporter: Martin Wiesner > Assignee: Martin Wiesner > Priority: Minor > Fix For: 2.3.4 > > > TokenizerME should detect and handle multi-dot abbreviations correctly. > Currently, this is not handled correctly. For instance, > German: "z.B." = "zum Beispiel" (for example) or, > Dutch: "e.v." = "en volgende" (and following) > are not tokenized correctly and extra tokens are returned. NOTE: no > whitespaces in between the dots in the above examples. > Aims: > * Fix the detection / handling of abbreviations for multi-dot abbreviations > * Provide test cases that cover these cases -- This message was sent by Atlassian Jira (v8.20.10#820010)