[ https://issues.apache.org/jira/browse/OPENNLP-141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17693839#comment-17693839 ]
ASF GitHub Bot commented on OPENNLP-141: ---------------------------------------- kinow commented on PR #506: URL: https://github.com/apache/opennlp/pull/506#issuecomment-1445860402 > > Couldn't find a good reference for Italian or Spanish > > So better create a separate Jira issue for this and consider it an improvement? This way, we could verify and/or ask the community for support before making too many assumptions. Good idea! Done: https://issues.apache.org/jira/browse/OPENNLP-1474 > Tokenizers alpha numeric optimization only recognizes a-z as alpha chars > ------------------------------------------------------------------------ > > Key: OPENNLP-141 > URL: https://issues.apache.org/jira/browse/OPENNLP-141 > Project: OpenNLP > Issue Type: Bug > Components: Tokenizer > Affects Versions: tools-1.5.0-sourceforge > Reporter: Jörn Kottmann > Assignee: Martin Wiesner > Priority: Minor > > The Tokenizer has an optimization which skips tokens which are only made of > numerics or alpha chars. In foreign languages the alpha chars contain umlauts > and other letters which are not included in the a-z range. -- This message was sent by Atlassian Jira (v8.20.10#820010)