krickert opened a new pull request, #1481: URL: https://github.com/apache/opennlp/pull/1481
TokenizerME, TokSpanEventStream, TokenizerFactory and the language Factory decide which candidate tokens the alphanumeric optimization leaves unsplit with a TokenizerCharacterPolicy instead of a java.util.regex Pattern. The tokenizer package has no @SuppressForbidden members left. The factory writes the policy to the model manifest as characterPolicy, in the text form of TokenizerCharacterPolicy.toString(), which TokenizerCharacterPolicy.parse(String) reads. A model from an earlier release with one of the alphaNumericPattern values OpenNLP wrote loads with the matching language policy, so the released models need no conversion. A model with a custom pattern fails to load with a message that names the pattern. A manifest with no policy and no pattern uses the policy for its language code. This removes API in 3.0 without a deprecation step: the Pattern parameters of the TokenizerFactory constructor, TokenizerFactory.create and the TokSpanEventStream constructor take a TokenizerCharacterPolicy, TokenizerFactory.getAlphaNumericPattern() is replaced by getCharacterPolicy(), and Factory.getAlphanumeric(String) and Factory.DEFAULT_ALPHANUMERIC by getCharacterPolicy(String) and DEFAULT_CHARACTER_POLICY. OpenNLP 2.x reads only alphaNumericPattern, so it applies its default pattern to a model written by this release. Tests: each language policy against its old regular expression as the oracle, TokenizerME output pinned with and without the optimization, a custom policy, the policy text form, and the manifest with the policy, every legacy pattern, a custom pattern, a malformed policy and a manifest without either property. The tokenizer chapter and the README describe the policy and the migration. An eval build is needed before merge, since the tokenizer path of every model changes. Fixes #1479 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
