Richard Zowalla created OPENNLP-1974:
----------------------------------------
Summary: RegEx removal part 8: EmojiCharSequenceNormalizer blanks
hyphens and BMP characters
Key: OPENNLP-1974
URL: https://issues.apache.org/jira/browse/OPENNLP-1974
Project: OpenNLP
Issue Type: Bug
Reporter: Richard Zowalla
Assignee: Kristian Rickert
Fix For: 3.0.0
Split out of OPENNLP\-1928 \(PR #1275\). Depends on OPENNLP\-1973.
The pattern of {{EmojiCharSequenceNormalizer}} was parsed as the range U\+D83C
to U\+10FC00 plus a literal hyphen, so it also blanked the hyphen and the BMP
characters from U\+D83C to U\+FFFF \(fullwidth and halfwidth forms, CJK
compatibility ideographs, Arabic presentation forms, the private use area,
U\+FE0F, unpaired surrogates\) and kept U\+10FC01 to U\+10FFFF.
The normalizer now replaces each run of supplementary code points \(U\+10000
and above\) with one space, using a code point scan instead of the regular
expression. It is part of the default language detector chain, so models
trained with an earlier release see different input; under
{{\-Dopennlp.compat.mode=LEGACY}} the earlier output is produced.
Before merge: language detector eval build.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
